Re: scheduled mailing from URL error

 
From: "Mary Ann Kelley" <maryann@PROTECTED>
Date: August 30th 2012
I just heard back from Host Gator tech support (which is incredibly awesome, BTW - they never tell me, "sorry, we don't support scripts running on your server") and they were able to fix the problem. It was a problem in .htaccess:

"I've found the issue, mostly. The .htaccess file was modified yesterday. This is an extremely large htaccess file. Something in there is blocking our wget bot or requests without user-agents, as when I attempted to wget the page from my workstation and your server, I received a 404 error, and no output. This is why you were receiving empty messages. I could not find the specific rule which was blocking the server from grabbing that page, but what I did is add a rule to the top to create an exception for that particular file. The page now loads when I wget it from my browser and your server, and a test message now sends the full html page. If you have any questions, or there is anything else I can do for you, please let me know. "

I think I know what it might have been. I am running a WP plugin to secure my WP installation and I recently allowed it to enable HackRepair.com's blacklist feature. http://hackrepair.com/blog/how-to-block-bots-from-seeing-your-website-bad-bots-and-drive-by-hacks-explained  Not sure how that affected Dada accessing the page, but I'm 99% sure that did it. I don't know anything about .htaccess other than how to do redirects, but maybe this line affected it (I'm taking a stab in the dark that it's banning requests without user agents)? 
RewriteRule ^.* - [F,L]
All the others seem to refer to specific bots, IPs, or hosts.

Thanks for looking into it for me!
Untitled Document
Warm regards,

Mary Ann



On Aug 30, 2012, at 4:56 PM, Justin J wrote:


These errors are interesting:

On Aug 30, 2012, at 2:34 PM, Mary Ann Kelley wrote:
Here are all the recent errors from a test list that got the same result:
[Thu Aug 30 11:49:45 2012] scheduled_mailings.pl: Can't use an undefined value as a HASH reference at scheduled_mailings.pl line 774.
[Thu Aug 30 11:50:28 2012] scheduled_mailings.pl: 403 Forbidden at ..//DADA/MailingList/Schedules.pm line 1173.

They have to do with accessing things through a password protected directory - and basically, the login isn't working. Had something changed there? Obviously, I can access,

http://www.menus4moms.com/todays_task_email.php

without a user/pass, so it's curious that this is failing. Trying it with my own test Dada Mail, I get almost the same error. I wonder if you're server is blocking HTTP requests from Dada Mail? That's usually done, by just looking at the User Agent (dumbly) and blocking things that looks suspicious. In my copy of Dada Mail, here's what the method that grabs content from a URL looks like:

/dada/DADA/MailingList/Schedules.pm

sub _from_url {

my $self    = shift;
my $url     = shift;
my $type    = shift;
my $record  = shift;

# Create a user agent object
require LWP::UserAgent;
my $ua = LWP::UserAgent->new;

# $ua->agent("MyApp/0.1 ");

if(defined($record->{$type . '_ver'}->{proxy})){
$ua->proxy(
['http', 'ftp'],
$record->{$type . '_ver'}->{proxy}
);
}

# Create a request
my $req = HTTP::Request->new(
GET => $url
);
# Pass request to the user agent and get a response back
my $res = $ua->request($req);
if(
 defined($record->{$type . '_ver'}->{url_username}) &&
 defined($record->{$type . '_ver'}->{url_password})
){
  $res->authorization_basic(
$record->{$type . '_ver'}->{url_username},
$record->{$type . '_ver'}->{url_password}
);
}
# Check the outcome of the response
if ($res->is_success) {
   return $res->content;
}
else {
   warn $res->status_line;
return undef;
}
}


If I just explicitly add a name to our "Browser",

$ua->agent('Mozilla/5.0 (compatible; ' . $DADA::CONFIG::PROGRAM_NAME . ')');

it seems to work!:


sub _from_url {

my $self    = shift;
my $url     = shift;
my $type    = shift;
my $record  = shift;

# Create a user agent object
require LWP::UserAgent;
my $ua = LWP::UserAgent->new;



$ua->agent('Mozilla/5.0 (compatible; ' . $DADA::CONFIG::PROGRAM_NAME . ')');



if(defined($record->{$type . '_ver'}->{proxy})){
$ua->proxy(
['http', 'ftp'],
$record->{$type . '_ver'}->{proxy}
);
}

# Create a request
my $req = HTTP::Request->new(
GET => $url
);
# Pass request to the user agent and get a response back
my $res = $ua->request($req);
if(
 defined($record->{$type . '_ver'}->{url_username}) &&
 defined($record->{$type . '_ver'}->{url_password})
){
  $res->authorization_basic(
$record->{$type . '_ver'}->{url_username},
$record->{$type . '_ver'}->{url_password}
);
}
# Check the outcome of the response
if ($res->is_success) {
   return $res->content;
}
else {
   warn $res->status_line;
return undef;
}
}

You can grab the entire file to try here:

https://gist.github.com/3540628


This exact change was made in the rest of Dada Mail, I don't know why it wasn't in Beatitude - I'll go ahead and fix that for the next release,

--

Justin J: Lead Dadaist.
url:         http://dadamailproject.com
email:    justin@PROTECTED
skype:   leaddadaist




On Aug 30, 2012, at 2:34 PM, Mary Ann Kelley wrote:

It is sending this page (but I tested it with other pages with the same result), so just do a "view source":
http://www.menus4moms.com/todays_task_email.php

Here are all the recent errors from a test list that got the same result:
[Thu Aug 30 11:49:45 2012] scheduled_mailings.pl: Can't use an undefined value as a HASH reference at scheduled_mailings.pl line 774.
[Thu Aug 30 11:50:28 2012] scheduled_mailings.pl: 403 Forbidden at ..//DADA/MailingList/Schedules.pm line 1173.
[Thu Aug 30 11:50:28 2012] scheduled_mailings.pl: multipart/mixed with no parts?! Something is screwy.... at ..//DADA/App/FormatMessages.pm line 647.
[Thu Aug 30 11:56:42 2012] scheduled_mailings.pl: Can't use an undefined value as a HASH reference at scheduled_mailings.pl line 774.
[Thu Aug 30 11:56:59 2012] scheduled_mailings.pl: 403 Forbidden at ..//DADA/MailingList/Schedules.pm line 1173.
[Thu Aug 30 11:56:59 2012] scheduled_mailings.pl: multipart/mixed with no parts?! Something is screwy.... at ..//DADA/App/FormatMessages.pm line 647.
[Thu Aug 30 13:40:44 2012] scheduled_mailings.pl: Can't use an undefined value as a HASH reference at scheduled_mailings.pl line 774.

_____________

I'm guessing this is the relevant error but I have no idea what it means:
[Thu Aug 30 11:56:59 2012] scheduled_mailings.pl: multipart/mixed with no parts?! Something is screwy.... at ..//DADA/App/FormatMessages.pm line 647.

So strange that it would just randomly start throwing this problem after working for years...
Warm regards,

Mary Ann


On Aug 30, 2012, at 4:21 PM, Justin J wrote:


Is there a chance you can post the source of the HTML produced somewhere, before it gets to Dada Mail? Without that, it'll be hard to debug what's going on.

Also, anything in the error log timestamped at the same time?

--

Justin J: Lead Dadaist.
url:         http://dadamailproject.com
email:    justin@PROTECTED
skype:   leaddadaist




On Aug 30, 2012, at 8:35 AM, Mary Ann Kelley wrote:

Hi,

I have had a mailing set up for years that has suddenly gone haywire. It is a scheduled mailing from a URL. The URL parses the php in the file code to update the content daily. Today, the email I received had only this as the content:


This is a multi-part message in MIME format...

------------=_1346313602-30933-0--

________________

When I view the message source, the headers are all there and the source stops at the above as well. If I test the email by manually pasting in the source code from the URL, it sends fine. Using the "Send a webpage" option is the problem. Any idea what might be causing this? I've tested it on pages that do not contain php (just straight HTML) and get the same result.

Warm regards,

Mary Ann





Post: mailto:dadadev@PROTECTED

Unsubscribe »




--

* Post:              mailto:dadadev@PROTECTED
* Unsubscribe: http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev=

Post: mailto:dadadev@PROTECTED

Unsubscribe »




--

* Post:              mailto:dadadev@PROTECTED
* Unsubscribe: http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev=

Post: mailto:dadadev@PROTECTED

Unsubscribe »

  • This mailing list is a public mailing list - anyone may join or leave, at any time.
  • This mailing list is a group discussion list (unmoderated)
  • Start a new thread, email: dadadev@dadamailproject.com

This is the developer discussion mailing list for Dada Mail.

If you are just looking for support Dada Mail, consult the message boards at:

https://forum.dadamailproject.com

Documentation for Dada Mail:

https://dadamailproject.com/d

Specifically, see the Error FAQ:

https://dadamailproject.com/d/FAQ-errors.pod.html

To post to this list, send a message to:

mailto:dadadev@dadamailproject.com

All subscribers of this list may post to the list itself.

Topics that are welcome:

  • Constructive critiques on the program (I like, "x", but, "y" needs some work - here's an idea on how to make this better...)
  • Bug/Error reports
  • Bug fixes
  • Request For Comments on any changes to the program
  • Help customizing Dada Mail for your own needs
  • Patches
  • Language Translations
  • Support Documentation/Doc editing, FAQ's, etc.
  • Discussion of any changes that you would like to be committed to the next version of Dada Mail -

Dada Mail is on Github:

https://github.com/justingit/dada-mail/

If you would like to fork, branch, send over PRs, open up issues, etc.

Privacy Policy:

This Privacy Policy is for this mailing list, and this mailing list only.

Email addresses collection through this mailing list are used explicitly to work within this email discussion list.

We only collect email addresses through our Closed-Loop Opt-In system.

We don't use your email address for any other purpose.

We won't be sharing your email address with any other entity.

Unsubscription can be done at any time. Please contact us at: justin@dadamailproject.com for any help regarding your subscription, including removal from the mailing list.

All mailing list messages sent from us will include a subscription removal link, which will allow you to remove yourself from this mailing list automatically, and permanently.

All consent to use your email address for any other purpose stated at the time of the mailing list subscription will also be revoked upon mailing list removal.