dadadev Re: dadadev Re: List Restarting - Progress Report (goofed on the example)

 
From: "John Collins" <john@PROTECTED>
Date: November 4th 2008
You're close-

It sent to the whole list of 19000+ with only server error and that was right as it started. It did stall a couple of times, sometimes for more than a minute, but eventually reloaded and continued on. Sending to the whole list took about 2 hours. (I used to send it in about one hour on 2.10.16 with a flat file DB and no external configs)

So here's what happened...

To assure that the tests were on equal footing, I had Westhost reset the mysql connection to default.

Then I created a short newsletter.  Dada sent the test msg to my own address just fine with no errors, but when I started sending to the list (19000+) the monitor your mailing screen didn't even come up.  Instead I got this:

Program Error (Server Error 500)
More information about this error may be available in the server error log and/or program error log.
cannot do statement! (at: _raw_db_hash) Lost connection to MySQL server during query
 at /DADA/Logging/Clickthrough.pm line 32

One of the admin screens was still open so I was able to click on the Monitor Your Mailings link again and got a different looking monitor screen from the one that normally comes up and then found that the email was still sending.

(this monitor screen is the one where the screen refreshes every 60 secs, not every 10 secs and has a smaller progress bar and no buttons for pausing,stopping--instead it has the words like this with question marks; Paused? Queued? Stale? Broken? See below for screen shot.)  This screen doesn't tell me how many per 60 sec refresh are being sent so I can't judge speed.




Eek, I goofed on that code change, here's the fixed version:


In:
dada/DADA/App/DBIHandle.pm
Find this line:


       $self->{dbh} = DBI->connect("$data_source", $user, $pass) ||  croak("can't connect to db: $!");


And change it to:


        $self->{dbh} = DBI->connect_cached("$data_source", $user, $pass,  {private_via_process => $$}) || croak("can't connect to db: $!");


Happy Hacking,


--
Justin J.


Dada Mail -  Write Once: Distribute Everywhere Software
url: http://dadamailproject.com


Demo:
http://demo.dadamailproject.com


Seen Dada Mail 3?
http://dadamailproject.com/features/3_0/









On Nov 4, 2008, at 2:07 AM, Justin J wrote:

Alright, I've been hitting my head against this problem for a few  days now. Here's a report:

Sunday, I tried to figure out that whole, "InactiveDestroy" lead I  had. It's a nerdy topic, but basically it has to do with Dada Mail  sending out its mass mailings and needing to connect to the SQL  database from the child it forked, as well as the parent, without  either mucking with the other.

I have no idea how this was working before, but it was, just not  correctly. The hunch was, however the way I was doing it, made the  SQL connections somehow persistent. I guess looking  back, that's  sort of a wild idea, but anyways, I think I've solved the problem  The Right Way,

       2219912 3.0.0 - DBI attribute: InactiveDestroy used incorrectly
https://sourceforge.net/tracker2/?func=detail&aid=2219912&group_id=13002&atid=113002

so I added it as a bug report and committed some changes and fixed  that.  That took all Sunday.

Alongside doing all that, I found some pretty glaring bugs in the  mailing mointor code - a lot of it had to do with when you have to  reload (erm, "restart") a mailing - there's a lot of it that was  broken and I know a lot of people reported this, with the weirdest  bug reports that I couldn't put my head around - things like seeing  that Dada Mail was trying to reload/restart the mailing a few dozen  times without success. I figured out - I think, what was happening  with that, so, again, added a new bug report with an isolated problem:

   2219972         3.0.0 - Mailout.pm - associate method broken
   https://sourceforge.net/tracker2/?func=detail&aid=2219972&group_id=13002&atid=113002


and committed some changes to fix that.

Figuring that, I found a few more problems:

     2219959         3.0.0 - no check to see if mailout exists after sleeping
       https://sourceforge.net/tracker2/?func=detail&aid=2219959&group_id=13002&atid=113002

    2219954 3.0.0 - Guts.pm sub available_lists param, -Dont_Die broken
     https://sourceforge.net/tracker2/?func=detail&aid=2219954&group_id=13002&atid=113002

So that's all good and well, but nothing that I really could see  would cause this problem that Westhost is saying that John has -  namely, that there's 200+ mysql connections.

Dada Mail should be pretty good at reusing the same database handle  again and again, so it doesn't have to make so many connections.  Perhaps 2 or 3 for some really complicated stuff, like sending out a  mass mailing, but nothing more,

*Surely* nothing like 200.

It must be a problem with Westhost. Right? Right. I'm infallible.

But,

but to double-check I turned on Dada Mail's Tracing, to tell me how  me all about it's connections to the Database (and made a little  tweaks to the code) to see how many connections, say, a mailing to  one person takes.

Forty-Nine.


Forty-Nine connections to send one mass mailing message. Add one  more subscriber, get another connection.

Incredible.

Not all these are at the same time, but it's pretty clear there's  room for improvement.

And for the last few hours, I've been banging my head on trying to  get Dada Mail's, "Hey, use the same database handle you already  have!" system to work better. And it's really hard to get better.

And then, I gleaned on this:

       http://search.cpan.org/~timb/DBI-1.607/DBI.pm#connect_cached

Which basically is an easy way to reused database connections.

Yippee.

So, if you have Dada Mail 3.0, try this quick fix, and let me know  how it works for you:

In:

   dada/DADA/App/DBIHandle.pm

find this line:

       $self->{dbh} = DBI->connect("$data_source", $user, $pass) ||  croak("can't connect to db: $!");

And change it to:

        $self->{dbh} = DBI->connect_cached("$data_source", $user, $pass,  {private_via_process => $i}) || croak("can't connect to db: $!");

I know that the word, "Snappy" is used to describe perceived changes  in speed in computer apps, but this change surely will make your  Dada Mail a little, snappier.

To be honest, it's almost like a new program. See if it works for  you (and if it doesn't, let me know)

I'll be releasing a new snapshot with these changes soon,


--
Justin J.

Dada Mail -  Write Once: Distribute Everywhere Software
url: http://dadamailproject.com

Demo:
http://demo.dadamailproject.com

Seen Dada Mail 3?
http://dadamailproject.com/features/3_0/







--

Post: <mailto:dadadev@PROTECTED>

Unsubscribe:
<http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev/>

List Information: <http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev>

Archive: <http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadev>


-- 
=============================
John Collins
Meetings and Mixers
PO Box 80461               
Rancho Santa Margarita, CA 92688-0461
949-689-7070
info@PROTECTED
http://www.meetingsandmixers.com
See our new Business Directories at:
http://www.meetingsandmixers.com/directories/index.php

Post:
mailto:[list_settings.discussion_pop_email]

Unsubscribe:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/[list]/

List Information:
[PROGRAM_URL]/list/[list_settings.list]

Archive:
[PROGRAM_URL]/archive/[list_settings.list]

Attachments

  • This mailing list is a public mailing list - anyone may join or leave, at any time.
  • This mailing list is a group discussion list (unmoderated)
  • Start a new thread, email: dadadev@dadamailproject.com

This is the developer discussion mailing list for Dada Mail.

If you are just looking for support Dada Mail, consult the message boards at:

https://forum.dadamailproject.com

Documentation for Dada Mail:

https://dadamailproject.com/d

Specifically, see the Error FAQ:

https://dadamailproject.com/d/FAQ-errors.pod.html

To post to this list, send a message to:

mailto:dadadev@dadamailproject.com

All subscribers of this list may post to the list itself.

Topics that are welcome:

  • Constructive critiques on the program (I like, "x", but, "y" needs some work - here's an idea on how to make this better...)
  • Bug/Error reports
  • Bug fixes
  • Request For Comments on any changes to the program
  • Help customizing Dada Mail for your own needs
  • Patches
  • Language Translations
  • Support Documentation/Doc editing, FAQ's, etc.
  • Discussion of any changes that you would like to be committed to the next version of Dada Mail -

Dada Mail is on Github:

https://github.com/justingit/dada-mail/

If you would like to fork, branch, send over PRs, open up issues, etc.

Privacy Policy:

This Privacy Policy is for this mailing list, and this mailing list only.

Email addresses collection through this mailing list are used explicitly to work within this email discussion list.

We only collect email addresses through our Closed-Loop Opt-In system.

We don't use your email address for any other purpose.

We won't be sharing your email address with any other entity.

Unsubscription can be done at any time. Please contact us at: justin@dadamailproject.com for any help regarding your subscription, including removal from the mailing list.

All mailing list messages sent from us will include a subscription removal link, which will allow you to remove yourself from this mailing list automatically, and permanently.

All consent to use your email address for any other purpose stated at the time of the mailing list subscription will also be revoked upon mailing list removal.