---

Justin J November 12, 2011 10:50 PM

During my optimization pass, I found a massive memory leak that happens during a mass mailing of an HTML message. The mass mailing process potentially starts eating up many hundreds of megs of memory, which could potentially start slowing things way, way down, or even get your sending process killed. This underlying problem could be the culprit of a lot of mysterious sending problems with Dada Mail. I've been able to remove the problem with the development stuff I've been doing and was also able to backport the bug, so it'll be available in v4.8.4 of Dada Mail which I can release very soon, instead of everyone having to wait until v4.9.0 of Dada Mail, which isn't very ready for prime time.

Optimization work is going very well, with around 5x performance increase, which is massively huge. Really.

Here's the report from sending a ~212k HTML message to 300 subscribers:

http://dadamailproject.com/images/dev/html_template_pro_optimizations2.jpg

Remember: that the message isn't actually sent, just written to the file, so there's going to be a little discrepancy on real-world performance. This is also being run via a profiler, which may impose it's own performance loss (not sure!). So, it may be more useful to take a birds-eye view of these gains in performance.


The first block of results is of v4.8.3 (basically) - sending 300 messages out takes 510s - ~.59 messages/second.

The second block of results is v4.8.3, with the memory leak fixed - 454s for 300 messages - ~.66 messages/second

The third block is with all the current optimizations made:

* this memory leak fixed,
* using HTML::Template::Pro instead of HTML::Template
* updating the MIME::Entity CPAN modules to the latest version
* various refinement in code

The results of 129s to send the same 300 messages - ~ 2.3 messages/second seems to be a massive improvement to me. I'm kind of out of ideas on how to optimize further without a massive code restructuring, so I'm probably going to stop now, merge these changes with the rest of the dev code and get an alpha out for everyone to play with.

And finally, out of morbid fascination, I whipped up a similar profiling for an old copy of Dada Mail: version 2.10.16. Here's the results:

http://dadamailproject.com/images/dev/mass_sending_profile_2_10_16.jpg

Pretty interesting to say the least :) Speed is more ~8.80 messages/second - almost 4x faster than with all the optimizations I've done to Dada Mail 4.x! I'm a little confused, actually with the report, since the logs actually say it took around 2 or 3 seconds to "send" the entire 300 messages. Anyways, it's much, much faster. John was right.

So, why is 4.x so much slower? 2.x didn't really have a templating system. It's template "tags" was just a simple search and replace. A lot of the advanced features we have now, like UTF-8 encoding, just aren't in v2.x of Dada Mail. The biggest speed culprit though, seems to be the advanced handling of email messages themselves - v4.x treats email messages as an object, 2.x much less so. This goes into the most basic things that Dada Mail 4.x does.