Re: Template Optimizations/Enhancements
From:
"MM John" <john@PROTECTED>
In-Reply-To:
Re: Template Optimizations/Enhancements
Date:
November 11th 2011
The
backwards compatibility issue is easily solved by dupllcating, renaming
and archiving all the old files including the database. Then they
wouldn't be deleted or overwritten and could be reinstated. Should have
thought of that when I wrote it.
I don't understand the below sentence
Yes we are saying the same thing below, just from a different point of view. The result is the same.
Since speed is a such high priority for me, I don't see that I'll invest the time to test it. 11 hours for one mailing would not work for me, not when I can send it in one hour with a perfectly workable program.
But I'll keep lurking on this list, maybe something will change.
John
-------------------------------------------------
John Collins
In The Calendar.com
c949 689 7070
john@PROTECTED
http://www.inthecalendar.com
I don't understand the below sentence
It's pretty hard to know what your deliverability is actually with Dada Mail 2x. Do you have any idea (meaning: data)?
Yes we are saying the same thing below, just from a different point of view. The result is the same.
I follow both our maths and we're talking about different things. I'm saying there's a savings of ~5.5 hours for a 25k list, you're saying that a 25k list would still take 11.1 hrs. 24secs * 2500 / 3600 secs/hr = 16.67 hrs, right? So the time savings is still 5.5 hours.
Since speed is a such high priority for me, I don't see that I'll invest the time to test it. 11 hours for one mailing would not work for me, not when I can send it in one hour with a perfectly workable program.
But I'll keep lurking on this list, maybe something will change.
John
Justin J
November 11, 2011 2:26 PMOn Nov 11, 2011, at 7:38 AM, MM John wrote:Before testing it, I would have to be sure that I would not be changing something that would prevent me from going back to the earlier version. (if it ain't broke, etc)I work really hard for forward-compatibility - being able to work with old lists in with a new Dada Mail, but it's not too too realistic for me to be able to do the opposite. That's a lot to ask for.The improvement that has me the most interested in upgrading has not been implemented yet. It's the changes suggested to Bounce handler in which soft bounces are cleared if mail to a recipient is accepted on subsequent mailings. There are other nice features, but I've gotten by without them so far.You can help sponsor that, as well :)So I need the speed, they should be received by the subscribers at close to the same time of day. I can send the 25,000 in about an hour with the old version so I can get all three out in 3 - 4 hours. Close enough.In all honesty, it's probably not going to happen. Dada Mail 4 does a lot more than Dada Mail 2. Speed isn't a huge priority with me, although I don't mind working on efficiency. The vast majority of users of Dada Mail have limits of 500 messages they are allowed to send, per hour. If I can get out 2 or 3 fully templated out messages a second (10,000 messages sent an hour), I'm pretty darn happy. It's pretty hard to know what your deliverability is actually with Dada Mail 2x. Do you have any idea (meaning: data)?By your tests as indicated here http://dadamailproject.com/images/dev/html_template_pro_optimizations.jpg You state that you are saving 8 secs for 10 messages, extrapolating to a savings of 5.5hrs for a 25000 list. Maybe I'm missing something, but looking at the other column, it appears to indicate that it took 16secs to send ten. Therefore, 16secs * 2500 / 3600 secs/hr = 11.1 hrs. Ooops.I follow both our maths and we're talking about different things. I'm saying there's a savings of ~5.5 hours for a 25k list, you're saying that a 25k list would still take 11.1 hrs. 24secs * 2500 / 3600 secs/hr = 16.67 hrs, right? So the time savings is still 5.5 hours. But again, YMMV. These are artificial tests with a code profiler (which will also slow things down, probably), not sending anything out, just writing them email message to a file.JOn Nov 11, 2011, at 7:38 AM, MM John wrote:Before testing it, I would have to be sure that I would not be changing something that would prevent me from going back to the earlier version. (if it ain't broke, etc) The improvement that has me the most interested in upgrading has not been implemented yet. It's the changes suggested to Bounce handler in which soft bounces are cleared if mail to a recipient is accepted on subsequent mailings. There are other nice features, but I've gotten by without them so far. But I'd even forego that feature if I had to give up a lot of speed to get it. Your sample of an 180kb message is about my norm for my weekly mailer. It's mostly html formatted text with a few small logos that reside on my server. In addition to the large one, I have sometimes sent two more smaller ones on the same day. So I need the speed, they should be received by the subscribers at close to the same time of day. I can send the 25,000 in about an hour with the old version so I can get all three out in 3 - 4 hours. Close enough. By your tests as indicated here http://dadamailproject.com/images/dev/html_template_pro_optimizations.jpg You state that you are saving 8 secs for 10 messages, extrapolating to a savings of 5.5hrs for a 25000 list. Maybe I'm missing something, but looking at the other column, it appears to indicate that it took 16secs to send ten. Therefore, 16secs * 2500 / 3600 secs/hr = 11.1 hrs. Ooops. Whereas I can send ten in 1.05 secs based on the 7 per second rate I am getting. I use a Mac so I *think* I can ssh to the server using Terminal, but I've never done it before so I'd need some direction. I'm intrigued, but I need to know more. JohnJustin J November 9, 2011 8:40 PM On Nov 9, 2011, at 6:31 PM, Justin J wrote:But take in account how fast the templating engine is now, and how fast my tests say it could be. The savings just creating the email message would be a few minutes to... I dunno, an hour? per *25,000*. I don't know if the template engine is really a large bottle neck.Actually, depending on what I use to look into this, I get different answers. Using a tool called Devel::NYTProf (which is pretty incredible) it actually gives a dramatic speed improvement: http://dadamailproject.com/images/dev/html_template_pro_optimizations.jpg The top table is using HTML::Template, the bottom table is using HTML::Template::Pro and I tested just sending out a message to 10 people. The message is around 180 KB (so a very large message). It's showing a speedup of over 8 seconds. In that case, speeding up just that little bit for 25k subscribers would yield around 5 1/2 hours faster to send, if my math is correct. That's not bad, for such an easy optimization. Obv., YMMV depending a lot on the type of message you're sending.Justin J November 9, 2011 4:25 PM Hello Everyone, I've been toying (again?) with changing the template engine in Dada Mail (but keeping the syntax the same) from HTML::Template: http://search.cpan.org/dist/HTML-Template/ To HTML::Template::Pro http://search.cpan.org/dist/HTML-Template-Pro/ HTML::Template::Pro promises speedups between 10 and 25x over HTML::Template, which is pretty nice. The downside is that I won't be able to ship Dada Mail with HTML::Template::Pro enabled by default, as HTML::Template;:Pro requires you to compile the module. I can still make it an option for those looking for a speed/memory use improvement. Doing a quick test, I'm able to see around a 10x speed improvement over what is currently being used when making HTML screens in Dada Mail, and a 2.7x speedup for parsing templates in email messages. That seems a fairly good improvement, but the initial speeds of HTML::Template aren't super slow - for HTML screens, we're talking about .025 seconds to render the screen with HTML::Template, and .0025 seconds to render the screen with HTML::Template::Pro. I'm not sure of memory usage, but I'm guessing that memory using is lower with HTML::Template::Pro. You can follow this part of the project here: https://github.com/justingit/dada-mail/tree/features-html_template_pro And actually, all the different sub projects I have happening here: https://github.com/justingit/dada-mail/branches Looks as if v4.9.0 is going to be a big release.------------------------------------------------- John Collins In The Calendar.com c949 689 7070 john@PROTECTED http://www.inthecalendar.com Post: mailto:dadadev@PROTECTED Unsubscribe: http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev/ List Information: http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev Archive: http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadev Developer Info: http://dev.dadamailproject.com Mailing List Powered by Dada Mail-- * Post: mailto:dadadev@PROTECTED * Unsubscribe: http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev * List Information: http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev/ * Archive: http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadev Developer Info: http://dev.dadamailproject.com Mailing List Powered by Dada Mail http://dadamailproject.com/cgi-bin/dada/mail.cgi/what_is_dada_mail/Justin J November 9, 2011 5:31 PMI reverted to 2.11.16 quite some time ago because it could send out 25,000 emails about 180kb in size in about an hour, which is about 7 per second. I needed the speed because sometimes I send several in a day. Subsequent versions took much longer to send. I'd like to have the advanced features of the newer versions and would update in a heartbeat if the speeds were comparable.But take in account how fast the templating engine is now, and how fast my tests say it could be. The savings just creating the email message would be a few minutes to... I dunno, an hour? per *25,000*. I don't know if the template engine is really a large bottle neck. I sincerely think your mad :) to be using any version of 2.x of Dada Mail. Your missing out on way too many things that have been created that would work in your favor since that ancient release. I'd strongly encourage you to try out the version of Dada Mail with this feature enabled and I'd really really encourage you to sponsor the feature, to get it out the door faster :) Remember, we'll need access to the command line (via ssh) and a compiler to get HTML::Template::Pro installed.Justin J November 9, 2011 4:25 PM Hello Everyone, I've been toying (again?) with changing the template engine in Dada Mail (but keeping the syntax the same) from HTML::Template: http://search.cpan.org/dist/HTML-Template/ To HTML::Template::Pro http://search.cpan.org/dist/HTML-Template-Pro/ HTML::Template::Pro promises speedups between 10 and 25x over HTML::Template, which is pretty nice. The downside is that I won't be able to ship Dada Mail with HTML::Template::Pro enabled by default, as HTML::Template;:Pro requires you to compile the module. I can still make it an option for those looking for a speed/memory use improvement. Doing a quick test, I'm able to see around a 10x speed improvement over what is currently being used when making HTML screens in Dada Mail, and a 2.7x speedup for parsing templates in email messages. That seems a fairly good improvement, but the initial speeds of HTML::Template aren't super slow - for HTML screens, we're talking about .025 seconds to render the screen with HTML::Template, and .0025 seconds to render the screen with HTML::Template::Pro. I'm not sure of memory usage, but I'm guessing that memory using is lower with HTML::Template::Pro. You can follow this part of the project here: https://github.com/justingit/dada-mail/tree/features-html_template_pro And actually, all the different sub projects I have happening here: https://github.com/justingit/dada-mail/branches Looks as if v4.9.0 is going to be a big release.------------------------------------------------- John Collins In The Calendar.com c949 689 7070 john@PROTECTED http://www.inthecalendar.com Post: mailto:dadadev@PROTECTED Unsubscribe: http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev/ List Information: http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev Archive: http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadev Developer Info: http://dev.dadamailproject.com Mailing List Powered by Dada MailMM John November 9, 2011 5:06 PM You got my attention on this one. I reverted to 2.11.16 quite some time ago because it could send out 25,000 emails about 180kb in size in about an hour, which is about 7 per second. I needed the speed because sometimes I send several in a day. Subsequent versions took much longer to send. I'd like to have the advanced features of the newer versions and would update in a heartbeat if the speeds were comparable. FYI: using sendmail, no batches, no gaps; just one big gulp. It's faster than smtp sending for this particular version. John Collins Post: mailto:dadadev@PROTECTED Unsubscribe: http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev/ List Information: http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev Archive: http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadev Developer Info: http://dev.dadamailproject.com Mailing List Powered by Dada Mail Mailing List Powered by Dada MailJustin J November 9, 2011 4:25 PM Hello Everyone, I've been toying (again?) with changing the template engine in Dada Mail (but keeping the syntax the same) from HTML::Template: http://search.cpan.org/dist/HTML-Template/ To HTML::Template::Pro http://search.cpan.org/dist/HTML-Template-Pro/ HTML::Template::Pro promises speedups between 10 and 25x over HTML::Template, which is pretty nice. The downside is that I won't be able to ship Dada Mail with HTML::Template::Pro enabled by default, as HTML::Template;:Pro requires you to compile the module. I can still make it an option for those looking for a speed/memory use improvement. Doing a quick test, I'm able to see around a 10x speed improvement over what is currently being used when making HTML screens in Dada Mail, and a 2.7x speedup for parsing templates in email messages. That seems a fairly good improvement, but the initial speeds of HTML::Template aren't super slow - for HTML screens, we're talking about .025 seconds to render the screen with HTML::Template, and .0025 seconds to render the screen with HTML::Template::Pro. I'm not sure of memory usage, but I'm guessing that memory using is lower with HTML::Template::Pro. You can follow this part of the project here: https://github.com/justingit/dada-mail/tree/features-html_template_pro And actually, all the different sub projects I have happening here: https://github.com/justingit/dada-mail/branches Looks as if v4.9.0 is going to be a big release.------------------------------------------------- John Collins In The Calendar.com c949 689 7070 john@PROTECTED http://www.inthecalendar.com ------------------------------------------------- John Collins In The Calendar.com c949 689 7070 john@PROTECTED http://www.inthecalendar.com Post: mailto:dadadev@PROTECTED Unsubscribe: http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev/ List Information: http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev Archive: http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadev Developer Info: http://dev.dadamailproject.com
MM John
November 11, 2011 6:37 AM
Re: Re: Template Optimizations/Enhancements Before testing it, I would have to be sure that I would not be changing something that would prevent me from going back to the earlier version. (if it ain't broke, etc)
The improvement that has me the most interested in upgrading has not been implemented yet. It's the changes suggested to Bounce handler in which soft bounces are cleared if mail to a recipient is accepted on subsequent mailings. There are other nice features, but I've gotten by without them so far.
But I'd even forego that feature if I had to give up a lot of speed to get it. Your sample of an 180kb message is about my norm for my weekly mailer. It's mostly html formatted text with a few small logos that reside on my server. In addition to the large one, I have sometimes sent two more smaller ones on the same day. So I need the speed, they should be received by the subscribers at close to the same time of day. I can send the 25,000 in about an hour with the old version so I can get all three out in 3 - 4 hours. Close enough.
By your tests as indicated here
http://dadamailproject.com/images/dev/html_template_pro_optimizations.jpg
You state that you are saving 8 secs for 10 messages, extrapolating to a savings of 5.5hrs for a 25000 list.
Maybe I'm missing something, but looking at the other column, it appears to indicate that it took 16secs to send ten. Therefore, 16secs * 2500 / 3600 secs/hr = 11.1 hrs. Ooops.
Whereas I can send ten in 1.05 secs based on the 7 per second rate I am getting.
I use a Mac so I *think* I can ssh to the server using Terminal, but I've never done it before so I'd need some direction.
I'm intrigued, but I need to know more.
JohnPost:
mailto:dadadev@PROTECTEDUnsubscribe:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev/List Information:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadevArchive:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadevDeveloper Info:
http://dev.dadamailproject.comMailing List Powered by Dada Mail
Justin J
November 9, 2011 8:40 PMOn Nov 9, 2011, at 6:31 PM, Justin J wrote:But take in account how fast the templating engine is now, and how fast my tests say it could be. The savings just creating the email message would be a few minutes to... I dunno, an hour? per *25,000*. I don't know if the template engine is really a large bottle neck.Actually, depending on what I use to look into this, I get different answers. Using a tool called Devel::NYTProf (which is pretty incredible) it actually gives a dramatic speed improvement: http://dadamailproject.com/images/dev/html_template_pro_optimizations.jpg The top table is using HTML::Template, the bottom table is using HTML::Template::Pro and I tested just sending out a message to 10 people. The message is around 180 KB (so a very large message). It's showing a speedup of over 8 seconds. In that case, speeding up just that little bit for 25k subscribers would yield around 5 1/2 hours faster to send, if my math is correct. That's not bad, for such an easy optimization. Obv., YMMV depending a lot on the type of message you're sending.Justin J November 9, 2011 4:25 PM Hello Everyone, I've been toying (again?) with changing the template engine in Dada Mail (but keeping the syntax the same) from HTML::Template: http://search.cpan.org/dist/HTML-Template/ To HTML::Template::Pro http://search.cpan.org/dist/HTML-Template-Pro/ HTML::Template::Pro promises speedups between 10 and 25x over HTML::Template, which is pretty nice. The downside is that I won't be able to ship Dada Mail with HTML::Template::Pro enabled by default, as HTML::Template;:Pro requires you to compile the module. I can still make it an option for those looking for a speed/memory use improvement. Doing a quick test, I'm able to see around a 10x speed improvement over what is currently being used when making HTML screens in Dada Mail, and a 2.7x speedup for parsing templates in email messages. That seems a fairly good improvement, but the initial speeds of HTML::Template aren't super slow - for HTML screens, we're talking about .025 seconds to render the screen with HTML::Template, and .0025 seconds to render the screen with HTML::Template::Pro. I'm not sure of memory usage, but I'm guessing that memory using is lower with HTML::Template::Pro. You can follow this part of the project here: https://github.com/justingit/dada-mail/tree/features-html_template_pro And actually, all the different sub projects I have happening here: https://github.com/justingit/dada-mail/branches Looks as if v4.9.0 is going to be a big release.------------------------------------------------- John Collins In The Calendar.com c949 689 7070 john@PROTECTED http://www.inthecalendar.com Post: mailto:dadadev@PROTECTED Unsubscribe: http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev/ List Information: http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev Archive: http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadev Developer Info: http://dev.dadamailproject.com Mailing List Powered by Dada Mail-- * Post: mailto:dadadev@PROTECTED * Unsubscribe: http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev * List Information: http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev/ * Archive: http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadev Developer Info: http://dev.dadamailproject.com Mailing List Powered by Dada Mail http://dadamailproject.com/cgi-bin/dada/mail.cgi/what_is_dada_mail/
Justin J
November 9, 2011 5:31 PMI reverted to 2.11.16 quite some time ago because it could send out 25,000 emails about 180kb in size in about an hour, which is about 7 per second. I needed the speed because sometimes I send several in a day. Subsequent versions took much longer to send. I'd like to have the advanced features of the newer versions and would update in a heartbeat if the speeds were comparable.But take in account how fast the templating engine is now, and how fast my tests say it could be. The savings just creating the email message would be a few minutes to... I dunno, an hour? per *25,000*. I don't know if the template engine is really a large bottle neck. I sincerely think your mad :) to be using any version of 2.x of Dada Mail. Your missing out on way too many things that have been created that would work in your favor since that ancient release. I'd strongly encourage you to try out the version of Dada Mail with this feature enabled and I'd really really encourage you to sponsor the feature, to get it out the door faster :) Remember, we'll need access to the command line (via ssh) and a compiler to get HTML::Template::Pro installed.Justin J November 9, 2011 4:25 PM Hello Everyone, I've been toying (again?) with changing the template engine in Dada Mail (but keeping the syntax the same) from HTML::Template: http://search.cpan.org/dist/HTML-Template/ To HTML::Template::Pro http://search.cpan.org/dist/HTML-Template-Pro/ HTML::Template::Pro promises speedups between 10 and 25x over HTML::Template, which is pretty nice. The downside is that I won't be able to ship Dada Mail with HTML::Template::Pro enabled by default, as HTML::Template;:Pro requires you to compile the module. I can still make it an option for those looking for a speed/memory use improvement. Doing a quick test, I'm able to see around a 10x speed improvement over what is currently being used when making HTML screens in Dada Mail, and a 2.7x speedup for parsing templates in email messages. That seems a fairly good improvement, but the initial speeds of HTML::Template aren't super slow - for HTML screens, we're talking about .025 seconds to render the screen with HTML::Template, and .0025 seconds to render the screen with HTML::Template::Pro. I'm not sure of memory usage, but I'm guessing that memory using is lower with HTML::Template::Pro. You can follow this part of the project here: https://github.com/justingit/dada-mail/tree/features-html_template_pro And actually, all the different sub projects I have happening here: https://github.com/justingit/dada-mail/branches Looks as if v4.9.0 is going to be a big release.------------------------------------------------- John Collins In The Calendar.com c949 689 7070 john@PROTECTED http://www.inthecalendar.com Post: mailto:dadadev@PROTECTED Unsubscribe: http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev/ List Information: http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev Archive: http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadev Developer Info: http://dev.dadamailproject.com Mailing List Powered by Dada Mail
MM John
November 9, 2011 5:06 PM
Re: Template Optimizations/Enhancements You got my attention on this one.
I reverted to 2.11.16 quite some time ago because it could send out 25,000 emails about 180kb in size in about an hour, which is about 7 per second. I needed the speed because sometimes I send several in a day. Subsequent versions took much longer to send. I'd like to have the advanced features of the newer versions and would update in a heartbeat if the speeds were comparable.
FYI: using sendmail, no batches, no gaps; just one big gulp. It's faster than smtp sending for this particular version.
John CollinsPost:
mailto:dadadev@PROTECTEDUnsubscribe:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev/List Information:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadevArchive:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadevDeveloper Info:
http://dev.dadamailproject.com
-------------------------------------------------
John Collins
In The Calendar.com
c949 689 7070
john@PROTECTED
http://www.inthecalendar.com
- This mailing list is a public mailing list - anyone may join or leave, at any time.
- This mailing list is a group discussion list (unmoderated)
-
Start a new thread, email: dadadev@dadamailproject.com
This is the developer discussion mailing list for Dada Mail.
If you are just looking for support Dada Mail, consult the message boards at:
https://forum.dadamailproject.com
Documentation for Dada Mail:
Specifically, see the Error FAQ:
https://dadamailproject.com/d/FAQ-errors.pod.html
To post to this list, send a message to:
mailto:dadadev@dadamailproject.com
All subscribers of this list may post to the list itself.
Topics that are welcome:
- Constructive critiques on the program (I like, "x", but, "y" needs some work - here's an idea on how to make this better...)
- Bug/Error reports
- Bug fixes
- Request For Comments on any changes to the program
- Help customizing Dada Mail for your own needs
- Patches
- Language Translations
- Support Documentation/Doc editing, FAQ's, etc.
- Discussion of any changes that you would like to be committed to the next version of Dada Mail -
Dada Mail is on Github:
https://github.com/justingit/dada-mail/
If you would like to fork, branch, send over PRs, open up issues, etc.
Privacy Policy:
This Privacy Policy is for this mailing list, and this mailing list only.
Email addresses collection through this mailing list are used explicitly to work within this email discussion list.
We only collect email addresses through our Closed-Loop Opt-In system.
We don't use your email address for any other purpose.
We won't be sharing your email address with any other entity.
Unsubscription can be done at any time. Please contact us at: justin@dadamailproject.com for any help regarding your subscription, including removal from the mailing list.
All mailing list messages sent from us will include a subscription removal link, which will allow you to remove yourself from this mailing list automatically, and permanently.
All consent to use your email address for any other purpose stated at the time of the mailing list subscription will also be revoked upon mailing list removal.