Mailing List Sending FAQ
- Mailing List Sending FAQ
- Sending Messages in Batches
- Auto Pickup Options
- What is the Sending Monitor Screen and why is it so important for the Auto Pickup feature?
- During a Mass Mailing, is it safe to navigate away from the Sending Monitor screen?
- How do you get back to the Sending Monitor screen if you navigate away?
- Is there an alternative to auto-picking up mailouts, rather than the Sending Monitor screen?
- Mass Mailings/Auto Pickup API
- Multiple Mass Mailings at the Same Time
- Reference: Various Hosting Companies and Email Sending Quotas
Mailing List Sending FAQ
This FAQ specifically tries to answer questions about sending mass mailings using Dada Mail
This FAQ goes through quite a bit of detail on how to use the batch settings and auto pickup features in Dada Mail.
We'll also go over how different, separate mailouts interact with each other and what gets sent when.
Terminology
We'll be throwing a few words around that you may not be familiar with, so here's a list:
Mass Mailing
Sending a message to your mailing list. Usually, this means you're entire mailing list, or at least a part of it.
This type of mailing is a little different from the other types of mailings Dada Mail does, which include subscription and unsubscription requests, sending archives to individucals, etc. These are known as transaction emails
Mailout
The same thing as a Mass Mailing. These can be used interchangably. We do prefer, "Mass Mailing"
Summary
These changes should be done in your list control panel, Under, Manage List - Sending List Options
Enable Batching
To start: Set the batch settings like so:
Send: 1 message(s) per batch
and then wait: 8 seconds, before the next batch.
Enable: Auto-Pickup Dropped List Message Mailings
Thankfully "out of the box", these are the default settings of Dada Mail.
One last step:
Keep the Mailing Monitor Screen Open!
After you've submitted a mailing list message, keep that screen open until your message has completed!
Read below on how to set up a cronjob (scheduled task) to monitor your mailing for you.
Sending Messages in Batches
What's "Batch Sending"
Batch Sending is used primarily to slow down a mass mailing and space out message sending over a longer period of time. It has basically two parts to it:
How many messages get sent out at once (how large your batch is)
How long to wait before mailing another batch
During a Mass Mailing, Dada Mail will send a specific amount of messages out, wait a specific amount of time and then send another, "batch" of messages.
Batch sending does not work by sending a specific amount of messages during a specific amount of time. Close to that, but not that.
Batch setting is configured in the list's control panel: Log into your list and go to: Mail Sending - Mass Mailing Preferences.
Should I send my mailing in batches?
Yes.
One benefit of batch sending is that you'll receive a message in your Dada Mail usage logs when a batch is completed, with some verbose information on the status of your list mailing.
It sort of looks like this (broken up into a few lines:
[Fri Oct 13 00:44:41 2006] listshortname 127.0.0.1 Batch Successfully Completed
Subject:Davka Message, Start Time: Mailing Started: 10/13/2006 00:44:11
process_has_stalled: 0 should_be_restarted: 0 first_access: 1160721851
percent_done: 0 is_batch_locked: 1 last_access: 1160721881 id:
<20061013004408.77295734@example.com> total_sending_out_num: 1014 type: list
total_sent_out: 4Why would I want to throttle down the speed of a mailing by using batching?
Consider these extremely important points:
Hourly Email Sending Limitations
Check up with your hosting company:
If you're on a shared hosting account, you have a limit imposed by the mail server on how many messages you can send out over a period of time.
For inexpensive hosting accounts - ones that cost between $5 - $50 a month, you're probably limited to 500 messages/hour. Please check with your own hosting company and get the exact amount that they say is, "OK". Set your batch settings a little lower than what the limit is - since you may receive sub/unsub requests, etc.
So, if your hosting account limits you to 500/messages an hour, try setting your batches to something like: 1 message, every 8 seconds. In fact, that may be the factory default setting.
A few tell-tale signs your going over you're sending limit:
Some of your subscribers aren't receiving your message
Like, the last half of your list, something like that. I'm a little wary of reports from people who pick at random subscribers and ask them if they have received the message - there are other reasons why someone wouldn't have gotten a message - but I digress...
You receive a whole lot of bounced messages
Like, the last half of your entire list bounce back and you know, know they'll valid email addresses, with mailboxes attached to them.
Sometimes the bounce will say something like:
user@example.com unrouteable mail domain "example.com"Real helpful, huh? Clearly you can tell, you've been going over your mail sending quota. Clearly.
That's one of the dummy things about mail servers and hosting companies - many don't have a good way to tell you when you go over a mail sending quota. Email your hosting company - tell them you'd like an easy way to know when you do. It may help.
Be Kind to Your Web Server
Sending out a lot of email messages is CPU and memory resource intensive.
Again, if you're using a shared hosting account, you're going to run into problems just blazing a few million messages - so don't. Usually, you'll have imposed limits on how much CPU Time you're taking up and also actual server memory. If you go over these limits, you're likely to get an angry (but calmly angry) email from your hosting account support team, telling you to knock it off.
Consider:
Let's say your email message - what you're sending, is 40k is size. You have 25,000 subscribers (a big list, by some scales). In that total mailing, you're moving almost a GIG of information out of the server. That's quite the load. If you try to do this without using batching - meaning, in the course of a few seconds/minutes - first off, unless you have some fat pipes, it's not going to happen and secondly if you do somehow make it out, that's a whole load the server has to take on.
Now, 25,000 messages at once will flood your mail server. It's going to hate you and probably tell you, in its own little way, to stop - it won't want to process that many messages at once. Sometimes this is by silently dropping the messages you send to it, into the Great Ether of the Intarweb. A big culprit in the, "My messages aren't being delivered" puzzle. It drops messages, because it's better to do this than crash.
And also consider that every mailing list message mailing that you do, will have bounces - for whatever reason. Sending a whole lot of messages out, will give you many messages coming in - from different places - basically, you'll give yourself a Denial of Service attack. Don't do that.
So have a heart, batch at something that's realistic. If you're on a shared server, realistic may mean what you're hosting companies impose. Follow their restrictions - don't ever go over. If it's too strict, tell them - ask them what can be done. Use your right as a consumer. Find a better host that'll work and tell your current host about it. Hosting companies are nothing if competitive against each other. By again, be realisitic: For a few bucks a month for hosting, you are not going to get the moon and stars - hopefully, you'll get something usable. If you need more, consider upgrading to a plan that allows more resources at your disposal. If you have a large enough list, you may have a big enough business to require it.
Auto Pickup Options
Once enabled, this feature will help make sure the process of sending out a mailing will succeed to completion.
You can configure the feature in your List Control Panel, under Mail Sending - Mass Mailing Preferences.
Unlike traditional Mailing List Managers (Mailman, Majordomo, EZelm), Dada Mail does not run as a daemon - (and this is simplifying things) which is a program that is always running, just waiting to fill out requests.
Shared server environments usually have a set limit on the amount of memory a program process can consume, how large the process itself can be, how many files it can have open and most importantly for us, how long a process can run for. It can be a restrictive environment.
This poses some problems, for this simple fact: Sending out a message to a lot of people takes time.
If your Subscribers list is long enough to go over the limitation of how long a process can run for, your mass mailing is going to fail, midway through, unless we use the Auto Pickup feature.
You have two settings to which to play with:
Auto-Pickup Dropped List Message Mailings
That turns on the feature. Have that checked.
Restart Mailings After Each Batch
This setting takes some explaination and we're going to get a little technical.
What is the Sending Monitor Screen and why is it so important for the Auto Pickup feature?
The Sending Monitor Screen will tell you how far along your mailout has gone.
In the list control panel, when you send out a message, by pressing the big green, Submit Mailing List Message button on the, Send a Message screen (and friends), the screen will refresh and you'll see another screen, stating your mailing is on its way! and give you another big green button which is labeled, Monitor Your Mailing!.
Press that button.
The screen will refresh again to the Sending Monitor Screen
Keep the Sending Monitor Screen open until your mailing has finished to make sure your mailout goes to completion.
The screen should refresh every few seconds to update the status of your mailing. During a refresh, if Dada Mail sees a mailing has been dropped, it will automatically pick up the lost mailing at the exact spot it was dropped. And that's how the auto-pickup feature works.
If you are not on the Sending Monitor screen, the Auto-Pickup feature will not work.
During a Mass Mailing, is it safe to navigate away from the Sending Monitor screen?
Yes! Absolutely.
The Sending Monitor screen does not control a mailout - it only monitors and reports what's going on. If it finds a mailout has dropped, it will initiate the auto-pickup function and restart your mailing.
But! If a mailing is dropped, it won't be restarted, until you go back to this Sending Monitor Screen, so it is a good idea to keep it open.
But, if you do navigate away by mistake, it's not the end of the world, just navigate back.
How do you get back to the Sending Monitor screen if you navigate away?
Log into your list via the list control panel. Under Send a Message, click on, Monitor Your Mailings. This screen will give you a list of the mailouts currently active. Click on the subject of the mailout you'd like to monitor. And you're back.
Is there an alternative to auto-picking up mailouts, rather than the Sending Monitor screen?
Indeed there is,
The, Mailing Monitor Plugin
See the Mailing Monitor Plugin Documentation:
http://dadamailproject.com/d/mailing_monitor.cgi.html
Which can be run via cronjob. If this plugin is installed and running correctly, you won't have to use the Monitor Your Mailings screen at all.
If you do use the, Restart Mailings After Each Batch, this plugin isn't really going to work very well, since you're probably going to run it via cron much less than is realistically needed for your mailing to go out in good time.
Setting, $MONITOR_MAILOUTS_AFTER_EVERY_EXECUTION to, 1
Another option, is to set the Config.pm variable, $MONITOR_MAILOUTS_AFTER_EVERY_EXECUTION to, 1.
When set to, 1 after every time Dada Mail is run via the, mail.cgi script, the mailing monitor will be run, exactly like the Mailing Monitor plugin and monitor your mailings. This is somewhat of an imperfect option for you, but could come in handy if you cannot or do not know how to set up a cronjob.
The problem is, you never are guaranteed when Dada Mail is going to be run, so your mailing monitoring will be somewhat lopsided.
Bridge
If you're using the Bridge plugin, it also has the same functionality as the Mailing Monitor plugin built in.
Mass Mailings/Auto Pickup API
See: http://dadamailproject.com/support/documentation-6_9_0/Mail_MailOut.pm.html
in Dada Mail itself, see the Mailing Monitor plugin noted above, or the sending_monitor() subroutine in the mail.cgi file.
Multiple Mass Mailings at the Same Time
Multiple Mass Mailings can go on at one time.
By default (conservatively) Dada Mail only allows you to have one mailout at one time. You can change this in your config using the variable, $MAILOUT_AT_ONCE_LIMIT.
This is to allow you to have a good idea on how to easily keep below any server restrictions on how much email you can send in a specified period of time.
For example, if you're allowed to send 500 messages and hour, and you have your batch settings set to send one message every 8 seconds, that's approx. 450 messages you'll be sending each hour. If you're only allowing one mailout to go out at one time, you'll be sending approx. 450 messages each hour - pretty easy.
There's a bit of wiggle room for other emails that may get sent out, like subscription and unsubscription requests.
Any mailouts that have been submitted to Dada Mail that are over the limit set in $MAILOUT_AT_ONCE_LIMIT will be queued. They'll just wait in line until the number of mailouts is less than the limit.
One exception to this rule is sending out test mass mailings. Test mailings will go through to completion, without having to wait in the queue.
The order at which awaiting mailouts are sent is usually chronological, first in, first out. If you submit a mailout that's submitted at 10:00am and submit another mailout at 11:00am, the mailout you submitted at 10:00 am will be sent out first.
There is one way to change this queue and that's by pausing a mailout.
Pausing a mailout will basically move that mailout to the bottom of the queue and it won't ever be reloaded to be sent until after its manually unpaused.
If you do unpause a mailout, it will jump back in line where it once was. For example, if you have three mailouts, one that is submitted at 10:00am and one at 11:00am and another at 12:00pm and you decide to pause the 10:00am one, it will stop sending out and the mailout submitted at 11:00am will start. The 10:00am mailout will be at the bottom of the queue.
If you then unpassed the 10:00am one, it will jump back where it once was, but since the 11:00am mailout is going, it won't restart until either the 11:00 am mailout has unexpectantly stopped, or the 11:00am mailout has finished.
Stale Mass Mailings
Mass Mailings can become stale, meaning, they've been inactive in mail sending for a specific period of time and won't automatically restart. This is to stop mailings that, for whatever reason, aren't active not become all of a sudden, active and start sending out a message that may be a little bit old in the news department.
By default, mailouts that haven't sent anything in a day are considered stale. You may change this time in the Config.pm variable, $MAILOUT_STALE_AFTER, which expresses this time in seconds.
Reference: Various Hosting Companies and Email Sending Quotas
If you can help in fleshing this list out, it would be really helpful to me - contact us via the boards: http://dadamailproject.com/support/boards or the mailing list: http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev/ and let us know!
Bluehost
According to http://helpdesk.bluehost.com/index.php/kb/article/000469
Problem
Why do my email messages keep getting returned?
Solution
If your emails keep getting returned with the message "Unrouteable Domain", then you have exceeded your hourly limit for outbound mail.
There is a default limit on all accounts of 50 messages sent per hour. This is a policy to help control spam on the internet and does not apply to your inbound email. You can receive as many emails as will fit in the accounts quota.
If you have a legitemate need for more email capacity on your account please let us know with a detailed explaination of why and we will gladly accomodate your needs.
Take note of that last response - it looks like they basically want you to let them know what you're doing. That's pretty responsible of them.
Also take note of this FAQ: http://helpdesk.bluehost.com/kb/index.php?x=&mod_id=2&root=27&id=325
Which states that you won't want to use SMTP sending on Bluehost.
Some notes from a BlueHost User:
Hourly Limit
BlueHost's default limit can be as low as a puny 50/hour, in the name of spam prevention. But they will happily lift that to a maximum of 500/hour if you call their (excellent, toll-free, 24/7) support line and present even a shred of evidence that your need is legitimate.
Process Time Limit
But increasing your hourly limit and configuring batch sending will still not get you going on BlueHost. Their servers are configured to kill the kind of processes that Dada spawns after about one minute.
Where does that leave you? Two options, I think:
Dedicated IP Address
If you upgrade your BlueHost service to a dedicated IP address (at the cost of $30 per year), they lift the process time limit. Before the advent of the auto pickup feature, this would have been the only solution.
Use Auto Pickup
In theory, auto pickup should do the trick, since any polite-sized batch should be completed in under a minute -- long before BlueHost kills the sending process. But I haven't tested this approach on BlueHost, so I can't swear that it'll work.
Dreamhost
According to http://wiki.dreamhost.com/index.php/Smtp-quota
DreamHost enforces outbound mail quotas. While trying to send mail, if you receive an error message which mentions "Quota Exceeded," you have surpassed your mail sending quota. If you hit this quota 2 or more times, your sending ability will be disabled until you contact support. In your support request, please include your email username and email address.
The outbound quota is 100 recipients / hour.
Please note this is NOT 100 emails per hour. A single email to 100 people in the CC/BCC fields would consume your entire quota.
For this host, you would definetely want to set batching, at a rate of 1 email, every 37 seconds or so. This limit is pretty strict.