Mailing List Sending FAQ



Mailing List Sending FAQ

This FAQ specifically tries to answer questions about sending mass mailings using Dada Mail

This FAQ goes through quite a bit of detail on how to use the batch settings and auto pickup features in Dada Mail.

We'll also go over how different, separate mailouts interact with each other and what gets sent when.

Terminology

We'll be throwing a few words around that you may not be familiar with, so here's a list:

  • Mass Mailing

    Sending a message to your mailing list. Usually, this means you're entire mailing list, or at least a part of it.

    This type of mailing is a little different from the other types of mailings Dada Mail does, which include subscription and unsubscription requests, sending archives to individucals, etc. These are known as transaction emails

  • Mailout

    The same thing as a Mass Mailing. These can be used interchangably.

Summary (What to do, without the, "Why")

These changes should be done in your list control panel, Under, Manage List - Sending List Options

Thankfully "out of the box", these are the default settings of Dada Mail.

One last step:

  • Keep the Mailing Monitor Screen Open!

    After you've submitted a mailing list message, keep that screen open until your message has completed!

    Read below on how to set up a cronjob (scheduled task) to monitor your mailing for you.

If you do all these steps, you will be on your way to trouble-free mailing list sending.


Sending Messages in Batches

What's "Batch Sending"

Batch Sending is used primarily to slow down a mass mailing and space out message sending over a longer period of time. It has basically two parts to it:

During a Mass Mailing, Dada Mail will send a specific amount of messages out, wait a specific amount of time and then send another, "batch" of messages.

Batch sending does not work by sending a specific amount of messages during a specific amount of time. Close to that, but not that.

Batch setting is configured in the list's control panel: Log into your list and go to: Mail Sending - Mass Mailing Preferences. You should see the settings to change. Pretty simple.

Should I send my mailing in batches? What are the benefits?

Yes. Yes you should.

One benefit of batch sending is that you'll receive a message in your Dada Mail usage logs when a batch is completed, with some verbose information on the status of your list mailing.

It sort of looks like this (broken up into a few lines:

 [Fri Oct 13 00:44:41 2006]     listshortname   127.0.0.1       Batch Successfully Completed
 Subject:Davka Message, Start Time: Mailing Started: 10/13/2006 00:44:11
 process_has_stalled: 0 should_be_restarted: 0 first_access: 1160721851
 percent_done: 0 is_batch_locked: 1 last_access: 1160721881 id: 
 <20061013004408.77295734@example.com> total_sending_out_num: 1014 type: list   
 total_sent_out: 4

Handy.

Why would I want to throttle down the speed of a mailing by using batching?

Good question! If you think about it, why wouldn't you want to send as many messages as fast as possible?

Sounds like a good idea, but consider these extremely important points:

  • Mail Server Restrictions?

    Check up with your host -

    I almost guarantee you that if you're on a shared hosting account - which is probably the vast majority of people who have personal sites, or small business sites, you have a limit imposed by the mail server on how many messages you can send out over a period of time. Guarantee you.

    For el-cheapo hosting accounts - ones that cost between $10 - $60 a month, you're probably limited to 500 messages/hour. Don't take this as truth though, check with your own hosting company and get the exact amount that they say is, "OK". Set your batch settings a little lower than what the limit is - since you may receive sub/unsub requests, etc.

    So, if your hosting account limits you to 500/messages an hour, try setting your batches to something like: 1 message, every 8 seconds. In fact, that may be the factory default setting.

    A few tell-tale signs your going over you're sending limit:

    • Some of your subscribers aren't receiving your message

      Like, the last half of your list, something like that. I'm a little wary of reports from people who pick at random subscribers and ask them if they have received the message - there are other reasons why someone wouldn't have gotten a message - but I digress...

    • You receive a whole lot of bounced messages

      Like, the last half of your entire list bounce back and you know, know they'll valid email addresses, with mailboxes attached to them.

      Sometimes the bounce will say something like:

       user@example.com
       unrouteable mail domain "example.com"

      Real helpful, huh? Clearly you can tell, you've been going over your mail sending quota. Clearly.

      That's one of the dummy things about mail servers and hosting companies - many don't have a good way to tell you when you go over a mail sending quota. Email your hosting company - tell them you'd like an easy way to know when you do. It may help.

  • Be Kind to Your Web Server

    Sending out a lot of email messages is CPU and memory resource intensive.

    Again, if you're using a shared hosting account, you're going to run into problems just blazing a few million messages - so don't. Usually, you'll have imposed limits on how much CPU Time you're taking up and also actual server memory. If you go over these limits, you're likely to get an angry (but calmly angry) email from your hosting account support team, telling you to knock it off.

    Consider:

    Let's say your email message - what you're sending, is 40k is size. You have 25,000 subscribers (a big list, by some scales). In that total mailing, you're moving almost a GIG of information out of the server. That's quite the load. If you try to do this without using batching - meaning, in the course of a few seconds/minutes - first off, unless you have some fat pipes, it's not going to happen and secondly if you do somehow make it out, that's a whole load the server has to take on.

    Now, 25,000 messages at once will flood your mail server. It's going to hate you and probably tell you, in its own little way, to stop - it won't want to process that many messages at once. Sometimes this is by silently dropping the messages you send to it, into the Great Ether of the Intarweb. A big culprit in the, "My messages aren't being delivered" puzzle. It drops messages, because it's better to do this than crash.

    And also consider that every mailing list message mailing that you do, will have bounces - for whatever reason. Sending a whole lot of messages out, will give you many messages coming in - from different places - basically, you'll give yourself a Denial of Service attack. Don't do that.

    So have a heart, batch at something that's realistic. If you're on a shared server, realistic may mean what you're hosting companies impose. Follow their restrictions - don't ever go over. If it's too strict, tell them - ask them what can be done. Use your right as a consumer. Find a better host that'll work and tell your current host about it. Hosting companies are nothing if competitive against each other. By again, be realisitic: For a few bucks a month for hosting, you are not going to get the moon and stars - hopefully, you'll get something usable. If you need more, consider upgrading to a plan that allows more resources at your disposal. If you have a large enough list, you may have a big enough business to require it.

Why are the batch settings so limited?

They're not.

Currently, without hacking about, you can send 1 to 180 messages and then wait 1 to 180 seconds. Sounds like not a lot of wiggle room, but it should be plenty -

Consider that one message, every 180 seconds (the slowest setting you can have) is 30 messages an hour. That's slooooow.

Sending 180 messages, every second (the fastest setting you can have) is a rate of 10,800 messages a minute - 648,000 messages an hour. If your mail server can even support that, hats off - unless it's specifically tuned to send copius amounts of mail, there's no way you'll be able to get up to that speed.

Honestly.

The fastest you're going to get, in pretty good conditions is 25,000 messages an hour(?) (untested, thus the question mark)

Hope I helped to dispell that myth.


Auto Pickup Options

The, Auto-Pickup Dropped List Message Mailings is going to be your best friend.

Once enabled, this feature will make sure the process of sending out a mailing will succeed to completion.

You can configure the feature in the list control panel. Log into your list and under Mail Sending - Mass Mailing Preferences, you should see the settings to change.

Why do you need to use the Auto Pickup feature?

Unlike traditional Mailing List Managers (Mailman, Majordomo, EZelm), Dada Mail does not run as a daemon - (and this is simplifying things) which is a program that is always running, just waiting to do something.

Daemons usually aren't allowed to run in shared server environments, so that's why Dada Mail isn't one. Shared server environments also usually have a set limit on the amount of memory a program process can consume, how large the process itself can be, how many files it can have open and most importantly for us, how long a process can run for.

So, that's the kind of environment we're working under. Quite restrictive.

This poses some problems, for this simple fact: Sending out a message to a lot of people takes a lot of time. If you've read the above, you've realized that you want to send your mailing list sendouts in batches. No question.

If your subscriber list is long enough to go over the limitation of how long a process can run for - which is pretty easy, your mailing is going to fail, mid way through somewhere, and this is bad.... if! you do not have a way to reload a list that has stopped at a specific part of your list, automatically. Which is what the auto-pickup feature does. So - no fretting.

Should you use it? Yes. Please, yes. It's going to help you make a successful mailing list and it'll allow you to send a mailout slower, which, as we went over in the batch sending FAQ's is actually a good thing.

How do you use the Auto Pickup Options?

You can configure the feature in the list control panel. Log into your list and under Mail Sending - Mass Mailing Preferences, you should see the settings to change.

You have two settings to which to play with:

What is the Sending Monitor Screen and why is it so important for the Auto Pickup feature?

The Sending Monitor Screen will tell you how far along your mailout has gone.

In the list control panel, when you send out a message, by pressing the big green, Submit Mailing List Message button on the, Send a Message screen (and friends), the screen will refresh and you'll see another screen, stating your mailing is on its way! and give you another big green button which is labeled, Monitor Your Mailing!.

Press that button.

The screen will refresh again to the Sending Monitor Screen

Keep the Sending Monitor Screen open until your mailing has finished to make sure your mailout goes to completion.

The screen should refresh every few seconds to update the status of your mailing. During a refresh, if Dada Mail sees a mailing has been dropped, it will automatically pick up the lost mailing at the exact spot it was dropped. And that's how the auto-pickup feature works.

If you are not on the Sending Monitor screen, the Auto-Pickup feature will not work.

If you are not on the Sending Monitor screen, the Auto-Pickup feature will not work.

If you are not on the Sending Monitor screen, the Auto-Pickup feature will not work.

During a mailout, is it safe to navigate away from the Sending Monitor screen?

Yes! Absolutely.

The Sending Monitor screen does not control a mailout - it only monitors and reports what's going on. If it finds a mailout has dropped, it will initiate the auto-pickup function and restart your mailing.

But! If a mailing is dropped, it won't be restarted, until you go back to this Sending Monitor Screen, so it is a good idea to keep it open.

But, if you do navigate away by mistake, it's not the end of the world, just navigate back.

How do you get back to the Sending Monitor screen if you navigate away?

Quite easily:

Log into your list via the list control panel. Under Send a Message, click on, Monitor Your Mailings. This screen will give you a list of the mailouts currently active. Click on the subject of the mailout you'd like to monitor. And you're back.

Is there an alternative to auto-picking up mailouts, rather than the Sending Monitor screen?

Indeed there is,

The, Mailing Monitor Plugin

See the Mailing Monitor Plugin Documentation:

http://dadamailproject.com/d/mailing_monitor.cgi.html

Which can be run via cronjob. If this plugin is installed and running correctly, you won't have to use the Monitor Your Mailings screen at all.

If you do use the, Restart Mailings After Each Batch, this plugin isn't really going to work very well, since you're probably going to run it via cron much less than is realistically needed for your mailing to go out in good time.

Setting, $MONITOR_MAILOUTS_AFTER_EVERY_EXECUTION to, 1

Another option, is to set the Config.pm variable, $MONITOR_MAILOUTS_AFTER_EVERY_EXECUTION to, 1.

When set to, 1 after every time Dada Mail is run via the, mail.cgi script, the mailing monitor will be run, exactly like the Mailing Monitor plugin and monitor your mailings. This is somewhat of an imperfect option for you, but could come in handy if you cannot or do not know how to set up a cronjob.

The problem is, you never are guaranteed when Dada Mail is going to be run, so your mailing monitoring will be somewhat lopsided.

This option is currently experimental.

Dada Bridge

If you're using the Dada Bridge plugin, it also has the same functionality as the Mailing Monitor plugin built in (although, no report will ever be printed out). Because of this, you won't be absolutely required to run both, but if you want to run both, stagger the timing of the cronjobs - if you're running Dada Bridge every 5 minutes, run the Mailing Monitor plugin every 6.

I'm a nerd. Any documentation on the Mailout/Auto Pickup API?

Indeed, see: http://dadamailproject.com/support/documentation-4_9_1/Mail_MailOut.pm.html

in Dada Mail itself, see the Mailing Monitor plugin noted above, or the sending_monitor() subroutine in the mail.cgi file.


Multiple Mailouts at the Same Time

Multiple Mailouts can go on at one time.

By default (conservatively) Dada Mail only allows you to have one mailout at one time. You can change this in your config using the variable, $MAILOUT_AT_ONCE_LIMIT.

This is to allow you to have a good idea on how to easily keep below any server restrictions on how much email you can send in a specified period of time.

For example, if you're allowed to send 500 messages and hour, and you have your batch settings set to send one message every 8 seconds, that's approx. 450 messages you'll be sending each hour. If you're only allowing one mailout to go out at one time, you'll be sending approx. 450 messages each hour - pretty easy.

There's a bit of wiggle room for other emails that may get sent out, like subscription and unsubscription requests.

Any mailouts that have been submitted to Dada Mail that are over the limit set in $MAILOUT_AT_ONCE_LIMIT will be queued. They'll just wait in line until the number of mailouts is less than the limit.

One exception to this rule is sending out test mass mailings. Test mailings will go through to completion, without having to wait in the queue.

The order at which awaiting mailouts are sent is usually chronological, first in, first out. If you submit a mailout that's submitted at 10:00am and submit another mailout at 11:00am, the mailout you submitted at 10:00 am will be sent out first.

There is one way to change this queue and that's by pausing a mailout.

Pausing a mailout will basically move that mailout to the bottom of the queue and it won't ever be reloaded to be sent until after its manually unpaused.

If you do unpause a mailout, it will jump back in line where it once was. For example, if you have three mailouts, one that is submitted at 10:00am and one at 11:00am and another at 12:00pm and you decide to pause the 10:00am one, it will stop sending out and the mailout submitted at 11:00am will start. The 10:00am mailout will be at the bottom of the queue.

If you then unpassed the 10:00am one, it will jump back where it once was, but since the 11:00am mailout is going, it won't restart until either the 11:00 am mailout has unexpectantly stopped, or the 11:00am mailout has finished.

Stale Mailouts

Mailouts can become stale, meaning, they've been inactive in mail sending for a specific period of time and won't automatically restart. This is to stop mailings that, for whatever reason, aren't active not become all of a sudden, active and start sending out a message that may be a little bit old in the news department.

By default, mailouts that haven't sent anything in a day are considered stale. You may change this time in the Config.pm variable, $MAILOUT_STALE_AFTER, which expresses this time in seconds.


Reference: Various Hosting Companies and Email Sending Quotas

If you can help in fleshing this list out, it would be really helpful to me - contact us via the boards: http://dadamailproject.com/support/boards or the mailing list: http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev/ and let us know!

Bluehost

http://bluehost.com

According to http://helpdesk.bluehost.com/index.php/kb/article/000469

Problem

Why do my email messages keep getting returned?

Solution

If your emails keep getting returned with the message "Unrouteable Domain", then you have exceeded your hourly limit for outbound mail.

There is a default limit on all accounts of 50 messages sent per hour. This is a policy to help control spam on the internet and does not apply to your inbound email. You can receive as many emails as will fit in the accounts quota.

If you have a legitemate need for more email capacity on your account please let us know with a detailed explaination of why and we will gladly accomodate your needs.

Take note of that last response - it looks like they basically want you to let them know what you're doing. That's pretty responsible of them.

Also take note of this FAQ: http://helpdesk.bluehost.com/kb/index.php

Which states that you won't want to use SMTP sending on Bluehost.

Some notes from a BlueHost User:

  • Hourly Limit

    BlueHost's default limit is a puny 50/hour, in the name of spam prevention. But they will happily lift that to a maximum of 500/hour if you call their (excellent, toll-free, 24/7) support line and present even a shred of evidence that your need is legitimate.

  • Process Time Limit

    But increasing your hourly limit and configuring batch sending will still not get you going on BlueHost. Their servers are configured to kill the kind of processes that Dada spawns after about one minute.

Where does that leave you? Two options, I think:

  • Dedicated IP Address

    If you upgrade your BlueHost service to a dedicated IP address (at the cost of $30 per year), they lift the process time limit. Before the advent of the auto pickup feature, this would have been the only solution.

  • Use Auto Pickup

    In theory, auto pickup should do the trick, since any polite-sized batch should be completed in under a minute -- long before BlueHost kills the sending process. But I haven't tested this approach on BlueHost, so I can't swear that it'll work.

    
    =back

Cyboretum

http://www.cyboretum.com

Currently, there is no limit to outbound email sending, within reason. Contact them directly if you have any q's.

Dreamhost

http://dreamhost.com

According to http://wiki.dreamhost.com/index.php/Smtp-quota

DreamHost enforces outbound mail quotas. While trying to send mail, if you receive an error message which mentions "Quota Exceeded," you have surpassed your mail sending quota. If you hit this quota 2 or more times, your sending ability will be disabled until you contact support. In your support request, please include your email username and email address.

The outbound quota is 100 recipients / hour.

Please note this is NOT 100 emails per hour. A single email to 100 people in the CC/BCC fields would consume your entire quota.

For this host, you would definetely want to set batching, at a rate of 1 email, every 37 seconds or so. This limit is pretty strict.


Debugging a Failed Mailing List Sendout

Logs

Set up the various logs available in Dada Mail. We'll be going through the various logs specific for list sending.


Loading