How to add and customise the Sender header

[KiwiBloke]

Hi, I'd like to add the Sender header to the emails being sent out by Dada Mail, as currently no Sender header exists and I find it useful to use (I've always used it with my previous list software, setting it to the list email address while the From header is that of the person who sent the email).

I thought that I could use the %EMAIL_HEADERS section to get Dada to add the Sender header, but this doesn't seem to work. I set the following in the %EMAIL_HEADERS section of the .dada_config file:

Sender => 'listaddress@example.com',

However there was still no Sender header at all in the emails I sent from the list afterwards.

Is there any reason why Dada is not adding the Sender header and setting it to whatever I've specified here?

Also, am I able to use template tags here in the %EMAIL_HEADERS section? Or is there some other way I can set this to the list address (normally <!-- tmpl_var list_settings.discussion_pop_email -->) so that it uses the correct value for multiple lists?

[justin]

What was your prev. list software?

Is there any reason why Dada is not adding the Sender header and setting it to whatever I've specified here?

Without trying it myself, sounds like a bug. I'd like to do some research on just what the Sender: header is for (and what usually sets it - the app that creates the message initially, or the MTA?

Also, am I able to use template tags here in the %EMAIL_HEADERS section? Or is there some other way I can set this to the list address (normally <!-- tmpl_var list_settings.discussion_pop_email -->) so that it uses the correct value for multiple lists?

I think currently for performance reasons, only a few headers are looked at, for templating out:

From
To
Reply-To
Return-Path
Errors-To

If you look at the type of headers in there, it looks like, "Sender" would fit in (it's not a configurable setting, at the moment).

[justin]

I'd like to do some research on just what the Sender: header is for (and what usually sets it - the app that creates the message initially, or the MTA?

So here's what RFC2822 says about the, "Sender" header:

3.6.2. Originator fields

The originator fields of a message consist of the from field, the
sender field (when applicable), and optionally the reply-to field.
The from field consists of the field name "From" and a
comma-separated list of one or more mailbox specifications. If the
from field contains more than one mailbox specification in the
mailbox-list, then the sender field, containing the field name
"Sender" and a single mailbox specification, MUST appear in the
message. In either case, an optional reply-to field MAY also be
included, which contains the field name "Reply-To" and a
comma-separated list of one or more addresses
.

If Dada Mail isn't setting it, it's a pretty big bug. I'll add an issue about this.

[justin]

Actually, it goes on:

If the originator of the message can be indicated
by a single mailbox and the author and transmitter are identical, the
"Sender:" field SHOULD NOT be used. Otherwise, both fields SHOULD
appear.

So, it may not be (or SHOULD NOT) as it's written, used, since Dada Mail does keep the From: header of the original sender. I will check out as to why you can't set your own default headers, though, so you can do whatever you'd like easy and at your discretion.

[justin]

...and at least in my tests, I can set my own, "Sender" header. To make sure nothing down the line is stripping it out, I usually set the $MAIL_SETTINGS and $MASS_MAIL_SETTINGS variables like this:

$MAIL_SETTINGS = '>>' . $TMP . '/mail.txt';
$MASS_MAIL_SETTINGS = '>>' . $TMP . '/mail.txt';


and just open up the files. Are you tweaking the, %EMAIL_HEADERS hash the right way?

http://dadamailproject.com/d/Config.pm. ... umentation

[KiwiBloke]

Okay, I think I've gotten this one sorted now. I changed the $MAIL_SETTINGS variable to point to a .txt file as you described, and the Sender header was there in the .txt file. In the end I've narrowed the culprit down to sendmail. If I set the Sending Preferences to sendmail, then the Sender header is gone when I receive the email in my Inbox. But if I set the Sending Preferences to use an SMTP server, then the Sender header is there, and is correctly set based on what I've put in the %EMAIL_HEADERS section.

(BTW, it looks like setting the $MAIL_SETTINGS variable to point to a .txt file only works when the Sending Preferences page is set to use sendmail rather than SMTP. If I set it to SMTP, but also set the $MAIL_SETTINGS variable to point to a .txt file, the .txt file is not used and the email still gets sent out via SMTP. Is this by design, or a bug?)

Anyway, I guess there must be something odd going on with sendmail on the web host that I'm currently using - seems kinda strange to me, but who knows. I'll stick with using the SMTP server for now anyway.

In answer to some of the other questions and comments you've written above.....

What was your prev. list software?

I've been using MDaemon, a really great commercial Windows mail server which includes mailing list functionality (I'm also one of their official beta testers, which might not surprise you!)

I'd like to do some research on just what the Sender: header is for (and what usually sets it - the app that creates the message initially, or the MTA?

The Sender header can be set by either the MTA or the MUA (mail client). Some clients such as MS Outlook will generate the Sender header when used in an Exchange environment, when one user has been given "send on behalf of" delegate permissions for another user's mailbox on the Exchange server (eg. the secretary having permission to send mail on behalf of the boss). In that case, the From field would be that of the secretary, and the Sender field (meaning "sent on behalf of") would be that of the boss.

Furthermore, when Outlook *receives* an email which contains both the From and the Sender headers, it will show both names in the GUI, with one being "on behalf of" the other.

I've seen other mailing lists use the Sender header to indicate the name of the mailing list itself, and this is what I want to use it for too. It makes the most sense to use it when the From header is always that of the actual sender (their own name/address) rather than that of the mailing list - this would typically be the case with a discussion list, but it could also be the case with an announcement list too, if you set your announcement list to show your own name/address in the From field rather than the mailing list's name/address.

In those scenarios, you then would put the mailing list's name/address into the Sender header instead of the From header. Thus, the recipient can see the name/address of the person who wrote the email in the From field, as well as the name/address of the mailing list which distributed the email (essentially "on behalf of" the sender) in the Sender field. It makes sense when you think of it that way, because the mailing list really is sending your email "on behalf of" you, in a similar way to how a secretary might send an email on behalf of their boss.

As you quoted from the RFC, "If the originator of the message can be indicated by a single mailbox and the author and transmitter are identical, the "Sender:" field SHOULD NOT be used. Otherwise, both fields SHOULD appear." Thus, if the mailing list name/address is already being used in the From field, then there's no need to repeat it again in the Sender field, and so the Sender field should be omitted.

In my case, I'm using Dada Bridge as a discussion list, which means the From field will always be that of the original sender (eg. John Smith). So I want to have the Sender header inserted, and for it to indicate the name/address of my mailing list. Ideally I'd like it to use <!-- tmpl_var list_settings.list_name --> for the name part and <!-- tmpl_var list_settings.discussion_pop_email --> for the email address part (which I equate to being the address of the mailing list), but I also think it's best to leave it up to the list owner to configure.

When an email configured this way, using the Sender header, is received by a client which supports it (such as Outlook), then the sender information would be displayed like this in the Outlook GUI:

My Mailing List [listaddress@example.com]; on behalf of John Smith [john.smith@hotmail.com]

...although the actual From column in the Inbox view of Outlook would still only display "John Smith", which is what the From header contains. So there's an advantage here of making it more clear to the recipient as to where the email came from, ie. what mailing list it came through, especially if the recipient might not know who the sender actually is (maybe it's a very big mailing list and the recipients don't know each other) - the recipient at least can see the name of the mailing list there in the GUI, if he's using a client like Outlook (perhaps other clients/webmail interfaces also show it too). So it's one more way of visibly promoting openness and clarity about the origin of a mailing list email, which can only be a good thing.

Perhaps a new control in the GUI can be added so this can be configured accordingly for those who want it?

I definitely think this should be (optionally) available at least to list owners who are using Dada Bridge as a discussion list, for the emails which are processed by the Dada Bridge plugin (ie. sent to the POP3 mailbox). But actually I also think the Sender header should be used when a message is sent from the "Send a Message" page, for those owners who are using Dada Bridge as a discussion list. Eg. I can imagine myself on some occasions wanting to send a message "as the administrator" directly from the Send a Message page, rather than from my own mail client, and I'd still want the Sender header inserted then, because my list is still a discussion list even though I'm not sending that particular message via the POP3 mailbox. Make sense?!

Cheers,
Jeremy

[justin]

If I set it to SMTP, but also set the $MAIL_SETTINGS variable to point to a .txt file, the .txt file is not used and the email still gets sent out via SMTP. Is this by design, or a bug?

Design - you're changing basically the, sendmail command, so yeah, it's only going to work when sending via sendmail.

I've seen other mailing lists use the Sender header to indicate the name of the mailing list itself, and this is what I want to use it for too

Can you give an example?

In those scenarios, you then would put the mailing list's name/address into the Sender header instead of the From header.

But that is exactly opposite of what the RFC states, correct?

I definitely think this should be (optionally) available at least to list owners who are using Dada Bridge as a discussion list, for the emails which are processed by the Dada Bridge plugin (ie. sent to the POP3 mailbox). But actually I also think the Sender header should be used when a message is sent from the "Send a Message" page, for those owners who are using Dada Bridge as a discussion list. Eg. I can imagine myself on some occasions wanting to send a message "as the administrator" directly from the Send a Message page, rather than from my own mail client, and I'd still want the Sender header inserted then, because my list is still a discussion list even though I'm not sending that particular message via the POP3 mailbox. Make sense?!

Do you want to put this in the wishlist? I don't know if the feature is a high priority for me, but at least others could see/add to the idea. I still think you're a great candidate to fork Dada Mail, try writing out the feature yourself and start doing your own testing. :)

[KiwiBloke]

I've seen other mailing lists use the Sender header to indicate the name of the mailing list itself, and this is what I want to use it for too

Can you give an example?

Sure. MDaemon, the commercial email server application I've been using for many years, automatically adds a Sender header for its mailing lists - and their developers are real sticklers for following the RFCs. I know that Majordomo and Mailman also add Sender headers by default as well, although they can be disabled if a list owner doesn't want it. I'm sure other list software has this functionality too.

In those scenarios, you then would put the mailing list's name/address into the Sender header instead of the From header.

But that is exactly opposite of what the RFC states, correct?

Not at all. RFC2822 specifically allows for this, and even encourages it (using the word SHOULD):

The originator fields indicate the mailbox(es) of the source of the
message. The "From:" field specifies the author(s) of the message,
that is, the mailbox(es) of the person(s) or system(s) responsible
for the writing of the message. The "Sender:" field specifies the
mailbox of the agent responsible for the actual transmission of the
message. For example, if a secretary were to send a message for
another person, the mailbox of the secretary would appear in the
"Sender:" field and the mailbox of the actual author would appear in
the "From:" field. If the originator of the message can be indicated
by a single mailbox and the author and transmitter are identical, the
"Sender:" field SHOULD NOT be used. Otherwise, both fields SHOULD
appear.

I've underlined the key part above. The "agent responsible" doesn't have to be a human - the RFC doesn't require this. It's simply whoever/whatever is responsible "for the actual transmission of the message". In the case of a mailing list, it is the mailing list itself which is responsible for the actual transmission of the message to all the recipients. That's why other mailing list software will (optionally) put the mailing list address into the Sender field, doing so with the knowledge that they are not violating this RFC in the process.

The only time when the Sender header should specifically NOT be used is when the Sender header and From header would contain the same address, as noted in the quote above. Which might be the case with an announce list. Otherwise it's fine to use the Sender header in this way.

As I mentioned previously, this has the key benefit of making it trivial for the recipient to see, in the GUI of his mail client, where the email came from, especially if the recipient is not familiar with the person in the From field, as can be the case with a discussion list. His mail client may show the message as coming from "Mailing List Name on behalf of John Smith", which is clearly an advantage to the recipient.

Other benefits include:

• Easier to filter mailing list messages, if your client has the ability to filter on the Sender header (so you can filter them into a separate folder etc).
  
  
• Can be helpful for whitelisting emails (or to help prevent the recipient's mail client or webmail provider from wrongly putting them into the Spam folder) - the recipient can add the mailing list address to his Address Book and/or his Whitelist (as Hotmail etc often have such a whitelist feature), and then the emails are less likely to be wrongly binned because the mailing list address also appears in the Sender header (it won't appear in the From header as that will be set to the actual person who sent the email). If you have a large discussion list, it's not possible for each recipient to whitelist the addresses of every other person on the list who might possibly send an email through the list. So the ability to simply whitelist the mailing list address itself, which can often be achieved just by adding it to your Address Book, is a good option, if the mailing list address is included there in the Sender header. I'm sure that not all webmail systems or mail clients will consider the Sender header in the first place, but some may, and every possible clue we can give to a mail client as to the identity and reliability of the email should be taken advantage of. That's also why we use other headers such as List-ID and X-BeenThere etc - it's not only to help with identification by the recipients themselves, but also by the recipient's client.
  
  
• A couple of the leading email authentication systems actually rely specifically on the Sender header for authenticating mail from a discussion list. Specifically, both Sender-ID and DomainKeys utilise the Sender header for authentication of mail from a discussion list (where the From header is set to the address of the individual sender and therefore changes from email to email).
  

Do you want to put this in the wishlist? I don't know if the feature is a high priority for me, but at least others could see/add to the idea.

Okay - where do I find the wishlist? And in the meantime, do you think you could at least add the ability for me to use template tags for the Sender header within the %EMAIL_HEADERS variable? That way I can at least add this header via my config file. If I only have one mailing list then I can set the Sender header there to a hard-coded email address, but if I've got multiple lists, then the Sender header should be different for each list, and the best way I can accomplish that is with template tags.

And I'm hoping my explanations here will help pursuade you to make it a slightly higher priority in the medium term! :) I guess having a customisable control for it in the Dada web UI (which allows for template tags too) would be my ideal goal.

I should note that some list owners might want to set their Sender header email address to the "list owner" address (ie. the bounce address), while other owners such as myself might want to have it set to the POP3 distribution list address. Both options are valid. And I imagine that announcement list owners probably won't need it at all, and some discussion list owners might not care for it either - so I think it should be optional and fully customisable.

I still think you're a great candidate to fork Dada Mail, try writing out the feature yourself and start doing your own testing.

I think the main problem with me doing this is that I don't know how to code in Perl (or whatever language Dada is written in)! I do know HTML, and I'm smart enough to know how to search through the files for something and tinker with the code where it's very obvious (such as modifying the Subject to take account of both RE and FW, like I did with that submission on Git the other day) - but I'd have no clue where to even begin with adding a brand new feature such as a new Sender header or whatever else. Believe me, if I knew how to do that, then I'd just go and do it and then offer the code back to you for inclusion in Dada, instead of having these long (but fun!) discussions here in the forums! ;)

[justin]

Wishlist: https://github.com/justingit/dada-mail/wiki/WishList

[KiwiBloke]

Wishlist: https://github.com/justingit/dada-mail/wiki/WishList

I must be blind, but I can't for the life of me see how to edit that page! (And I am currently logged into Github). I do see a link there for deleting the page, but not for editing it. :(