Email authentication should work out of the box and we should not rely upon domain owners to do it themselves

  • [アーティクル]

This is going to be a long post. Sorry. I didn’t have time to write a shorter one.

Who should be responsible for setting up email authentication records?

For years, I have been discussing the virtues of publishing email authentication records including SPF, DKIM, and DMARC. There are plenty of tutorials and documentation on the web, and I have written many articles in support of it, explaining its basics as well, and how it works in Office 365.

SPF, DKIM, and DMARC can be challenging to set up for numerous reasons:

  1. SPF requires that you know who all of your senders are and keep them up-to-date. This is hard to do in large, decentralized service where any department can contract out another email sender to send on your domain’s behalf.

  2. SPF also has a maximum 10 DNS lookups. In the modern world, it’s easy to exceed that limit with only a handful of 3rd parties sending as you. Office 365 alone requires you to have a spare lookup limit of 3 DNS lookups.

  3. DKIM is tricky to set up, and even trickier to rotate keys. You have to download and install the software, and then keep the keys in sync across all those servers you are running. Then, you have to publish keys in DNS making sure to copy/paste correctly. Then, when it comes time to update the keys, you have to do it all over again – unless, of course, you are using Office 365 in which case there is slim chance of copy/paste errors and you never have to rotate keys.

  4. DMARC is challenging to set up because you have monitor it all yourself and then make the decision to be more or less aggressive on your mail stream.

For domain owners who are highly motivated to do this sort of thing – people like me and all my friends at MAAWG – this is time-consuming but most of us enjoy it and so it’s not too bad.

However, I don’t think we should expect small business owners to do it. Instead, what I think should happen is this:

If you provide email infrastructure on behalf of someone else, you are responsible for helping your customers set up DKIM/DMARC alignment, and you should rotate keys automatically.

For example, if you are a large bulk email provider bulksender.com, you should:

1. Ensure that your customer sets up DKIM and then you rotate their keys automatically

a) Tell your customers to publish two common keys in their DNS, and these should be predictable. For example, for the domain contoso.com:

HereIsMyFirstKey._domainkey.contoso.com IN TXT "v=DKIM1; k=rsa; p=<public key #1>

HereIsMySecondKey._domainkey.contoso.com IN TXT "v=DKIM1; k=rsa; p=<public key #2>

b) The above is the bare minimum you should do. Ideally, you should not ask your customers to publish TXT records at all for DKIM, but instead publish CNAMEs that point to your own infrastructure:

HereIsMyFirstKey._domainkey.contoso.com IN CNAME HereIsMyFirstKey._domainkey.bigsender.com

HereIsMySecondKey._domainkey.contoso.com IN CNAME HereIsMySecondKey._domainkey.bigsender.com

Ideally, everyone would get their own key. However, this may not be possible as not everyone can store and replicate multiple keys securely. In that case, you may be stuck with common keys.

c) Third, the big sender should sign with the first key and then 6-12 months later, flip over to the second key. The first key should remain in place but then updated to new values a week or two later. Then, 6-12 months later, repeat the process and flip back.

Doing it this way ensures that the customer has DMARC alignment based upon the d=domain in the DKIM signature, and their DKIM keys are rotated automatically without them having to do anything. If you ask your customer to update their DKIM keys, my guess is that perhaps 5% of them will ever do it.

So don’t make them do it. Do it for them automatically by putting them into a good state from the beginning in that their email from your infrastructure is DKIM/DMARC aligned. You can still have your own 5321.MailFrom be your own domain for the purposes of bounce-processing and passing SPF.

Here in Office 365, we’re working on modifying the signup process such that in addition to publishing a couple of DNS records to verify your domain, you also have to publish some CNAMEs to get your DKIM properly set up, too. We use the DKIM selectors selector1 and selector2.

2. If you are a domain registrar with access to the domain’s DNS settings, you should auto-populate their SPF record with options to manage it themselves

SPF records are hard to maintain yourself for a couple of reasons. The first is if you are a small business owner, you probably have better things to do than play around with DNS settings and updating your SPF record (unless you are an email geek like me; if you’re not an email geek you do what your email service tells you to do). I see plenty of small domains and big companies get their SPF record syntactically wrong.

The second reason SPF records are hard to maintain is because once your business grows, you have to keep maintaining a list of 3rd party senders who send on your behalf and keep adding them to your SPF record, but SPF has a cap of 10 DNS lookups. In 2016, it’s easy to exceed that limit. Office 365 takes up 3, Google Apps takes up 3, and SalesForce takes up 2. You’re now at 8. How many more bulk senders can you add? The normal solution is to break it up by subdomains so some senders send from email.contoso.com, others from e-mail.contoso.com, and so forth.

That’s a pain to maintain, too. Believe me, I know, because I’ve worked with teams internal to Microsoft to set it up. The process of setting up these subdomains and then making DNS changes through the change review board is not fast, and even I find it hard to juggle so many different teams’ requests.

A big innovation I see in this space is through the use of SPF macros. Macros allow you as a domain owner to publish a wildcard in your SPF record and instead of returning a static SPF record, it looks it up dynamically. For example, suppose contoso.com’s traditional SPF record looked like this:

contoso.com IN TXT "v=spf1 ip4:1.2.3.4 include:spf.protection.outlook.com include:_spf.google.com include:salesforce.com include:bigsender.com –all"

Contoso knows that it has gone over the limit, so instead it publishes a macro in its SPF record:

contoso.com IN TXT "v=spf1 include:%{i}._ip.{%h}._ehlo.{%d}._spf.contoso.com –all"

Huh?

What macros do is tell the receiving mail server to substitute the part in %{…} with the corresponding value from the email you are checking with the metadata from the email. In the above:

%{i} means the connecting IP
%{h} means the connecting mail server’s EHLO or HELO string
%{d} means the domain you are looking up

If a mail server were to look up the SPF record for contoso.com and see that weird syntax, it would then do a second lookup substituting in the parameters from the message. So, if the email comes from the IP 1.2.3.4 from a mail server whose EHLO string is mailserver.com, and checking the domain contoso.com, then substituting the corresponding parts it looks up the SPF record for 1.2.3.4._ip.mailserver.com._ehlo.contoso.com._spf.contoso.com. The response might be something like this:

contoso.com IN TXT "v=spf1 ip4:1.2.3.4 –all"

A mail server could then do a regular SPF evaluation to see if 1.2.3.4 is in that particular SPF record. In this case it is.

If instead a message arrived from a mail server with an EHLO string NAM02-CY1-obe.outbound.protection.outlook.com from the IP 65.55.159.61, the server would look up 65.55.159.61._ip.NAM02-CY1-obe.outbound.protection.outlook.com._ehlo.contoso.com._spf.contoso.com then the DNS server might return:

contoso.com IN TXT "v=spf1 include:spf.protection.outlook.com –all"

The mail server then does a normal SPF lookup on this record to see if the IP is in there (it is).

Or, instead if a message arrived with an EHLO string of localhost from the IP 254.1.1.1, the server would get back the macro and then do a second lookup on 254.1.1.1._ip.localhost._ehlo.contoso.com._spf.contoso.com. The DNS server might return:

contoso.com IN TXT "v=spf1 –all"

Because 254.1.1.1 is not in your SPF record at all, the dynamic SPF says it’s not there and tells the verifier to stop checking.

So, rather than you having to publish a static DNS record, you have a backend database that parses out variables from the DNS query and dynamically returns the corresponding response.

Receiver –> DNS –> parse out DNS query –> look up in table ---+
|
|
+----------------------------+
|
+
+-------+---------+------------------------------------------------+
| IP | EHLO | SPF record to return |
+-------+---------+------------------------------------------------+
| IP #1 | EHLO #1 | v=spf1 include:1.2.3.4 –all |
| IP #2 | EHLO #2 | v=spf1 include:spf.protection.outlook.com –all |
| IP #3 | EHLO #3 | v=spf1 include:_spf.google.com –all |
| IP #4 | EHLO #4 | v=spf1 include:_spf.bigsender.com –all |
+-------+---------+------------------------------------------------+
|
|
Return corresponding SPF record                                 |
|
Receiver does normal SPF check

Instead of publishing all of your 3rd parties who send as you in DNS, you would use a backend database (preferably through a nice, pretty UX) to add all your senders (the fact that this publishes an SPF record is abstracted from you; you don’t even need to know what SPF is). Ideally, it would have a checkbox for the most common large senders but also let you manually enter IPs that are not associated with any common large sender.

Who do you use to send email?

_x_ Office 365
___ Google Apps
_x_ SalesForce
_x_ Big Sender
_x_ Click here to enter in IPs manually

You could enter in as many as you like because you aren’t worried about exceeding any DNS lookups, the lookup table parses it all out, looks it up in the table, and then dynamically returns the result depending upon the IP/EHLO combination.

It’s an elegant way around the 10 DNS lookups, helps prevent syntax errors, lets you stuff all of your senders into your root domain (@contoso.com instead of @email.contoso.com) and makes it easier to manage over the long term.

I, as an author of this blog and speaking of my own opinion and not necessarily on behalf of Microsoft, bring all this up and describe it this way because there is a small company that provides exactly this type of service called ValiMail (my examples above using macros copies their syntax).

I’m familiar with their service and I like what they do, it’s a great way to outsource your email authentication so you don’t have to do it yourself. I also like how they have a lot of automation involved when you set up DMARC records, your SPF record self-populates. That sets you up into a good state and keeps you there.

If you’re looking for a service to do this, you may want to see what they offer.

And this brings me back to the registrars.

It would be great if registrars provided a service like this for all their customers. When you registered a new domain, you would get a wildcard SPF record by default with the option to manually do it yourself. The registrar would also publish a default DMARC record for you with the option for you to manually do it yourself. In that way, if you don’t know how to do it yourself, your registrar takes care of it for you. And if you feel confident to do it yourself (or you’re a geek like me and enjoy doing it yourself) you could take over by picking the “I want to do it myself” option.

But in either case, your email is SPF/DMARC aligned and kept up-to-date with minimal inputs from you, right from the beginning of domain registration. I understand that registrars would then have to provide this service, but maybe they could contract that out.

Putting it all together

So you see, rather than making domain owners be responsible for their own email authentication, service providers should be responsible for helping their customers set it up:

  1. If your service sends email, make sure that your customer enables DKIM/DMARC alignment by telling them what CNAMEs to publish (or barring that, at least a TXT record) and ensure that they cannot complete signup without them.
  2. If your service is a registrar or can control DNS records, auto-populate their SPF record to get SPF/DMARC alignment with an option to do it themselves. Ideally, you would also publish a DMARC record. That’s a story for another post (or maybe when I get the motivation, come back and extend this post).

Even though there are many administrators today who set this up and take it as a cost of doing business, there are probably even more small business owners (and even medium and large) who don’t have the expertise to set it up and maintain.

I think that this is the best way to get email authentication compliance. Let’s make it simple so all a customer or domain owner has to do is answer a few questions and the backend takes care of the rest.

Forward!

Comments

  • Anonymous
    January 27, 2016
    Great post. I am one of the small business IT professionals you are speaking about. I actually wear about 24 hats around here and manually dealing with SPF and DKIM/DMARC is the last thing I have time for. The main selling point of switching to exchange on office 365 was to minimize my involvement with email. I don't have the bandwidth to stay abreast of all the best practices for exchange policies and I would hope the product would do this for me. So if I understand correctly 365 still needs manual CNAME changes to enable DMARC? I am still failing DMARC auth on mxtoolbox and I'd love to get that set up. I have used your GoDaddy auto publishing feature and it worked great. Super time saver. I would love to flip a switch to enable DMARC and have it post the CNAMES as well. Am I to assume that if the SPF txt entry is correct that I should be good to go with DMARC CNAMES? Sorry, like I said I am new to this! Thanks for the great blog.

  • Anonymous
    January 27, 2016
    The comment has been removed

  • Anonymous
    February 03, 2016
    When will Microsoft start publishing DMARC records for .onmicrosoft.com and .mail.onmicrosoft.com domains ? Right now it is an easy target.

  • Anonymous
    July 06, 2017
    The comment has been removed