(Not) Using the Additional Spam Filtering option for SPF hard fail to block apparently internal email spoofing

  • Article

 Recently, I’ve noticed that sometimes customers in Office 365 will login to the Exchange Admin Center, go to Protection –> Spam Filter –> Advanced Options and enable the Advanced Spam Filtering (ASF) option for “SPF Hard Fail.”

 The reason people do this is to stop messages from arriving into a customer’s organization that look like it’s an internal message, but really aren’t internal messages. Instead, they are phishing messages from an outside attacker that is spoofing the domain tomake it look like a regular internal message.

For example:

image

The above resembles an internal message from one high-level employee to another, but it isn’t. It’s a phisher impersonating the sender.

Since SPF is a technology that can be used to block spoofing, people naturally assume that it is appropriate to use here. But when it comes to using the ASF rule for SPF Hard Fail and using it to block apparently-internal spoofing messages, my advice is this: Don’t. It doesn’t catch most of these types of messages and you’ll probably end up getting a lot of false positives from senders who have misconfigured SPF record.

Instead, use DMARC to stop these types of messages. You can do this by creating an Exchange Transport Rule and setting up a DMARC record, even with an action of p=none.

The following instructions are for a straightforward scenario – your MX record points to Office 365 and you don’t route mail out-and-back to the service. More advanced customers already know this stuff, but less advanced ones can appreciate this guidance.

First, publish a DNS record for your domain with a DMARC record that doesn’t reject any email.

For example, for the domain contoso.com:

_dmarc.contoso.com IN TXT "v=DMARC1; p=none; pct=100"

After you set this up, any email that sends as you – either in the SMTP MAIL FROM or the header.from (the one you see in your email client) that doesn’t authenticate will fail a DMARC check. However, because you haven’t set up a policy that says to take action, the spam filter within Office 365 takes no special action.

Second, create an Exchange Transport Rule (mail flow -> rules) that has the following syntax:

*Apply this rule if
A message header includes… "Authentication-Results" header includes "dmarc=fail action=none header.from=contoso.com"
AND
The sender is located… Outside the organization
*Do the following…
Set the spam confidence level (SCL) to 9
AND
Set the message header to this value… Set the message header "X-ETR" to the value "DMARC=fail and set the SCL to 9"

You will need to add each domain you have for the dmarc=fail action=none header.from=<your domain> part of the rule.

What happens now is that if a message with your domain enters your organization, unless it passes SPF or DKIM authentication, the message will be marked as spam and take the high confidence spam action (spam quarantine, junk folder, or whatever action you have specified).

This means that any malicious phisher that is attempting to fake out anyone in your domain will be stopped from getting it through. But, the flip side is also true. If anyone sends email into your organization legitimately (such as a marketing newsletter), it will be marked as spam unless you add it to your SPF record (recommended) or configure an Allow rule for it.

At this point, you can stop if you never send marketing email or care about who’s spoofing you, but instead you just want the spoofs to stop.
 

Third, you may need to create exceptions to the above rule for known spoofers

You will be surprised to find out that there are many different senders that send email as your domain to your domain that you had no idea about. This is the case for every single large company I have worked with. Therefore, you will have to do some ETRs to exclude from filtering.

Ideally, you would add all third parties who send email as you to you into your SPF record, or set up DKIM so they can sign on your behalf. But if you can't, here's how you workaround it.
 

Workaround #1 - Set up an ETR for third parties who can authenticate with SPF

Not all third parties allow you to customize the SMTP MAIL FROM. They can customize the header.from to send email as you, but that will fail DMARC because the SPF domain (the bulk sender) will not align with the header.from domain (your domain). To not mark these as spam, create an ETR:

*Apply this rule if
A message header includes... "Authentication-Results" header includes "spf=pass"
AND
A message header includes... "Authentication-Results" header contains the text "smtp.mailfrom=<bulk sender's domain>"
AND
The sender domain is <your domain>
  AND
    The sender is located… Outside the organization
*Do the following…  
    Set the spam confidence level (SCL) to -1
  AND
    Set the message header to this value… Set the message header "X-ETR" to the value "Skip filtering for known good bulk sender spoofing <your domain> to not get marked as spam by DMARC ETR"

* AND Stop processing more rules

 What this does is look for senders that you trust and skips filtering for them without having to manage IP allow lists. This rule must run at a higher priority than the DMARC rule in Step 2. Because you also have the condition that the sender domain is your own domain, this forces the header.from address to contain your domain while the SPF domain is the bulk sender. 

Workaround #2 - Set up an ETR for third parties who cannot authenticate with SPF by using IP+sender domain combination

Even today in 2015, not all senders send with SPF. Or, there are line-of-business applications on cloud-hosted applications (e.g., Microsoft Azure, Rackspace, Amazone EC2, and so forth) that simply relay email. In that case, you'll need to do this:

*Apply this rule if
The sender domain is <your domain>
AND
The sender IP is <IP sending you the email>
AND
The sender is located… Outside the organization
*Do the following…
Set the spam confidence level (SCL) to -1
AND
Set the message header to this value… Set the message header "X-ETR" to the value "Skip filtering for known IP address spoofing <your domain> to not get marked as spam by DMARC ETR"

* AND Stop processing more rules

* Sender applies to Header and not envelope

This looks for IP addresses that spoof you and skip filtering so they can avoid getting marked as spam. It's more of a pain to manage IPs instead of domains (with SPF pass), however. This rule must also run at a higher priority than the DMARC ETR in Step 2.
 

Fourth, if you have the ASF rule for SPF hard fail enabled, disable it.

You don’t need this rule enabled if you’re trying to stop spoofing of your own domain. The above works better.

Fourth, make sure you have enabled outbound DKIM signing in Office 365.

You can do that by following the instructions here:

https://blogs.msdn.com/b/tzink/archive/2015/10/08/manually-hooking-up-dkim-signing-in-office-365.aspx

Fifth, keep going to get to DMARC p=reject.

The above Exchange Transport Rule with the DMARC combination works for messages sent into your organization if it flows through Office 365 [1]. However, other phishers can still spoof your domain and send to people you communicate with. In that case, you need to ensure that your domain can’t be spoofed by using DMARC with p=reject.

That’s beyond the scope of this blog post, but I recommend working with a third party vendor to set up DMARC reporting and then working with them to analyze which IPs belong in your SPF record and which ones do not, and then eventually getting to publish a stricter DMARC policy.

Three from the industry are:

- Agari,https://agari.com/ – Microsoft used Agari to improve its SPF authentication
- ReturnPath, https://www.returnpath.com/ – provides good tools for senders and receivers
- DMARCIAN, https://dmarcian.com/ – has some good tools for smaller domains

Each of these companies can help you extend the required DMARC reporting addresses in your DMARC record to inventory all of your senders so you can avoid false positives and eventually remove the ETR you set up above.

Sixth, we know this is a bit complicated but we are looking at ways to fix it.

Setting up DMARC records and corresponding ETRs is fine for a large organization that knows how to manage its Office 365 protection, and update its DNS records. However, smaller companies often have trouble doing either of these.

We understand this and are thinking of ways to simplify it.

* * * * * * * * * * * *

This article’s instructions should help cut down on the apparently-internal-spoofing problem that we see today. Let us know if it works for you, and if it doesn’t, what parts are lacking.

[1] Office 365 should be your primary email filter. This means that your MX record must point to Office 365 – not to a 3rd party filter, and not to an on-premise mail server. If you do either of those, you will generate lots of false positives. Here's our latest guidance on supported mail flow routing in Office 365:

Mail flow best practices for Exchange Online and Office 365
https://technet.microsoft.com/EN-US/library/jj937232(v=exchg.150).aspx

 

Related articles:

Comments

  • Anonymous
    July 21, 2015
    Another great article, thank you. :)

  • Anonymous
    August 10, 2015
    Shouldn't it be "dmarc=fail action=none header.from=contoso.com"

  • Anonymous
    October 01, 2015
    "... make sure you have enabled outbound DKIM signing in Office 365....it is targeted by the end of Q3, 2015." Any update on this?

  • Anonymous
    October 27, 2015
    Great article, thanks ! I've implemented it as you exposed but I had to make an exception to the rule like this : Exception when header'Authentication-Results' contains 'spf=pass' Because when receiving email from a third-party mail server not able to add a DKIM signature (thank you for the recent post about this subject for Office 365) : while the SPF test passed (the 3rd party server is in the SPF for a long time), the dmarc result is marked as failed. I prefered to rely on the SPF record in that case. Now I'm worrying how DMARC will react when I change the value "p=none" to "p=reject" in that case (3rd party server not signing with DKIM but pass SPF). It should pass but regarding this result, I'm not sure. Can you inform about that ?

  • Anonymous
    November 01, 2015
    Forget my last message, there was a problem with alignment (DMARC). So it's OK :-)

  • Anonymous
    February 25, 2016
    The comment has been removed

  • Anonymous
    January 31, 2017
    So do i only need to create DMARC and Transport rule to block Spoof emails?or should i also need to create SPF record for my domain?

  • Anonymous
    November 01, 2017
    Hello,Thank you for the very informative post. Trying to do Workaround #1 for our bulk mailer but when adding the exception rule I cannot select a second "A message header includes..." it's grayed out. All I can select the is the first "A message header includes"... and add the value spf=pass. Without that I cannot put our bulk mailer's domain name. How can I make this work?

    • Anonymous
      November 01, 2017
      Also, when adding the smtp.mailfrom= do I put our bulk mailer's domain or the full email address?
      • Anonymous
        November 03, 2017
        Just the domain, not the full email address.
    • Anonymous
      December 28, 2017
      Yonko, did you figure this out?