EOP Mysteries Solved – Inbound messages from a particular sender arrive with no subject or body

  • [アーティクル]

This is an interesting case that I recently worked on and would like to share as part of this series. An organization that uses Exchange Online Protection was receiving automated emails from a partner. These emails were arriving with empty subjects and bodies. When the partner sent the automated emails to a Hotmail.com / outlook.com address, they would also arrive with no subject or body.


 
However, when the partner sent these emails to a Gmail account, the would arrive just fine with the subject and body intact.


 
Huh. So what’s going on here?

Problem and Environment

If it wasn’t for the successful Gmail test, I would have immediately said the problem was with the third party sending application / server. Test messages from the third party looked fine in Gmail, but in Hotmail.com / Outlook.com, and after passing through EOP, the messages showed no subject or body.
This organization had their mailboxes in an on-premises mail environment that was protected with Exchange Online Protection.

Troubleshooting

Test 1 – Reproduce the problem

First things first, I first want to reproduce the problem. Maybe I will get lucky and everything will now be working! So I have the third party send a single email from their application server to the following recipients.

  • My customer
  • My gmail.com account

The customers on-premises mailbox received the message and indeed there was no subject or body. I received the same message in my Gmail account and saw subject and body for the message. I verified that the copy my customer received traversed through EOP before being delivered to their on-premises environment.

Ok, so the problem is still occurring and incredibly easy to reproduce. Let’s move on.

Test 2 – Have the third party send to more accounts

Now I’m curious if the problem is related only to this customer and their EOP tenant, or will the same thing happen for any organization using Exchange Online or Exchange Online protection? Next I ask for the third party to send another test message to a range of accounts. Following are the recipients accounts along with the results.

  • My work account (hosted in Exchange Online) – message arrived with no subject or body
  • A second Exchange Online mailbox – message arrived with no subject or body
  • My Outlook.com account – message arrived with no subject or body
  • My gmail.com account – message arrived with the subject and body intact

These results surprised me. I did not anticipate Outlook.com also not being able to display the subject or body.

Based on experience, I know that some mail services follow the SMTP RFC specification much more strictly than others. I also know that Microsoft follows the SMTP RFC quite strictly.

My guess at this point is that the third party is sending the message via SMTP that is not within the SMTP RFC specification. And so when a Microsoft email service (consumer or business) receives the message, it cannot be parsed properly and thus we end up with the subject and body missing.

Test 3 – When in doubt, run WireShark

A network trace never lies, and so next I turn to WireShark to either prove, or disprove my guess that the third party is sending outside the SMTP specification. I setup WireShark on my Exchange 2013 server and have the third party send me a message directly. Here’s what I captured.

The connection to my server was made fine, however when it was time for the DATA to be sent, WireShark interprets the received packet as being Malformed. While this doesn’t always represent a problem, it certainly requires investigation.

In the same capture, I received a message from my customer in which I can see the IMF packet was received fine and is interpretable by WireShark (not malformed), so I know that WireShark doesn’t just interpret all IMF packets as malformed.

Test 4 – Telnet test

I know that the third party is using an application to send these messages. Next I’d like to try and isolate the sending server from the sending application. I ask the third party to send me a message from their application server, but using Telnet.

I received the message just fine in both my Outlook.com account and my Exchange Online account, both subject and body were intact. Based on this troubleshooting, the problem is looking to be with the partners sending application. My guess still is that the application is not sending within the SMTP RFC, and that GMail is more lenient to the spec which is why it will show the subject and body of the messages sent from the application.

Test 5 – Final test

As a final test, I asked the partner to run a network capture on their application server and capture the application sending an outbound message. This network trace also showed an unreadable packet coming out of the application server. At this point I’m feeling pretty confident that the problem is with the sending application.

Resolution and Root Cause

After investigation, the partner found that their application was not properly inserting Carriage-Return Line-Feeds into sent messages. This caused the sent messages to be out of spec for the SMTP RFC and caused Outlook.com and Exchange Online to not properly parse the message.

The partner updated their code so that it properly inserted line-feeds, and sent another round of test messages. This time the messages showed up just fine in Outlook.com and Exchange Online, showing both subject and body.

Success!!

Wrap-up

In a problem like this there is no right or wrong order of what to try and where to look. My approach is just one of many that could have been taken. In the end, my goal was to systematically rule out the various points in the mail flow to get down to the component that was causing the problem. Mission accomplished!

Comments

  • Anonymous
    October 11, 2015
    great write up
    thanks
  • Anonymous
    May 04, 2017
    We just ran into the exact same issue with a internally developed application. We had already turned to Wireshark, and this article helped us put it all together, so thank you! As FYI, the application was sending via Exchange 2010 just fine, but when we moved it to Exchange 2016, we starting seeing this issue.