When a certificated based connector is not working

  • [アーティクル]

I recently worked with an organization that had an Exchange Online inbound connector which accepted mail from their on-premises Exchange environment. This connector was scoped by IP, and the organization wanted to change it to be scoped by certificate instead due to an upcoming change in Exchange Online. For more details on this change see Important notice for Office 365 email customers who have configured connectors.

Mail flow for this organization was as follows.

4011992_en_3

For Exchange Online to be a relay in this scenario, there needs to be an inbound connector setup in Exchange Online that is of type “on-premises.” To identify mail from the on-premises server, the Exchange Online inbound connector can either use the IP of the on-premises server, or the certificate that will be presented from on-premises. For some environments, after the upcoming change in July, the Exchange Online inbound connector will need to be scoped by certificate. Please see the above link to find out if you need to convert the scope on your Exchange Online inbound connector from IP based to certificate based.

For the organization that I was working with, they had an existing Exchange Online inbound connector which was scoped by IP, and the goal was to change it to be scoped by certificate. To reduce the chance of mail disruption, they created a second Exchange Online inbound connector, which was scoped to identify mail from on-premises by the certificate that was presented.
Because of propagation time in Office 365, we waited for an hour after the second connector was created before disabling the original IP based inbound connector. Shortly after disabling the IP based inbound connector, the on-premises server started receiving the following error from Exchange Online.

550 5.7.64 TenantAttribution; Relay Access Denied

For some reason, mail sent from on-premises to Exchange Online was not being attributed to the Exchange Online inbound connector which was certificate scoped. As such, Exchange Online was rejecting the mail because the service is not an open relay. In my humble experience, the three most common causes of this are as follows.

  • The certificate being presented by the on-premises servers had expired
  • The certificate being presented by the on-premises servers was not issued by a trusted CA
  • The on-premises servers are not presenting a certificate

For these cases, I always first obtain a network capture, as 99% of the time, these will show the exact problem. My go to network capture program is Wireshark. I have a previous blog post that includes handy Wireshark filters that I use in my troubleshooting which is aptly named Useful Wireshark Filters for Mail Flow Troubleshooting.

The trace provided by the organization consisted of 2187 rows, a lot of which was not SMTP traffic. The trace was taken while the IP based connector in Exchange Online was enabled, and so it didn’t contain the “Relay Access Denied” error, and so I cannot search for that. The trace also contains many SMTP conversations, and I’d like to see just one.
I start by filtering my results to only show traffic that was sent over port 25 using the following filter.

Tcp.port == 25

Next I search for the string “mail.protection.outlook.com,” and I look for an SMTP code of 220. This is the name that Exchange Online response with when a connection is first made to it.

string-search

As there are many SMTP conversations between Exchange Online and the on-premises servers, I pick my first result.

mail-service-ready

To view only traffic in this particular conversation, right click, select Follow, and then TCP Stream.

follow-tcp-stream

I am now left with 83 lines, which is pretty good considering we started with 2187. The rows that I’m interested in are the ones that show the certificate exchanges between Exchange Online and the on-premises server.

certificate-lines

The first highlighted line is Exchange Online presenting its certificate to the on-premises server. Digging into this packet I can see the Common Name, or CN, of the certificate being presented.

exo-cert

If I want to dig further, I can expand the above line and view all sorts of interesting details about the Exchange Online certificate.

Next I look at the second highlighted yellow line which is where the on-premises server sends its certificate to Exchange Online.

client-cert

Interestingly, there is no certificate being presented by the on-premises server, which explains why the mail is not being attributed to the Exchange Online inbound connector which is certificate scoped.

Our investigation quickly turned to the on-premises server where it was discovered that the organization's certificate was actually not associated with their Exchange send connector. Once the on-premises certificate was properly associated with the Exchange send connector we tested again. This time we had wonderful success, and there was much rejoicing! The certificate was properly being presented by Exchange on-premises, which allowed for the mail to be properly attributed to the organization's Exchange Online inbound connector.

Words of wisdom

When in doubt, grab a network capture.

Cheers,

Andrew

Comments

  • Anonymous
    February 23, 2017
    good troubleshootingthanks for sharing