Get-FederationInformation Fails to onmicrosoft.com with Bad Gateway Error

  • [アーティクル]

Usually the easy part of Exchange Online hybrid setup, stumped me recently. During the Exchange Hybrid Configuration Wizard (HCW), the wizard attempts to retrieve federation information from Exchange Online, and in our case this kept failing. There are many reasons why this could happen, so in this article I hope to give you the understanding on what is happening under the covers, so you can troubleshoot it more effectively, and also explain what was wrong in my specific scenario.

The HCW runs a series of PowerShell commands to setup hybrid, one of them is a command to get federation information from Exchange Online (for your O365 tenant email domain i.e. tenantname.mail.onmicrosoft.com). Note, as part of the wizard, it also performs the reverse query i.e. from Exchange Online, get federation information for on-premises domain. This article only covers the federation request to Exchange Online, perhaps in another article I can explain the reverse.

If you are having issues in the HCW with getting federation information from Exchange Online, the first thing to do is to run the PowerShell command yourself from the Exchange Management Console, with a verbose switch, so you can get more information about the issue. Example command:

Get-FederationInformation –domainname tenantname.mail.onmicrosoft.com –BypassAdditionalDomainValidation $true –verbose

The result of this might already point you in the direction of what the issue is. Now, let me explain what happens when you, or the HCW runs the Get-FederationInformation command to the onmicrosoft.com domain.

  1. Uses AutoDiscover to locate the autodiscover.svc service in Exchange Online for the email domain in question
  2. Calls the GetFederationInformation “method” on the autodiscover.svc service, using the service.svc end point location returned from step 1

The screen shot below shows the Fiddler trace for a successful call.

image

Note: The last request in the screenshot above is to the autodiscover.svc service.

As you can see, your Exchange Server where you are running this command from, needs access to the appropriate web end points in order for the command to succeed. In most cases where we see the command fail, it is because of an outbound access issue, and often a proxy server or firewall is blocking access to these resources, or the proxy configuration on the Exchange Server is not configured correctly.

“Obvious” things you would do on your Exchange Server to see if the server can access these resources:

  1. Open web browser and try and access the URL’s in question, that is:
    1. https://autodiscover.tenantname. mail.onmicrosoft.com/autodiscover/autodiscover.xml (note that this is HTTP, not HTTPS. This should redirect automatically to the URL below)
    2. https://autodiscover-s.outlook.com/autodiscover/autodiscover.xml (this should now prompt for credentials. Ignore the response of the page, the point is you should get prompted which means you can access the resource.)
    3. https://pod51050.outlook.com/autodiscover/autodiscover.svc (the server name will very likely be different for you. Like above you should be prompted for credentials and this proves connectivity)
    4. The steps above confirm your browser with it's proxy settings can connect to the endpoints. Interestingly, it doesn't matter if your web browser can or cannot connect, it is important that the Get-FederationInformation PowerShell command can! Read on to understand more.
  2. Telnet to:
    1. autodiscover.tenantname.mail.onmicrosoft.com port 80
    2. autodiscover-s.outlook.com port 443
    3. pod51050.outlook.com port 443
    4. As with the web browser tests, it is not that important if Telnet can or cannot connect, it is important that the Get-FederationInformation PowerShell command can!

It is still useful to run the above tests, because it allow you to understand if there are connection issues with the well known proxy configurations from the web browser or Telnet. However, those connection requests might all succeed, but the Get-FederationInformation PowerShell command doesn't. Why? This is because the Exchange Server or the Exchange Management Console PowerShell session might be using different proxy settings to your web browser or Telnet.

Here is a list of common proxy configuration “places” your Exchange Server would use, so check these to understand what your configuration looks like.

  1. Web Browser – e.g. in Internet Explorer, this is under Tools, Internet Options, Connections
  2. Windows HTTP (WinHTTP) – this allows for HTTP/S applications besides the web browser to use a proxy, and is more of a system wide setting on the server. This article describes how to view and set this.
  3. Proxy client – you might have a proxy client application on your sever e.g. TMG Client, which has a proxy configuration option
  4. Exchange Server – you can configure Exchange Server itself to use a specific proxy server. You will know this has been done because when you run the Get-FederationInformation command with the verbose switch, one of the first command outputs will tell you it is using a proxy server. You can also use the Get-ExchangeServer command to see if a proxy has been set for the Exchange Server.

Now, the last and probably most important thing to uncover, is how the Exchange Management Console PowerShell session connects to the web resources. It could be that it is not using any of the proxy configuration options mentioned above. In my scenario where we were getting a bad gateway error when running Get-FederationInformation, we could connect to the end points using the web browser and Telnet, but not through Exchange Management Console PowerShell. This is because Exchange Management Console PowerShell behind the scenes opens a remote PowerShell session to itself. Remote PowerShell runs in the context of IIS, and more specifically it runs in the context of an IIS application pool. Requests to the web resources are therefore made under the context of the IIS application pool, and this application therefore needs access to the end points. It is pretty rare for it not to have access, or have different proxy settings than the rest of the server, but it can happen.

The IIS application pools runs as the “Local System” account on the server. In our case this account had been sent a proxy setting via a GPO in the past and in theory it had since been removed from users, but it wasn't removed from the “Local System” account. Therefore, the Get-FederationInformation command was trying to connect via a proxy server that was no longer around. The screen shot below shows the 2 IIS PowerShell application pools. Note the “Local System” account.

image

To change the local system account settings, we needed to run through the steps discussed in this article.

In summary, your Exchange Server needs access to a bunch of end points in Office 365, and typically if you have issues it is because it cannot connect to these. Look at your proxy settings and understand how your Exchange Server is trying to connect to these end points. Ensure that the route allows for the connections to succeed. You could have firewall or proxy server rules preventing access, or you might have miss-configured proxy settings on the server. I prefer to bypass proxy servers and allow direct internet connections, but in some cases this might not be feasible in your environment.

Both Fiddler and Network Monitor are extremely useful tools when working with issues like these, but in some production environment having these installed might not be allowed by the organization.

Hopefully this article helps you understand what is happening under the covers, and allows for better troubleshooting if you come across Get-FederationInformation issues.

Good Luck,
Michael