Verify ExpressRoute connectivity

This article helps you verify and troubleshoot Azure ExpressRoute connectivity. ExpressRoute extends an on-premises network into the Microsoft Cloud over a private connection commonly facilitated by a connectivity provider. ExpressRoute connectivity traditionally involves three distinct network zones:

• Customer network
• Provider network
• Microsoft datacenter

This article helps you identify if and where a connectivity issue exists. You can then seek support from the appropriate team to resolve the issue.

Overview

The following diagram shows the logical connectivity of a customer network to the Microsoft network through ExpressRoute.

In the preceding diagram, the numbers indicate key network points:

1. Customer compute device (for example, a server or PC).
2. Customer edge routers (CEs).
3. Provider edge routers/switches (PEs) that face customer edge routers.
4. PEs that face Microsoft Enterprise Edge ExpressRoute routers (MSEEs). This article calls them PE-MSEEs.
5. MSEEs.
6. Virtual network gateway.
7. Compute device on the Azure virtual network.

At times, this article references these network points by their associated number.

Depending on the ExpressRoute connectivity model, network points 3 and 4 might be switches (layer 2 devices) or routers (layer 3 devices). The ExpressRoute connectivity models are cloud exchange colocation, point-to-point Ethernet connection, or any-to-any (IPVPN).

In the direct connectivity model, there are no network points 3 and 4. Instead, CEs (2) are directly connected to MSEEs via dark fiber.

If the cloud exchange colocation, point-to-point Ethernet, or direct connectivity model is used, CEs (2) establish Border Gateway Protocol (BGP) peering with MSEEs (5).

If the any-to-any (IPVPN) connectivity model is used, PE-MSEEs (4) establish BGP peering with MSEEs (5). PE-MSEEs propagate the routes received from Microsoft back to the customer network via the IPVPN service provider network.

The following sections represent the logical steps in troubleshooting an ExpressRoute circuit.

Verify circuit provisioning and state

Provisioning an ExpressRoute circuit establishes a redundant layer 2 connection between CEs/PE-MSEEs (2/4) and MSEEs (5). For more information on how to create, modify, provision, and verify an ExpressRoute circuit, see the article Create and modify an ExpressRoute circuit.

Verification via the Azure portal

In the Azure portal, open the page for the ExpressRoute circuit. The section of the page lists the ExpressRoute essentials, as shown in the following screenshot:

In the ExpressRoute essentials, Circuit status indicates the status of the circuit on the Microsoft side. Provider status indicates if the circuit has been provisioned or not provisioned on the service-provider side.

For an ExpressRoute circuit to be operational, Circuit status must be Enabled, and Provider status must be Provisioned.

Verification via PowerShell

To list all the ExpressRoute circuits in a resource group, use the following command:

Get-AzExpressRouteCircuit -ResourceGroupName "Test-ER-RG"

To select a particular ExpressRoute circuit in a resource group, use the following command:

Get-AzExpressRouteCircuit -ResourceGroupName "Test-ER-RG" -Name "Test-ER-Ckt"

Here's an example response:

Name                             : Test-ER-Ckt
ResourceGroupName                : Test-ER-RG
Location                         : westus2
Id                               : /subscriptions/***************************/resourceGroups/Test-ER-RG/providers/***********/expressRouteCircuits/Test-ER-Ckt
Etag                             : W/"################################"
ProvisioningState                : Succeeded
Sku                              : {
                                    "Name": "Standard_UnlimitedData",
                                    "Tier": "Standard",
                                    "Family": "UnlimitedData"
                                   }
CircuitProvisioningState         : Enabled
ServiceProviderProvisioningState : Provisioned
ServiceProviderNotes             :
ServiceProviderProperties        : {
                                    "ServiceProviderName": "****",
                                    "PeeringLocation": "******",
                                    "BandwidthInMbps": 100
                                   }
ServiceKey                       : **************************************
Peerings                         : []
Authorizations                   : []

To confirm that an ExpressRoute circuit is operational, pay particular attention to the following fields:

CircuitProvisioningState         : Enabled
ServiceProviderProvisioningState : Provisioned

Validate peering configuration

After the service provider has completed provisioning the ExpressRoute circuit, multiple routing configurations based on external BGP (eBGP) can be created over the ExpressRoute circuit between CEs/MSEE-PEs (2/4) and MSEEs (5). Each ExpressRoute circuit can have one or both of the following peering configurations:

• Azure private peering: traffic to private virtual networks in Azure
• Microsoft peering: traffic to public endpoints of platform as a service (PaaS) and software as a service (SaaS)

For more information on how to create and modify routing configuration, see the article Create and modify routing for an ExpressRoute circuit.

Verification via the Azure portal

In the Azure portal, you can check the status of an ExpressRoute circuit on the page for that circuit. The section of the page lists the ExpressRoute peerings, as shown in the following screenshot:

In the preceding example, Azure private peering is provisioned, but Azure public and Microsoft peerings aren't provisioned. A successfully provisioned peering context would also have the primary and secondary point-to-point subnets listed. The /30 subnets are used for the interface IP address of the MSEEs and CEs/PE-MSEEs. For the peerings that are provisioned, the listing also indicates who last modified the configuration.

Verification via PowerShell

To get the configuration details for Azure private peering, use the following commands:

$ckt = Get-AzExpressRouteCircuit -ResourceGroupName "Test-ER-RG" -Name "Test-ER-Ckt"
Get-AzExpressRouteCircuitPeeringConfig -Name "AzurePrivatePeering" -ExpressRouteCircuit $ckt

Here's an example response for a successfully configured private peering:

Name                       : AzurePrivatePeering
Id                         : /subscriptions/***************************/resourceGroups/Test-ER-RG/providers/***********/expressRouteCircuits/Test-ER-Ckt/peerings/AzurePrivatePeering
Etag                       : W/"################################"
PeeringType                : AzurePrivatePeering
AzureASN                   : 12076
PeerASN                    : 123##
PrimaryPeerAddressPrefix   : 172.16.0.0/30
SecondaryPeerAddressPrefix : 172.16.0.4/30
PrimaryAzurePort           :
SecondaryAzurePort         :
SharedKey                  :
VlanId                     : 200
MicrosoftPeeringConfig     : null
ProvisioningState          : Succeeded

A successfully enabled peering context would have the primary and secondary address prefixes listed. The /30 subnets are used for the interface IP address of the MSEEs and CEs/PE-MSEEs.

To get the configuration details for Microsoft peering, use the following commands:

$ckt = Get-AzExpressRouteCircuit -ResourceGroupName "Test-ER-RG" -Name "Test-ER-Ckt"
Get-AzExpressRouteCircuitPeeringConfig -Name "MicrosoftPeering" -ExpressRouteCircuit $ckt

If a peering isn't configured, you get an error message. Here's an example response when the stated peering isn't configured within the circuit:

Get-AzExpressRouteCircuitPeeringConfig : Sequence contains no matching element
At line:1 char:1
    + Get-AzExpressRouteCircuitPeeringConfig -Name "MicrosoftPeering ...
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : CloseError: (:) [Get-AzExpr...itPeeringConfig], InvalidOperationException
        + FullyQualifiedErrorId : Microsoft.Azure.Commands.Network.GetAzureExpressRouteCircuitPeeringConfigCommand

Validate ARP

The Address Resolution Protocol (ARP) table provides a mapping of the IP address and MAC address for a particular peering. The ARP table for an ExpressRoute circuit peering provides the following information for each interface (primary and secondary):

• Mapping of the IP address for the on-premises router interface to the MAC address
• Mapping of the IP address for the ExpressRoute router interface to the MAC address (optional)
• Age of the mapping

ARP tables can help validate layer 2 configuration and troubleshoot basic layer 2 connectivity issues.

To learn how to view the ARP table of an ExpressRoute peering and how to use the information to troubleshoot layer 2 connectivity issues, see Getting ARP tables in the Resource Manager deployment model.

Validate BGP and routes on the MSEE

To get the routing table from MSEE on the primary path for the private routing context, use the following command:

Get-AzExpressRouteCircuitRouteTable -DevicePath Primary -ExpressRouteCircuitName ******* -PeeringType AzurePrivatePeering -ResourceGroupName ****

Here's an example response:

Network : 10.1.0.0/16
NextHop : 10.17.17.141
LocPrf  :
Weight  : 0
Path    : 65515

Network : 10.1.0.0/16
NextHop : 10.17.17.140*
LocPrf  :
Weight  : 0
Path    : 65515

Network : 10.2.20.0/25
NextHop : 172.16.0.1
LocPrf  :
Weight  : 0
Path    : 123##

The following example shows the response of the command for a peering that doesn't exist:

Get-AzExpressRouteCircuitRouteTable : The BGP Peering AzurePublicPeering with Service Key ********************* is not found.
StatusCode: 400

Confirm the traffic flow

To get the combined primary and secondary path traffic statistics (bytes in and out) of a peering context, use the following command:

Get-AzExpressRouteCircuitStats -ResourceGroupName $RG -ExpressRouteCircuitName $CircuitName -PeeringType 'AzurePrivatePeering'

Here's an example output of the command:

PrimaryBytesIn PrimaryBytesOut SecondaryBytesIn SecondaryBytesOut
-------------- --------------- ---------------- -----------------
     240780020       239863857        240565035         239628474

Here's an example output of the command for a nonexistent peering:

Get-AzExpressRouteCircuitRouteTable : The BGP Peering AzurePublicPeering with Service Key ********************* is not found.
StatusCode: 400

Test private peering connectivity

Test your private peering connectivity by counting packets arriving at and leaving the Microsoft edge of your ExpressRoute circuit on the MSEE devices. This diagnostic tool works by applying an ACL to the MSEE to count the number of packets that hit specific ACL rules. Using this tool allows you to confirm connectivity by answering questions such as:

• Are my packets getting to Azure?
• Are they getting back to on-premises?

Run a test

1. To access the diagnostic tool, select Diagnose and solve problems from your ExpressRoute circuit in the Azure portal.

   

2. Select Connectivity & Performance issues.

   

3. In the Tell us more about the problem you are experiencing dropdown list, select Issues with Private peering.

   

4. Scroll down to the Test private-peering connectivity section and expand it.

   

5. Run the PsPing test from your on-premises IP address to your Azure IP address, and keep it running during the connectivity test.

6. Fill out the fields of the form. Be sure to enter the same on-premises and Azure IP addresses that you used in step 5. Then select Submit and wait for your results to load.

   

Interpret results

When your results are ready, you have two sets of them for the primary and secondary MSEE devices. Review the number of matches in and out, and use the following scenarios to interpret the results:

• You see packet matches sent and received on both MSEEs: This result indicates healthy traffic inbound to and outbound from the MSEEs on your circuit. If loss is occurring either on-premises or in Azure, it's happening downstream from the MSEEs.

• If you're testing PsPing from on-premises to Azure, received results show matches, but sent results show no matches: This result indicates that traffic is coming in to Azure but isn't returning to on-premises. Check for return-path routing issues. For example, are you advertising the appropriate prefixes to Azure? Is a user-defined route (UDR) overriding prefixes?

• If you're testing PsPing from Azure to on-premises, sent results show matches, but received results show no matches: This result indicates that traffic is coming in to on-premises but isn't returning to Azure. Work with your provider to find out why traffic isn't being routed to Azure via your ExpressRoute circuit.

• One MSEE shows no matches, but the other shows good matches: This result indicates that one MSEE isn't receiving or passing any traffic. It might be offline (for example, BGP/ARP is down).

  • You can run additional testing to confirm the unhealthy path by advertising a unique /32 on-premises route over the BGP session on this path.
  • Run "Test your private peering connectivity" using the unique /32 advertised as the on-premise destination address and reveiw the results to confirm the path health.

Your test results for each MSEE device look like the following example:

src 10.0.0.0 dst 20.0.0.0 dstport 3389 (received): 120 matches
src 20.0.0.0 srcport 3389 dst 10.0.0.0 (sent): 120 matches

This test result has the following properties:

• IP port: 3389
• On-premises IP address CIDR: 10.0.0.0
• Azure IP address CIDR: 20.0.0.0

Verify availability of the virtual network gateway

The ExpressRoute virtual network gateway facilitates the management and control plane connectivity to private link services and private IPs deployed to an Azure virtual network. Microsoft manages the virtual network gateway infrastructure and sometimes undergoes maintenance.

During a maintenance period, performance of the virtual network gateway may reduce. To troubleshoot connectivity issues to the virtual network and see if a recent maintenance event caused reduce capacity, follow these steps:

1. Select Diagnose and solve problems from your ExpressRoute circuit in the Azure portal.

   

2. Select the Performance Issues option.

   

3. Wait for the diagnostics to run and interpret the results.

   

   If maintenance was done on your virtual network gateway during a period when you experienced packet loss or latency. It's possible that the reduced capacity of the gateway contributed to connectivity issues you're experiencing for the targeted virtual network. Follow the recommended steps. To support a higher network throughput and avoid connectivity issues during future maintenance events, consider upgrading the virtual network gateway SKU.

Next steps

For more information or help, check out the following links:

• Microsoft Support
• Create and modify an ExpressRoute circuit
• Create and modify routing for an ExpressRoute circuit