Express Route Routing Issues (Azure to On-premises route)

@Jaykishan Bairagi ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

Wrt, "outbound traffic of internet from Azure has to go through Azure firewall and not to On-premises."

• As long as you are not advertising 0.0.0.0/0 via ExpressRoute and there are UDRs pointing 0.0.0.0/0 to Azure Firewall IP, this should be fine.

Wrt, "Traffic flow from Azure to On-Premises >> Intermittently working"

• If you want Azure firewall to filter Traffic flow from Azure to On-Premises, make sure the UDRs used are the exact range of the OnPrem Network and not 0.0.0.0/0
• See : How Azure selects a route
  • Longest prefix match algorithm would route OnPrem Network directly to Gateway without routing via the Firewall.
• Can you confirm if this is how the setup is configured.
• Once this is done,
  • You informed there are intermittent failures of ICMP/TCPPing
  • Do you see the traffic in Azure Firewall? During failure or not?

Cheers,

Kapil

Hi Kapil,

Thanks for quick reply.

Wrt, "outbound traffic of internet from Azure has to go through Azure firewall and not to On-premises." ->

All workload subnets >> Single Route Table (Outbound) >> 0.0.0.0/0 UDR + All on-premises network ranges >> Next Hop >> Virtual appliance >> Azure FW Private IP -> Does this work?

Wrt, "Traffic flow from Azure to On-Premises >> Intermittently working"

The above route table is single route table with all UDRs for Internet outbound from Azure + outbound to express route through Azure firewall.

1. Yes, we can see the outbound traffic to express route is listed in the Azure Firewall. However, the on-premises where I'm telnetting to is sending an INVALID flag to Azure Firewall subnet. This is same for all on-premises destinations. Inbound to Azure from on-premise doesn't have any issues and logged in Azure Firewall
2. I tried removing the 0.0.0.0/0 from the routetable and has the on-premises destination range DESTINATIONIP/16 >> same issue
3. I've even added just the /32 of destination ip to the route table >> same issue
4. Output of psping ran from one of the windows virtual machine to destination (on-premises server) Connecting to DESTINATIONIP:16722 (warmup): from 0.0.0.0:58849: This operation returned because the timeout period expired. Connecting to DESTINATIONIP:16722: Sent = 0, Received = 0, Lost = 0 (0% loss), Minimum = 0.00ms, Maximum = 0.00ms, Average = 0.00ms

@Jaykishan Bairagi ,

All workload subnets >> Single Route Table (Outbound) >> 0.0.0.0/0 UDR + All on-premises network ranges >> Next Hop >> Virtual appliance >> Azure FW Private IP -> Does this work?

• This is correct
• This should send internet traffic directly to Internet from Firewall
• OnPrem traffic will be sent to OnPrem via Firewall

P.S: Make sure there is a UDR at the GatewaySubnet also, pointing Azure traffic to the firewall.

Wrt "Yes, we can see the outbound traffic to express route is listed in the Azure Firewall. However, the on-premises where I'm telnetting to is sending an INVALID flag to Azure Firewall subnet"

• Then this looks like a configuration issue from the OnPrem server(s) only
• Did you check why the OnPrem server would reply with a "INVALID flag" from local logs.
  • Is the packet from Azure to OnPrem reaching the OnPrem server (PC) or just reaching the OnPrem device (Router/Firewall) ?

Only reason I can think of is : Suboptimal routing between virtual networks.

Make sure the ExpressRoute Gateway uses the ExpressRoute Circuit in the same region by assigning a higher weight to the local connection than to the remote

Cheers,

Kapil

Hi @KapilAnanth-MSFT -

#1. P.S: Make sure there is a UDR at the GatewaySubnet also, pointing Azure traffic to the firewall. ->

On the Inbound Routetable [Propagate gateway routes = YES] >> Express route gateway subnet (attached) >> UDR - Our vNet address destined from on-premises >> Next Hop >> Virtual Appliance >> Azure Firewall private IP.

So, that's current route table. I tried now adding a UDR of on-premise one range where the destination server is and still receiving the "Connecting to DESTINATIONIP:16722 (warmup): from 0.0.0.0:58849: This operation returned because the timeout period expired. Connecting to DESTINATIONIP:16722: Sent = 0, Received = 0, Lost = 0 (0% loss), Minimum = 0.00ms, Maximum = 0.00ms, Average = 0.00ms"

@KapilAnanth-MSFT -

P.S: Make sure there is a UDR at the GatewaySubnet also, pointing Azure traffic to the firewall. >> Yes, I've a UDR like below for all express route inbound traffic -

inbound route table [Propagate gateway routes = YES] >> UDR - vNet range /23 >> next hop >> virtual appliance >> azure firewall private IP. >> working from on-premises to connect to Azure

Also, tried this just now -

inbound route table [Propagate gateway routes = NO] >> UDR - vNet range /23 >> next hop >> virtual appliance >> azure firewall private IP. >> not working both directions.

Also, tried this just now -

inbound route table [Propagate gateway routes = YES] >> UDR - vNet range /23 + on-premises destination server network range >> next hop >> virtual appliance >> azure firewall private IP. >> not working both directions.

I noticed if I remove the "Express route virtual network gateway" from the inbound route table then I can successfully psping from Azure to On-Premises without any issues but On-premises can't reach to Azure servers. looks like only one route is working.

Make sure the ExpressRoute Gateway uses the ExpressRoute Circuit in the same region by assigning a higher weight to the local connection than to the remote >> Yes, we already have the weights assigned to local connections to "100" as a local regional circuit preference and the fallback connections to other region circuits as "0"

I've gone through this article suspecting something like this asymmetric routing not sure how to fix this but this matches our problem here-

from the below article reference-

"For Instance, let’s say in the below example, SourceIP 10.10.0.68 is trying to connect to the Destination IP 10.10.0.132 on destination Port RDP 3389 with a random source port 51369. We have set up an asymmetric routing scenario where we made sure that this traffic is not routed through the firewall, however, the return traffic is configured to go through the firewall that is from 10.10.0.132 to 10.10.0.68 with source port 3389 and destination port 51369. In this case, by using the Flow Trace Logs, we can clearly see if there is an invalid packet that is not recognized by the Firewall that could be causing the connection issues. "

https://techcommunity.microsoft.com/t5/azure-network-security-blog/logging-and-metrics-enhancements-to-azure-firewall-now-in/ba-p/3820087