Routing Issues with S2S VPN VNET Peered with ExpressRoute VNET

@RahulRana-1085 Thanks for posting your query on Microsoft Q&A.
Our team is looking into it and will get back as soon as possible.

Hello @RahulRana-1085 ,

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.Looking at your setup, I would like to point out few things below:

VNET1(S2S VPN) and VNET2(ExpressRoute) are currently peered so that infrastructure in VNET2 can talk to the on-prem devices connected via VNET1.

The above is not possible. If you have ExpressRoute and site-to-site coexistence, transit routing isn't supported.

To enable transit routing between ExpressRoute and Azure VPN, you need to set up Azure Route Server. But the Azure VPN and ExpressRoute gateway must be deployed in the same virtual network as Route Server in order for BGP peering to be successfully established.

Refer: https://video2.skills-academy.com/en-us/azure/vpn-gateway/vpn-gateway-vpn-faq#how-do-i-enable-routing-between-my-site-to-site-vpn-connection-and-my-expressroute

https://video2.skills-academy.com/en-us/azure/expressroute/how-to-configure-coexisting-gateway-portal#to-enable-transit-routing-between-expressroute-and-azure-vpn

https://video2.skills-academy.com/en-us/azure/virtual-network/virtual-network-configure-vnet-connections#configure-transit-routing-in-an-expressroute-connection

https://video2.skills-academy.com/en-us/azure/route-server/expressroute-vpn-support#how-does-it-work

The hub and spoke model define the below configuration:

• Hub virtual network. The hub virtual network hosts shared Azure services. Workloads hosted in the spoke virtual networks can use these services. The hub virtual network is the central point of connectivity for cross-premises networks.
• Spoke virtual networks. Spoke virtual networks isolate and manage workloads separately in each spoke. Each workload can include multiple tiers, with multiple subnets connected through Azure load balancers. Spokes can exist in different subscriptions and represent different environments, such as Production and Non-production.

Refer: https://video2.skills-academy.com/en-us/azure/architecture/networking/architecture/hub-spoke?tabs=cli

https://video2.skills-academy.com/en-us/azure/cloud-adoption-framework/ready/azure-best-practices/hub-spoke-network-topology

But in your case, all 3 Vnets are hub Vnets.

It is recommended to connect all networks to the same ExpressRoute circuit by using any-to-any ExpressRoute connectivity model and linking all Vnets to the same ExpressRoute circuit.

Or use ExpressRoute and Site-to-Site coexisting connections in the same Vnet to connect to different on-premises sites as below:

https://video2.skills-academy.com/en-us/azure/expressroute/how-to-configure-coexisting-gateway-portal#configure-a-site-to-site-vpn-to-connect-to-sites-not-connected-through-expressroute

So, first we need to re-align the existing setup and requirement to proceed further to Azure Firewall configuration.

Regards,

Gita

Thank you for your response @GitaraniSharma-MSFT !

Sorry I wasn't clear in what I meant by:

VNET1(S2S VPN) and VNET2(ExpressRoute) are currently peered so that infrastructure in VNET2 can talk to the on-prem devices connected via VNET1.

In VNET2, I have Azure VMs that allows me to manage/SSH into the on-prem devices. The S2S VPN in VNET1 is using one cellular provider and the ExpressRoute in VNET2 is using another. The on-prem devices cannot talk to each other.

Flow for S2S:

On-Prem Device --> Cellular Provider --> S2S VPN --> Azure VNET1 (and public internet)

Flow for VNET2 ExpressRoute:

On-Prem Device --> Cellular Provider --> ExpressRoute --> Azure VNET2 (using private peering)

Now since VNET1 and VNET2 are peered, I can do this:

Azure VM in VNET2 --> (ssh) --> On-Prem Device connected via S2S VPN

So in essence, regardless of the connectivity source, all traffic from on-prem devices should ideally pass through the Azure Firewall in VNET2 first before hitting any VM subnets or public internet.