ER Circuit co-exist for same location but different provider

C NIRMAL KUMAR 1

We have a customer whose request is to migrated a ER circuit from existing provider to a new provider.

He is not going to totally migrate all the traffic to new circuit. He is planning to implement it in a phase. For example A,B,C etc. is the network routes currently using Old provider ER circuit. He wanted to move A traffic alone to new ER circuit and B and c will still need to use Old ER circuit.

Is the above achievable?

We suggest them to use AS-path prepend. So new provider will only advertise A traffic to new ER with AS-path prepend and B&C routes will only be available in old ER. In this case will B&C use OLD ER to reach on-prem.

We tested the above, Without As-path prepend for A network alone On-prem to azure able to reach via new ER. But azure to on-prem the request is getting timedout at azure gateway level. Any specific reason for this? In this case old ER has A,B,C routes & New ER has A routes alone. So azure should load-balances on-premises bound traffic across all the ExpressRoute paths using Equal-cost multi-path (ECMP) routing.

Why in my case it's not happening. Do I need to add any additional config on azure side.

1 answer

KapilAnanth-MSFT 39,211 Reputation points Microsoft Employee

2024-03-18T15:37:02.25+00:00
@nirmal kumar ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

Can you please elaborate your architecture in a more detailed manner.

You said "new provider will only advertise A traffic to new ER with AS-path prepend"

But for the testing, you said "We tested the above, Without As-path prepend for A network alone On-prem to azure"

Are you using AS Path Prepend or not?

With your architecture,

I take it that A,B,C Networks are the OnPrem Network Ranges

But please correct me if I am wrong.

In that case ,

The control is totally up to you.

You can advertise A network via NewCircuit and B,C Network via OldCircuit.

If you are advertising A network only via NewCircuit, there is no need for AS Path Prepending.

AS Path Prepending only comes into picture when both the NewCircuit and OldCircuit advertise A's address prefix.

I understand that the latter is your case

In that case, AS Path prepend is used to reduce the preference. The lengthier the AS Path, the less it is preferred.

So, ideally, for A's traffic to use the new Circuit, you must advertise AS Path prefix for A in the older Circuit (so it is less preferred)

See : AS PATH prepending

Note

The maximum number of circuits in the same peering location that can be linked to a sinle virtual network is 4

Cheers,

Kapil
Please sign in to rate this answer.
C NIRMAL KUMAR 1 Reputation point

2024-03-19T06:34:59.2533333+00:00

Yes, A,B & C are network on-premises range.

In testing we have not used any As path prepend. Both old and new ER has A network routes.

OLD ER - has - A,B,C routes

NEW ER - has - A routes only

In the above case, When we tried to reach one of the on-prem vm in A network from azure. The traffic is getting timeout at gateway. Is this the expected behavior. As per MS doc azure will route traffic based on ECMP. Then why the traffic is getting timedout at gateway.

KapilAnanth-MSFT 39,211 Reputation points Microsoft Employee

2024-03-19T08:34:16.2966667+00:00

@nirmal kumar ,

I don't think that is the correct approach.

The document states the use of ECMP for routing between virtual networks and not for OnPrem traffic

Since Equal-Cost-Multi-Path (ECMP) routing is used to load-balance inter-VNet traffic, some traffic flows take the longer path and get routed at the remote ExpressRoute circuit.

And how are you saying the packets are getting timed out at the Gateway itself?

It is possible that the traffic went via one ExR and your OnPrem is dropping the traffic because of Asymmetrical routing?

See : Challenges of using multiple ExpressRoute circuits

Make sure the OnPrem router is capable of receiving traffic from both the Circuits (ISP)

NOTE:

Asymmetrical routing over ExpressRoute private peering doesn't necessarily block traffic flow.

In any case, please follow one of the below recommended practices to make Azure to prefer one ExpressRoute circuit over another.

Advertising more specific route over the preferred ExpressRoute circuit compared to other ExpressRoute circuit(s)

Configuring higher Connection Weight on the connection that links the virtual network to the preferred ExpressRoute circuit

Advertising the routes over less preferred ExpressRoute circuit with longer AS Path (AS Path prepend)

Cheers,

Kapil

C NIRMAL KUMAR 1 Reputation point

2024-03-20T05:24:35.0066667+00:00

The below statement is in MS-DOC. Attaching link for your reference.

https://video2.skills-academy.com/en-us/azure/expressroute/designing-for-disaster-recovery-with-expressroute-privatepeering#challenges-of-using-multiple-expressroute-circuits

By default, if you advertise routes identically over all the ExpressRoute paths, Azure load-balances on-premises bound traffic across all the ExpressRoute paths using Equal-cost multi-path (ECMP) routing.

Is there any preference that express route will choose over another on the below mentioned strategies.

Advertising more specific route over the preferred ExpressRoute circuit compared to other ExpressRoute circuit(s)

Configuring higher Connection Weight on the connection that links the virtual network to the preferred ExpressRoute circuit

Advertising the routes over less preferred ExpressRoute circuit with longer AS Path (AS Path prepend)

For example if I have a specific route on old circuit and weight on new circuit. Azure gateway will choose which ER.

KapilAnanth-MSFT 39,211 Reputation points Microsoft Employee

2024-03-20T12:12:02.6966667+00:00

@nirmal kumar ,

I agree.

By default, if you advertise routes identically over all the ExpressRoute paths, Azure load-balances on-premises bound traffic across all the ExpressRoute paths using Equal-cost multi-path (ECMP) routing.

However, it is possible that Azure is not dropping the traffic and could be one of the intermediaries.

To address your queries,

Advertising more specific route over the preferred ExpressRoute circuit compared to other ExpressRoute circuit(s) takes the highest precedence

If same routes are advertised over two ExR Circuits, then the Connection Weight is considered. The connection with higher Connection Weight is preferred.

In case same routes are advertised over two ExR Circuits and they same Connection Weight, then AS Path prepend is considered. The connection with shorter AS Path prepend is preferred.

So, in your example,

"if I have a specific route on old circuit and weight on new circuit. Azure gateway will choose which ER."

Old Circuit is preferred.

See:

Kindly let us know if this helps or you need further assistance on this issue.

Thanks,

Kapil

Please don’t forget to close the thread by clicking "Accept the answer" wherever the information provided helps you, as this can be beneficial to other community members.
Sign in to comment

Share via

ER Circuit co-exist for same location but different provider

1 answer