Azure front door always routes request to the latest or last origin in an origin group of blob storages.

Zeeshan Ashraf 0

Actally i was expecting front door to select the nearest and fastest blob storage to upload files, like i added 4 origins:

Central Australia.
North India
US east 2
UAE north.

Based on my assumptions, it should have selected uae north for me because it has the lowest latency. I have set same weights and priorities for each origin. But it always routes requests to the last origin in number. What am i doing wrong?
User's image

User's image

Luke Murray 11,076 Reputation points MVP

2024-07-15T05:24:45.9766667+00:00

Try this first: Switch the protocol to HTTPS and check whether Session affinity is enabled.
Zeeshan Ashraf 0 Reputation points

2024-07-17T10:40:08.7633333+00:00
@KapilAnanth-MSFT
Thanks for replying, brother. Regarding your questions:

Is this still happening? Yes.

How are you saying UAE North is the one with the lowest latency? I measured it myself, plus I actually checked this on a website. Also checked in front door metrics.
I confirmed it by uploading files; in the US, my files take twice as long as in the UAE. I live in Pakistan.

Is it because of the geographical location of the client?
I think this should not matter, as you say; it all depends on latency, and yes, it is not acting according to latency terms.

Do you mean the request is always served by "wedfludev"?
Yes! That's how it is.

And if you swap the order of "wedfludev" and "testfrontwed", are requests always served by "testfrontwed"?
Yup. It's weird.

Make sure you have not enabled caching in AFD - this could be giving you the response from a cached endpoint that originally routed to "wedfludev". Nope, no caching.

Have you enabled session affinity?
No.

Let me know if you wanna see any other configuration settings...
Zeeshan Ashraf 0 Reputation points

2024-07-17T10:44:25.16+00:00

@Luke Murray I have not enabled session affinity plus i tried it with HTTPS protocol. I don't know if it has to do something with blob storages accounts themselves?
I have checked in metrics as well, health probes counts are in k's and wedfludev, the last one is neither at top or bottom, also in metrics, wedfludev has high latency as compared to others.

1 answer

KapilAnanth-MSFT 43,471 Reputation points Microsoft Employee

2024-07-15T05:18:54.1833333+00:00
@Zeeshan Ashraf ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

Looking at your configuration, the load distribution should be round robin provided all the origins have a latency of less than 300 ms.

When you say, "it should have selected uae north for me because it has the lowest latency"

Is this is still happening?

May I ask how you are saying UAE North is the one with the lowest latency.

Is it because of the geographical location of the client?

The 'closest' origin isn't necessarily closest as measured by geographic distance. Instead, Azure Front Door determines the closest origin by measuring network latency. Read more about Azure Front Door routing architecture.

When you say, "But it always routes requests to the last origin in number."

Do you mean the request is always served by "wedfludev"?

And if you swap the order of "wedfludev" and "testfrontwed", requests are always served by "testfrontwed"?

Make sure you have not enabled caching in AFD - this could be giving you the response from a cached endpoint that originally routed to "wedfludev"

Have you enabled session affinity ?

Cheers,

Kapil
Please sign in to rate this answer.
Zeeshan Ashraf 0 Reputation points

2024-07-17T10:48:43.9133333+00:00

@Luke Murray @KapilAnanth-MSFT
I actually found a workaround. I completely ditched Front Door because serving files was not our main priority. I calculated latencies on user sign-in and also set a default account in case anything fails. I also added a fallback mechanism where, if a blob upload fails, the next one is selected based on latency order. This is working fine for now, but I know this is not the best approach.

KapilAnanth-MSFT 43,471 Reputation points Microsoft Employee

2024-07-17T11:08:07.3633333+00:00

@Zeeshan Ashraf ,

Thanks for keeping us updated.

I see you are now currently not using AFD.

As next steps, you can consider checking Health probe log of AFD by Configuring logs from Portal

You can consider the "TotalLatency" field and understand what are the various latencies from your Origins

You can use the "POP" field to understand the region from which the probe was sent

See : Microsoft POP locations

In case they are more than 300ms (latency sensitivity), then they will not be considered for load balancing

Luke Murray is correct, we should use HTTPS as probing method instead of HTTP as this is required by the Storage Account.

Cheers,

Kapil

Zeeshan Ashraf 0 Reputation points

2024-07-18T12:18:55.9433333+00:00

@KapilAnanth-MSFT Thanks alot!
I think we are almost about to solve my problem. Yes you were right, the probes were failing with 400 error.

{ "time": "1970-01-01T00:00:21.8032240Z", "resourceId": "/SUBSCRIPTIONS/A32D337D-985D-4CED-A88C-52D85BC2075F/RESOURCEGROUPS/WEDFLUDEV/PROVIDERS/MICROSOFT.CDN/PROFILES/WEDFLUDEVFRONTDOOR", "category": "FrontDoorHealthProbeLog", "operationName": "Microsoft.Cdn/Profiles/FrontDoorHealthProbeLog/Write", "properties": {"healthProbeId":"c58b522f0e2f483a874f22e4296bdaa6","pop":"MAA","httpVerb":"HEAD","result":"OriginError","httpStatusCode":"400","probeUrl":"https://wedfludev.blob.core.windows.net:443/","originName":"wedfludev.blob.core.windows.net","originIP":"20.60.88.100","totalLatencyMilliseconds":"959","connectionLatencyMilliseconds":"236","DNSLatencyMicroseconds":"4000"}}

I don't know why it should fail like this, i mean microsoft should have configured a head base route on blobs? Or is it something wrong with my configuration?
Then i shifted to Get route, and i placed a test.json file like this,

https://wedfludev.blob.core.windows.net/ping/test.json

i can access it through browser.
and now logs are different but i dont understand what status 0 means here:

{ "time": "1970-01-01T00:00:21.8032240Z", "resourceId": "/SUBSCRIPTIONS/A32D337D-985D-4CED-A88C-52D85BC2075F/RESOURCEGROUPS/WEDFLUDEV/PROVIDERS/MICROSOFT.CDN/PROFILES/WEDFLUDEVFRONTDOOR", "category": "FrontDoorHealthProbeLog", "operationName": "Microsoft.Cdn/Profiles/FrontDoorHealthProbeLog/Write", "properties": {"healthProbeId":"fd2f3451986c4185a1cac06d01ed7338","pop":"DM","httpVerb":"GET","result":"DNSTimeout","httpStatusCode":"0","probeUrl":"https://wedfludev.blob.core.windows.net:443/ping/test.json","originName":"wedfludev.blob.core.windows.net","originIP":"","totalLatencyMilliseconds":"3000","connectionLatencyMilliseconds":"n/a","DNSLatencyMicroseconds":"3000000"}}

I am a newbie at front door, please dont mind. Any help is grealty appreciated.

KapilAnanth-MSFT 43,471 Reputation points Microsoft Employee

2024-07-18T13:18:57.4033333+00:00

@Zeeshan Ashraf ,

Glad we are able to make progress.

Health Probe logs make sense

If you access the endpoint "https://wedfludev.blob.core.windows.net" , we get 400 response only, even from browser

So having a dedicated container to respond to probing is the correct approach

Wrt, "microsoft should have configured a head base route on blobs" - this does not seem to be the case

However, you should be able to use HEAD with "https://wedfludev.blob.core.windows.net/ping/test.json"

Wrt the new logs,

"httpStatusCode" being 0 means time-out

Specifically DNS time out issue in your case

As you have configured "totalLatencyMilliseconds" to be 3000 ms, the request took more than that.

This is evident as the DNS Lookup itself took 3000000 microseconds (which is 3000ms) so the request never made it

I believe the storage account in question is in EastUS2 and "DM" is Central US (Des Moines, Iowa)

Ideally, this latency should not be the case

However, let's not forget the main issue here.

Is the issue still happening?

Or is the traffic now getting load balanced across all storage (expected behavior)

i.e., Provided that you have configured a dedicated container in all other storage accounts of name "/ping/test.json"

If so, can you check the HealthProbeLogs once again?

I believe the above log is a one-off issue, or specific to the POP "DM"

Is the "httpStatusCode" 0 for other POPs and across all Storage Accounts?

Cheers,

Kapil

Zeeshan Ashraf 0 Reputation points

2024-07-18T14:44:30.85+00:00

@KapilAnanth-MSFT Thanks,
I have changed it back to Head and result is same, actually i am getting this 0 code at all other POP's as well as all other storage accounts.

As you have configured "totalLatencyMilliseconds" to be 3000 ms, the request took more than that.

Can you point me out where it is configurable?

Anyway, I will check uploads and see if issue was resolved. Thanks alot again.

KapilAnanth-MSFT 43,471 Reputation points Microsoft Employee

2024-07-19T08:54:42.1533333+00:00

@Zeeshan Ashraf ,

Wrt, "As you have configured "totalLatencyMilliseconds" to be 3000 ms, the request took more than that."

Apologies.

This value seems to be not configurable.

I got it mixed up with the "LatencySensitivity" in Origin Group

Also, you must note that Azure Front Door only logs every failed health probe request.

This means, successful probes are not logged

So it is highly possible that there is many successful probes and load balancing is happening as expected.

Cheers,

Kapil
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure front door always routes request to the latest or last origin in an origin group of blob storages.

1 answer

Your answer