Azure Container App returns 503 when scaling from 0 with multiple concurrent requests

Pedro Antunes 15

Hi,

I'm currently facing this issue with my Azure Container App (ACA):

When the ACA receives at the same time 4 or more requests and if the app has been "Scaled to 0", some requests return status 503 while others complete just fine.

status: 503 body: upstream connect error or disconnect/reset before headers. reset reason: connection termination

However, if ACA is already up, and I make the same requests (or even more), ACA works as expected and scales as expected.

This only happens if I make at least 4 requests at the same time. If I make 3 or less, ACA starts and works as expected too.

This even happens even if you make requests to an endpoint that does not exist inside the app (in this case it is a NodeJS express server).

I found this issue on Github that is relatable with my problem, but I don't know if there are any updates:

https://github.com/microsoft/azure-container-apps/issues/295

Ryan Hill 26,866 Reputation points Microsoft Employee

2024-01-03T18:36:49.47+00:00

Hi @Pedro Antunes during this time, have you checked the history of the replica count via the portal Metrics blade? Do you consistently see 503s as replica counts increase? Do the 503s go away after a few minutes?
Pedro Antunes 15 Reputation points

2024-01-04T17:54:01.57+00:00

Hi Ryan!

During the described test only 1 replica is created.

But we already tested with more requests in order to reach the replication rule, and when a new replica status is "Activating", some 503 errors are also returned with other replicas already running.

Basically, every time a new instance status is "Activating", 503 requests are returned. These 503 errors usually stop when all instances are in status "Running".
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

3 answers

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more
Ryan Hill 26,866 Reputation points Microsoft Employee

2024-01-04T20:33:51.0566667+00:00

Hey @Pedro Antunes

Thanks for clarifying above. Based on what you've provided, definitely sounds like this is a cold start scenario where your app isn't completely ready to accept requests yet. It is recommended that you have at least 1 replica to avoid these sorts of issues, but I understand you want to scale from 0.

Having said that, if you're only experiencing this issue when you're starting with 0 replicas and the transition from activating to running is in the tune of minutes with a low number of concurrent requests, then we would need to work more closely with you to investigate further, just comment down below. However, if the startup time is minimal and response time is paramount for your application, then you may need to consider running at least 1 replica to avoid this.
Please sign in to rate this answer.
Pedro Antunes 15 Reputation points

2024-01-05T17:35:54.6866667+00:00

Hi Ryan!

Thanks for your fast response!

I did try testing with a default hello world image that Azure provides and I figured that, when I set min replicas to 0, the 503 occurs likes I described in the first place, but when I already have 1 replica running, ACA works properly. So the problem seems to be only when ACA is scaled to 0 and then some concurrent requests happens (reminder: This only happens when are 4 or more concurrent requests)

Ryan Scott 0 Reputation points

2024-01-08T04:55:07.19+00:00

Hey guys, I am "Subtletree" from the github issue mentioned in the question. I found this question when trying to google again for a solution.

As I said in the github issue, my app was working perfectly up until the start of December when it seems that something changed.

For me this happens even with a single request as long as that request has a large enough body size (above 100kb or so).

My app has a reasonably fast start up and is ready to accept requests within a few seconds.

I have tried dozens of different health probe configurations.

Could we be hitting some some non-configurable ingress request buffer size limit? Is there anyway to see ingress logs for azure container apps?

Cheers!

Ryan Hill 26,866 Reputation points Microsoft Employee

2024-01-10T19:19:14.4066667+00:00

@Pedro Antunes @Ryan Scott send me an email enclosed with your subscription id and container app name to AzCommunity[at]microsoft[dot]com with the subject line ATTN: Ryan so we can investigate this issue furhter.

Marnik Van Hileghem 5 Reputation points

2024-01-11T08:12:56.02+00:00

Hello Ryan Hill I have the same issue. Can I send you a mail as well?

Ryan Hill 26,866 Reputation points Microsoft Employee

2024-01-11T19:57:24.25+00:00

@Marnik Van Hileghem please do.

Ryan Scott 0 Reputation points

2024-01-11T23:25:42.7733333+00:00

Just emailed. Really appreciate you looking into this!

Pedro Antunes 15 Reputation points

2024-01-12T09:56:41.0866667+00:00

Me too, I already sent you the email. Thank you for your time!
Sign in to comment
Ryan Hill 26,866 Reputation points Microsoft Employee

2024-02-13T14:15:15.1633333+00:00

@Pedro Antunes thanks for reaching out via offline.

The team has determined that the issue was that private certs weren't being respected when activating the pod. Once this patch was applied to your cluster, your issue was resolved.

For others that were affected by this issue, the patch is in the latest platform rollout and should resolve any incidents. However, if you continue to see an issue, please let me know by commenting on this post.
Please sign in to rate this answer.
Shuky Riechard 0 Reputation points

2024-05-07T14:05:25.71+00:00

Hi Ryan,

It seems like we are encountering the same issue

We run a private ACA and scale from 0 to 1 takes a few minutes which sometimes results in timeout on the requestor side.

Can we be in contact to investigate if this is the same private certs matter?
Sign in to comment

Share via

Azure Container App returns 503 when scaling from 0 with multiple concurrent requests

3 answers