Container Instances stuck in "Waiting" state Production

Stephen.Hoey 25

A number of our containers have restarted over night and are now in waiting state it seems they are not able to pull the images and restart this has not happened before and the containers would restart without a problem and pull the images. These are our production environments and this now affects our SLA.

kobulloc-MSFT 25,651 Reputation points Microsoft Employee

2023-03-22T06:19:43.1766667+00:00

Hello, @Stephen.Hoey !
If an answer has been helpful, please consider accepting the answer to help increase visibility of this question for other members of the Microsoft Q&A community. If not, please let us know what is still needed in the comments so the question can be answered. Thank you for helping to improve Microsoft Q&A!
Pierre Gradit 0 Reputation points

2024-03-29T18:19:19.35+00:00

I have the same problem and no logs. How can I get some support?

5 answers

Ammar-Abdelqader01 1,006 Reputation points Microsoft Employee

2023-03-16T08:43:15.7266667+00:00

hello @Stephen.Hoey

Thanks for reaching out and I hope you are doing well.

sorry to hear that the containers restarted and it's still in a waiting state.

could you please Retrieve container logs and events, follow the steps from this link: https://video2.skills-academy.com/en-us/azure/container-instances/container-instances-get-logs

in the meantime, to help resolve this issue, your container instances are stuck in the "waiting state" to stop those containers and then start the containers again and check if the issue resolved.

check this link to stop and start containers or use the azure portal.

waiting for your feedback,

Please "Accept the answer" if the information helped you. This will help us and others in the community as well. Feel free to reply with any other questions or concerns.
Please sign in to rate this answer.

2 people found this answer helpful.
Olivier Vanekem 81 Reputation points

2023-03-16T08:59:41.8266667+00:00

I'm having similar issues since yesterday night. The Azure Container Instances were not responding. Restarting them not always work at the moment ("Technical Issue"). I would suspect a capacity issue in the Microsoft data centers (not able to create new Docker instances because of capacity). I'm in the West Europe zone.

Ármann Davíð Sigurðsson 5 Reputation points

2023-03-16T09:12:42.3133333+00:00

I am also experiencing the same issue and it is also affecting our production environment. Restarting or hard Stop/Start does not solve the issue, the containers just hang in the waiting state.

Ammar-Abdelqader01 1,006 Reputation points Microsoft Employee

2023-03-16T09:31:24.5266667+00:00

@Olivier Vanekem @Ármann Davíð Sigurðsson @Stephen.Hoey it looks like there is an issue, our team working to resolve this issue, and I will reply here once it is resolved.

for now can you please try to deploy your container in a different region?

Olivier Vanekem 81 Reputation points

2023-03-16T09:54:03.09+00:00

Thanks for confirming there is an issue @Ammar-Abdelqader01 It is not possible for me to deploy elsewhere since the containers are part of a much larger architecture all hosted in WE.

I think also it would be great if the Azure status page reflects the actual status (everything is green there...):

David Pfannholzer 0 Reputation points

2023-03-16T10:17:00.08+00:00

We experience the same problem... but on the alert site here: https://portal.azure.com/#view/Microsoft_Azure_Health/AzureHealthBrowseBlade/~/serviceIssues

is nothing to see.

Does this issue occur in a certain region or does it affect specific container configurations?

Ammar-Abdelqader01 1,006 Reputation points Microsoft Employee

2023-03-16T10:20:53.07+00:00

hello @Olivier Vanekem @David Pfannholzer

thank you for your feedback, the issue appeared lately, and This issue has been catching higher attention. It is under review.
I need to ask for your patience while we work to resolve this issue in west Europe region.

Ármann Davíð Sigurðsson 5 Reputation points

2023-03-16T10:38:18.97+00:00

@Ammar-Abdelqader01 is it possible to give us a rough estimate of when this issue would be solved so we can give some feedback to our customers?

Ammar-Abdelqader01 1,006 Reputation points Microsoft Employee

2023-03-16T10:44:27.3666667+00:00

@Ármann Davíð Sigurðsson

There is currently a known issue impacting ACI + Vnet integration in WestEurope region , its started on 7:30 UTC 16/3/2023 , ACI +Vnet instances will be in waiting/Creating or failed state we are actively investigating to solve this as soon as possible. We appreciate your understanding and apologies for the inconvenience, will work on resolving this as soon as possible.Will keep you posted.

Olivier Vanekem 81 Reputation points

2023-03-16T12:16:26.4433333+00:00

For those interested, restarting the containers work. Of course this can have incidence because (in my case) the containers are connected to virtual networks and receive a private IP address which is then stored in a private DNS (used by an Azure Application Gateway which then connects to the container's private IP). So I had to manually update the DNS with the newly allocated IPs (there is no possibility to "auto-register" the IP like it is possible using a virtual server).

Ármann Davíð Sigurðsson 5 Reputation points

2023-03-16T12:43:12.6233333+00:00

@Olivier Vanekem Then I guess you got lucky, I have been restarting and re-creating the container instances during the day and I always end up in the Waiting state.

Ammar-Abdelqader01 1,006 Reputation points Microsoft Employee

2023-03-16T12:59:05.5466667+00:00

@Olivier Vanekem @Ármann Davíð Sigurðsson

this issue with two categories :

1- is not using BYO VNET Mitigation Stop and Start (not restart) will resolve the issue.

2- using BYO VNET High Probability hitting the age-old IP Leak bug networking.

Mitigation: to delete the Subnet and recreate it (and redeploy ACI again of course)

ETA for fix is 6 hours or more

Paweł Fudala 0 Reputation points

2023-03-16T13:22:54.9233333+00:00

@Ammar-Abdelqader01 why the issue is not being anyhow visible on Service Health blade nor on status.azure.com?

Ármann Davíð Sigurðsson 5 Reputation points

2023-03-16T14:36:51.12+00:00

@Ammar-Abdelqader01 This fix only works sporadically, we have the same setup across multiple subscriptions and it is like flipping a coin if it works or not.

Ammar-Abdelqader01 1,006 Reputation points Microsoft Employee

2023-03-16T14:55:36.2666667+00:00

Hello, @Ármann Davíð Sigurðsson if you are not using your own VNET or if you are not using custom VNET Stop and start enough.

if you are using custom VNET you can delete the Subnet and recreate it with the same IP address range, but with a new name (and redeploy ACI again of course)

Ármann Davíð Sigurðsson 5 Reputation points

2023-03-16T15:57:35.0266667+00:00

@Ammar-Abdelqader01 Re-creating the subnet with a new name did the trick. Thanks for the assistance.
Sign in to comment
Gaston Muijtjens 5 Reputation points

2023-03-16T11:59:24.7666667+00:00
We were having exactly the same issue this morning on three separate subscriptions simultaneously. It caused an outage for some of our customer apps in production. Logs were not accessible from the Portal (stuck on loading) and the Azure CLI command returned the following error:

az container logs --resource-group <rg_name> --name <name>

(InternalServerError) Encountered an internal server error. The tracking activity id is '<id>', correlation id is '<id>'. Code: InternalServerError Message: Encountered an internal server error. The tracking activity id is '<id>', correlation id is '<id>'.

Restarting the Azure Container Instance resolved the issues for us. There are no visible messages in Azure Service Health or the Azure Status page.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment
E.G 0 Reputation points

2023-03-16T13:07:47.9+00:00

Hello,

We are experiencing the same issue, all instance containers are currently stuck in a waiting state. This issue is affecting various subscriptions and accounts within our infrastructure.

As you can imagine, this problem has severe consequences for our operations, particularly since some of the affected containers are in production environments. We first noticed this problem early this morning.

In light of the situation, we would greatly appreciate your prompt attention to this issue. We believe this issue is impacting a significant portion, if not all, of the infrastructure related to Azure container instances.

We appreciate your prompt assistance in resolving this matter.

Thank you.
Please sign in to rate this answer.
Ammar-Abdelqader01 1,006 Reputation points Microsoft Employee

2023-03-16T13:12:56.02+00:00

hello @E.G

I will reply to the same comment above:

this issue with two categories :

1- is not using BYO VNET Mitigation Stop and Start (not restart) will resolve the issue.

2- using BYO VNET High Probability hitting the age-old IP Leak bug networking.

Mitigation: to delete the Subnet and recreate it (and redeploy ACI again of course)

ETA for the fix is 6 hours or more
Sign in to comment
Praveen Mishra (debug) 0 Reputation points

2024-03-01T19:24:59.42+00:00

Hi Team, Do we have fix for above issue. Container going in 'waiting' state at every 10 seconds.
Please sign in to rate this answer.
Praveen Mishra (debug) 0 Reputation points

2024-03-01T19:25:56.4133333+00:00

kobulloc-MSFT 25,651 Reputation points Microsoft Employee

2024-03-04T18:51:51.58+00:00

Hello, @Praveen Mishra (debug) ! Please create a new thread to help with tracking and we'd be happy to help.
Sign in to comment
Javier Americo Bullon Tenorio 0 Reputation points

2024-06-20T03:46:53.3+00:00

Same issue for me, any solution? Thanks.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Share via

Container Instances stuck in "Waiting" state Production

5 answers