Container Instances fail to start the majority of the time

Bradley Pursglove 1 Reputation point
2020-11-11T21:26:25.217+00:00

I have a function app that programmatically defines a container instance using the Azure Fluent SDK. This uses a ClamAV image from our private container registry and mounts a shared volume for use across our generated container instances.

The majority of the time this container fails to start and the error message received contains little/no detail as to the cause, yet sometimes the function app is able to succeed in creating and starting a container instance, with no changes whatsoever to when it fails.

The most common error I receive is: 

Microsoft.Rest.Azure.CloudException: Long running operation failed with status 'Failed'. Additional Info:''
   at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.CheckForErrors()
   at Microsoft.Rest.ClientRuntime.Azure.LRO.PutLRO`2.CheckForErrors()
   at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.StartPollingAsync()
   at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.BeginLROAsync()
   at Microsoft.Rest.Azure.AzureClientExtensions.GetLongRunningOperationResultAsync[TBody,THeader](IAzureClient client, AzureOperationResponse`2 response, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Rest.Azure.AzureClientExtensions.GetLongRunningOperationResultAsync[TBody](IAzureClient client, AzureOperationResponse`1 response, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Rest.Azure.AzureClientExtensions.GetPutOrPatchOperationResultAsync[TBody](IAzureClient client, AzureOperationResponse`1 response, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ContainerInstance.Fluent.ContainerGroupsOperations.CreateOrUpdateWithHttpMessagesAsync(String resourceGroupName, String containerGroupName, ContainerGroupInner containerGroup, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ContainerInstance.Fluent.ContainerGroupsOperationsExtensions.CreateOrUpdateAsync(IContainerGroupsOperations operations, String resourceGroupName, String containerGroupName, ContainerGroupInner containerGroup, CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ContainerInstance.Fluent.ContainerGroupImpl.CreateResourceAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Core.ResourceActions.Creatable`4.Microsoft.Azure.Management.ResourceManager.Fluent.Core.ResourceActions.IResourceCreator<IResourceT>.CreateResourceAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Core.DAG.CreatorTaskItem`1.ExecuteAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Management.ResourceManager.Fluent.Core.DAG.TaskGroupBase`1.ExecuteNodeTaskAsync(DAGNode`1 node, CancellationToken cancellationToken)

By checking in the portal I can see that the container is in fact successfully created, but often has no IP and I'm unable to manually start it. Is there any reason why this would only work some of the time, and if so what's the reason for the failures?

Thanks.

Azure Container Instances
Azure Container Instances
An Azure service that provides customers with a serverless container experience.
670 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,572 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Eduardo Tenente La Torre 1 Reputation point
    2020-11-20T23:23:55.9+00:00

    The same problem is happening very often for me.
    9 out of 10 containers failed to start today.

    In my case, images are coming from dockerhub. 

    Microsoft.Rest.Azure.CloudException: Long running operation failed with status 'Failed'. Additional Info:''     at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.CheckForErrors()     at Microsoft.Rest.ClientRuntime.Azure.LRO.PutLRO`2.CheckForErrors()     at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.<StartPollingAsync>d__20.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at Microsoft.Rest.ClientRuntime.Azure.LRO.AzureLRO`2.<BeginLROAsync>d__15.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at Microsoft.Rest.Azure.AzureClientExtensions.<GetLongRunningOperationResultAsync>d__1`2.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at Microsoft.Rest.Azure.AzureClientExtensions.<GetLongRunningOperationResultAsync>d__0`1.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at Microsoft.Rest.Azure.AzureClientExtensions.<GetPutOrPatchOperationResultAsync>d__5`1.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()     at Microsoft.Azure.Management.ContainerInstance.Fluent.ContainerGroupsOperations.<CreateOrUpdateWithHttpMessagesAsync>d__8.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()     at Microsoft.Azure.Management.ContainerInstance.Fluent.ContainerGroupsOperationsExtensions.<CreateOrUpdateAsync>d__3.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()     at Microsoft.Azure.Management.ContainerInstance.Fluent.ContainerGroupImpl.<CreateResourceAsync>d__38.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()     at Microsoft.Azure.Management.ResourceManager.Fluent.Core.ResourceActions.Creatable`4.<Microsoft-Azure-Management-ResourceManager-Fluent-Core-ResourceActions-IResourceCreator<IResourceT>-CreateResourceAsync>d__15.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()     at Microsoft.Azure.Management.ResourceManager.Fluent.Core.DAG.CreatorTaskItem`1.<ExecuteAsync>d__6.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.GetResult()     at Microsoft.Azure.Management.ResourceManager.Fluent.Core.DAG.TaskGroupBase`1.<ExecuteNodeTaskAsync>d__14.MoveNext()  --- End of stack trace from previous location where exception was thrown ---     at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()     at Microsoft.Azure.Management.ResourceManager.Fluent.Core.Extensions.Synchronize[TResult](Func`1 function)     at Microsoft.Azure.Management.ResourceManager.Fluent.Core.ResourceActions.Creatable`4.Create()
    0 comments
  2. Mike Urnun 9,781 Reputation points Microsoft Employee
    2020-11-24T22:43:39.187+00:00

    I haven't heard from the OP in private messages and it's a bit difficult to diagnose the root cause of an intermittent issue without deeper insights and more details. If your Azure Subscriptions carry a support plan, I recommend opening a support case and working with a support engineer in a 1:1 setting to troubleshoot the root cause. If not, you can reach out to me at azcommunity at microsoft dotcom with subject line "Attn: murnun" and I'll be able to enable your subscription with a one-time free support case for resolving this issue -- be sure to include your subscription ID in your email.

    Also, If it helps, we've had reports of ClamAV image not running reliably and switching to different image resolving issues, i.e: https://video2.skills-academy.com/en-us/answers/questions/127797/vnet-intermittent-connection-problem-between-funct.html?childToView=167357#answer-167357

    0 comments