BUG in Azure Arc connected Kuberentes and App Service

Gábor Szabó 5

Hi,

I had a test environment where I deployed my own Kubernetes cluster, connected it with Azure Arc, deployed the AppService extension and created an AppService on it. Everything worked nicely.

When I finished my work, I simply deleted the resource group and my "App Service Kubernetes Environment" resource stuck there. The error message says: "Failed to delete Kubernetes environment <NAME>: The specified kube environment <NAME> cannot be deleted because it still contains resources."

Of course there are not AppServices attached to this resource and the K8s cluster's extension is also detached. No Apps and neither no App Service plans attached to the resource. So it is empty.

In the JSON, the extededLocation still refers to the custom location what I created but that is already deleted when I triggered the whole resource group deletion.

I tried to delete the resource via Azure CLI (az resource delete) as well and I got the same error message.

This happened with me second time. First time I rebuilt the components and I deleted manually in the right order. So that will be the workaround here as well.

I guess this is a bug as it didn't keep the dependency order during deletion. I wanted to report this it via the support channel but MS asks money for that hence I drop it here. Maybe a relevant person can translate it to the right place :)

Ryan Hill 27,771 Reputation points Microsoft Employee

2024-08-09T16:45:02.7433333+00:00

Thanks for posting @Gábor Szabó. In order to best assist you, I'll send you a private message to gain additional information.
Gábor Szabó 5 Reputation points

2024-08-09T17:40:38.1333333+00:00
Hi Ryan,

Thank for picking up. I just deleted the resource so I cannot give you exact info but here are the steps to reproduce:

Create a Kubernetes cluster (I did it outside of Azure)

Connect it with Azure Arc

Deploy the AppService extension (with Custom location and App Kube Environment)
FYI, I deployed these with CLI according to the documentation.

Deploy an AppService*

Turn off the cluster*

Delete the entire resource group

*I'm not sure if these steps are needed.

All resource (including the test AppService and its AppService Plan) will be gone except the App Kube Environment.

Likely the root cause:

The App Kube Environment (AKE) is attached to the Custom location.

The Custom location is NOT attached to the AKE

When I tried to delete the alone AKE, I got the "...it still contains resources." error.

I guess it tries to check the AppServices via the Custom location which has been gone already.

There are 0 AppService Plan and 0 AppService attached to the AKE.

It is likely that the RG deletion took the Custom location first which led to deadlock of AKE.

Final root cause: missing dependency between the AKE and the Custom Location

Workaround:

Create a K8s cluster

Attach it with Azure Arc

Deploy the AppService but only with the Custom Location

Delete the AKE. --> It will see the Custom Location resource and then it happily deletes itself :)

Delete the rest of the resources.

Reasonable corrections:

Add the dependency between the AKE and Custom Location. Don't allow to delete the Custom Location until an AKE refers to it.

OR allow to delete the AKE with the missing Custom Location. --> I think this is just a missing exception handling in the code and that causes this bug.

Finally check if it is working and send a beer to Gabor :D:D:D
Ryan Hill 27,771 Reputation points Microsoft Employee

2024-08-15T14:33:53.3333333+00:00

Thank you for the additional insight @Gábor Szabó. We'll continue to investigate this issue offline to determine the root cause.

1 answer

Ryan Hill 27,771 Reputation points Microsoft Employee

2024-09-02T01:08:16.8766667+00:00

Hi @Gábor Szabó

Thank you for sharing your repro steps and working offline with us. the issue has been resolved. Essentially what happened is that farm/website resources that were created in the kubeEnvironment were left over in reference sources. Those resources have been removed and the environment can be deleted. According to logs, it was resources from a while ago.

Rather than deleting the resource group, delete the resources created in the kubeEnvironment first then delete the cluster. I also suggest using Container Apps on Azure Arc Overview | Microsoft Learn rather than using app service on Arc.
Please sign in to rate this answer.
Gábor Szabó 5 Reputation points

2024-09-02T06:43:45.1566667+00:00

Hi Ryan,

Thank you for the feedback and it is in-line with my root cause analysis.

As the RG deletion is a standard method hence I still think this is a bug as those reference resources shall be deleted first and the dependences are missing. Average users should not be aware such deletion order but it is expected from Azure to handle it.

Please note that, I re-created the environment for you and the hanging resource what you refer was 1 day old only so this isn't a long forgotten item.

Indeed, for my use-case the Container Apps also fits. Nevertheless, my intention was to report a bug in AppService.

Ryan Hill 27,771 Reputation points Microsoft Employee

2024-09-03T19:04:11.2966667+00:00

Thanks @Gábor Szabó for your continued patience. I've informed the product group of this issue. If I get any sort of timeline, I'll update this comment. In the meantime, please feel free to mark this answered to benefit the broader community.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

BUG in Azure Arc connected Kuberentes and App Service

1 answer

Your answer