Pods in status Terminating and Job not deleted after a Security Update

Roberto Sarati 5

I've a AKS cluster composed by 2 nodes. The version of k8s is 1.29.6.

Every week, during Sunday, I've scheduled a Security update (Node security channel type = Node Image).

On the cluster there are a lot of cronjobs, few of them scheduled every 1 minute.

It happens that, for some reason, during nodes update the running scheduled job doesn't complete, the pod stays in "Terminating" status and no more pods are created because the job is not deleted.

Now I've manually deleted the job and:

cronjob is now able to run the jobs as expected
I still have a Terminating Pod I cannot delete (every if I specify wait=false, force, grace-period parameters)

So... how I can avoid this situation (and how I can delete these pods?).

Goncalo Correia 351 Reputation points Microsoft Employee

2024-08-05T08:49:15.0633333+00:00

Hey @Roberto Sarati

Thanks for posting your question here in Microsoft Q&A,

The hanging of terminating pods can be due to multiple factors, usually removing them from the API is the job of the Garbage Collection mechanism
https://kubernetes.io/docs/concepts/architecture/garbage-collection/

If its working as expected, you should not need to manually remove any pods from the API,
I suggest you start reviewing the node kubelet and containerd logs, it could be that some operation (e.g. unmounting volumes or similar) is preventing the actual containers from the pod to be removed from the node and the hanging Terminating pods in the API are a symptom of that,
Nikhil Duserla 2,665 Reputation points Microsoft Vendor

2024-08-07T06:45:54.99+00:00

Hi @Roberto Sarati,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Nikhil Duserla 2,665 Reputation points Microsoft Vendor

2024-08-09T03:54:05.6333333+00:00

Hi @Roberto Sarati,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Roberto Sarati 5 Reputation points

2024-08-10T15:25:40.6333333+00:00
Hi Goncarlo,

thanks for the reply. Main problem is that during the weekly scheduled node update (not done by myself), a job was scheduled (scheduled), pod assigned to the node, probably the node never run, the job stay up and the cronjob wasn't rescheduled anymore.

Now I've deleted the job manually (so the cronjob restarted creating jobs).

I cannot delete the job in status "Terminating" using kubectl delete podname --force command. If I stop this command and run kubectl get --raw "/api/v1/nodes/nodename/proxy/logs/messages" there's no message about what's wrong with pod deletion. So, my problems are:

how to I delete this "terminating" pod (or I do I get additional logs, can you suggest a procedure)

how do I prevent this the next time aks cluter updates itself.
Nikhil Duserla 2,665 Reputation points Microsoft Vendor

2024-08-13T11:01:26.9466667+00:00

Hi @Roberto Sarati,

Thank you for reaching us again.

Can you please try the command provided below and let us know if it works?

kubectl delete pod <pod_name> --grace-period=0 --force

terminationGracePeriodSeconds: 30

Preventing this issue in the future please refer to these: https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#jobs-and-cronjobs

If you have any further queries, do let us know.
Roberto Sarati 5 Reputation points

2024-08-14T11:51:13.63+00:00

Hi,

the command replies with

"Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.

pod "nome_del_pod-28712171-47kgg" force deleted"

but the command never terminates. And when I stop it (CTRL+C) the pod is still in the list (status terminating, and it is not possible to see the logs)
Nikhil Duserla 2,665 Reputation points Microsoft Vendor

2024-08-16T09:26:33.4433333+00:00

Hi @Roberto Sarati,

Thank you for reaching us again.

I see you are still facing the issue. I recommend using the following command:

kubectl patch pod <pod> -p '{"metadata":{"finalizers":null}}'

Please refer to these https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/#force-deletion

If you have any further queries, do let us know.
Roberto Sarati 5 Reputation points

2024-08-19T07:39:15.9833333+00:00

Hi,

The command returns following message:

The Pod "pod_name-28712171-47kgg" is invalid: metadata: Invalid value: "Burstable": Pod QoS is immutable

Roberto Sarati 5

Just to add additional info, here is what I have using "kubectl describe pod" command:

Name:                      ##name##-28712171-47kgg
Namespace:                 #namespace#
Priority:                  0
Service Account:           default
Node:                      aks-agentpool-22955496-vmss000002/10.224.0.100
Start Time:                Sun, 04 Aug 2024 02:11:00 +0200
Labels:                    batch.kubernetes.io/controller-uid=d74f59eb-0bd7-4182-ab5b-f7e7cfa3c841
                           batch.kubernetes.io/job-name=##name##-28712171
                           controller-uid=d74f59eb-0bd7-4182-ab5b-f7e7cfa3c841
                           job-name=##name##-28712171
Annotations:               <none>
Status:                    Terminating (lasts 15d)
Termination Grace Period:  0s
IP:
IPs:                       <none>
Controlled By:             Job/##name##-28712171
Containers:
  ##name##:
    Container ID:   containerd://dc1a7940da88bf0ad9db9213f0428c769f0a9e07b6d4e7433e3094cc8099909d
    Image:          crrepo.azurecr.io/scrapers/##name##:latest
    Image ID:       crrepo.azurecr.io/scrapers/##name##@sha256:0e0bdedba85095d037f105069cbe15db0c9d42dd19a6d21e7f2b46c13aa387ce
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Sun, 04 Aug 2024 02:11:01 +0200
      Finished:     Sun, 04 Aug 2024 02:11:16 +0200
    Ready:          False
    Restart Count:  0
    Environment:
      DOTNET_ENVIRONMENT:                                Production
      K8S__Settings__ConnectionString:                   <set to the key 'connectionString' in secret 'db-secret'>             Optional: false
      K8S__Serilog__WriteTo__1__Args__connectionString:  <set to the key 'connectionString' in secret 'app-insights-secret'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-94cs9 (ro)
Conditions:
  Type                        Status
  DisruptionTarget            True
  PodReadyToStartContainers   False
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  kube-api-access-94cs9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>Name:                      ##name##-28712171-47kgg
Namespace:                 dataingestion
Priority:                  0
Service Account:           default
Node:                      aks-agentpool-22955496-vmss000002/10.224.0.100
Start Time:                Sun, 04 Aug 2024 02:11:00 +0200
Labels:                    batch.kubernetes.io/controller-uid=d74f59eb-0bd7-4182-ab5b-f7e7cfa3c841
                           batch.kubernetes.io/job-name=##name##-28712171
                           controller-uid=d74f59eb-0bd7-4182-ab5b-f7e7cfa3c841
                           job-name=##name##-28712171
Annotations:               <none>
Status:                    Terminating (lasts 15d)
Termination Grace Period:  0s
IP:
IPs:                       <none>
Controlled By:             Job/##name##-28712171
Containers:
  ##name##:
    Container ID:   containerd://dc1a7940da88bf0ad9db9213f0428c769f0a9e07b6d4e7433e3094cc8099909d
    Image:          crrepo.azurecr.io/scrapers/##name##:latest
    Image ID:       crrepo.azurecr.io/scrapers/##name##@sha256:0e0bdedba85095d037f105069cbe15db0c9d42dd19a6d21e7f2b46c13aa387ce
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Sun, 04 Aug 2024 02:11:01 +0200
      Finished:     Sun, 04 Aug 2024 02:11:16 +0200
    Ready:          False
    Restart Count:  0
    Environment:
      DOTNET_ENVIRONMENT:                                Production
      K8S__Settings__ConnectionString:                   <set to the key 'connectionString' in secret 'db-secret'>             Optional: false
      K8S__Serilog__WriteTo__1__Args__connectionString:  <set to the key 'connectionString' in secret 'app-insights-secret'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-94cs9 (ro)
Conditions:
  Type                        Status
  DisruptionTarget            True
  PodReadyToStartContainers   False
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  kube-api-access-94cs9:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Nikhil Duserla 2,665 Reputation points Microsoft Vendor

2024-08-19T15:46:17.56+00:00

Hi @Roberto Sarati,

Thank you for sharing the information.

We are testing it and will get back to you soon.

1 answer

Nikhil Duserla 2,665 Reputation points Microsoft Vendor

2024-08-21T08:20:33.06+00:00

Hi @Roberto Sarati,

Thank you for sharing the information.

It seems that the issue might be due to the pod you’re trying to delete having a QoS class set to Burstable. Kubernetes automatically assigns a QoS class based on the resource requests and limits when the pod is created, and this QoS class cannot be changed later.

To resolve this, please update the pod’s QoS class to BestEffort before attempting to delete it. This adjustment should help you avoid the immutability error.

If you have any further queries, do let us know.
Please sign in to rate this answer.
Nikhil Duserla 2,665 Reputation points Microsoft Vendor

2024-08-22T10:34:20.4133333+00:00

Hi @Roberto Sarati,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Roberto Sarati 5 Reputation points

2024-08-26T15:47:16.17+00:00

Hi,

I've tried changing qos of the pod, also using kubectl edit #podname# but the value doesn't change. Any suggestion?

vipullag-MSFT 26,316 Reputation points

2024-08-28T01:41:41.02+00:00

Hello Roberto Sarati

Support team will be able to check and help on this. I would recommend you to open a azure support case.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Pods in status Terminating and Job not deleted after a Security Update

1 answer

Your answer