AKS 1.29, HPA unable to get metrics

Cloudy 206

Hi Folks !

I'm facing the following warnings on many hpa, "unable to get metrics"... cannot understand why, visibly i'm not the only one.

Do anybody have any clue on what to do in order to avoid that ?

Man thanks !

kubectl describe hpa apache-appl -n legacy
Name:                                                     apache-app
Namespace:                                                legacy
Labels:                                                   app.kubernetes.io/instance=legacy
                                                          app.kubernetes.io/managed-by=Helm
                                                          app.kubernetes.io/name=appl
                                                          app.kubernetes.io/part-of=app
                                                          app.kubernetes.io/version=0.1.0
                                                          helm.sh/chart=legacy-0.1.0
Annotations:                                              meta.helm.sh/release-name: legacy
                                                          meta.helm.sh/release-namespace: legacy
CreationTimestamp:                                        Wed, 10 Jul 2024 10:24:18 +0000
Reference:                                                Deployment/apache-app
Metrics:                                                  ( current / target )
  resource cpu on pods  (as a percentage of request):     2% (20m) / 95%
  resource memory on pods  (as a percentage of request):  57% (147651242666m) / 95%
Min replicas:                                             3
Max replicas:                                             10
Deployment pods:                                          3 current / 3 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  recommended size matches current size
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from memory resource utilization (percentage of request)
  ScalingLimited  True    TooFewReplicas    the desired replica count is less than the minimum replica count
Events:
  Type     Reason                        Age                 From                       Message
  ----     ------                        ----                ----                       -------
  Warning  FailedComputeMetricsReplicas  27m (x7 over 12h)   horizontal-pod-autoscaler  invalid metrics (1 invalid out of 2), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedGetResourceMetric       15m (x25 over 12h)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedGetResourceMetric       15m (x23 over 12h)  horizontal-pod-autoscaler  failed to get memory utilization: unable to get metrics for resource memory: no metrics returned from resource metrics API
  Warning  FailedComputeMetricsReplicas  15m (x18 over 12h)  horizontal-pod-autoscaler  invalid metrics (2 invalid out of 2), first error is: failed to get cpu resource metric value: failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API

Kasun Rajapakse 431 Reputation points

2024-08-05T18:10:25.8266667+00:00

Can you provide the YAML used for the HPA? This will help to understand the metric threshold

Cloudy 206

Hi @Kasun Rajapakse !

Please find the the hpa outputed as yaml, hope that is could help !

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    meta.helm.sh/release-name: legacy
    meta.helm.sh/release-namespace: legacy
  creationTimestamp: "2024-07-10T10:24:18Z"
  labels:
    app.kubernetes.io/instance: legacy
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: applegacy
    app.kubernetes.io/part-of: applegacy
    app.kubernetes.io/version: 0.1.0
    helm.sh/chart: legacy-0.1.0
  name: apache-applegacy
  namespace: legacy
  resourceVersion: "19604451"
  uid: 2b2e8c1c-d5a7-4523-ae92-e6a05e8373e5
spec:
  maxReplicas: 10
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 95
        type: Utilization
    type: Resource
  - resource:
      name: memory
      target:
        averageUtilization: 95
        type: Utilization
    type: Resource
  minReplicas: 3
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: apache-applegacy
status:
  conditions:
  - lastTransitionTime: "2024-07-10T10:24:34Z"
    message: recommended size matches current size
    reason: ReadyForNewScale
    status: "True"
    type: AbleToScale
  - lastTransitionTime: "2024-08-06T05:31:23Z"
    message: the HPA was able to successfully calculate a replica count from memory
      resource utilization (percentage of request)
    reason: ValidMetricFound
    status: "True"
    type: ScalingActive
  - lastTransitionTime: "2024-08-06T02:17:36Z"
    message: the desired replica count is less than the minimum replica count
    reason: TooFewReplicas
    status: "True"
    type: ScalingLimited
  currentMetrics:
  - resource:
      current:
        averageUtilization: 0
        averageValue: 9m
      name: cpu
    type: Resource
  - resource:
      current:
        averageUtilization: 58
        averageValue: "149065728"
      name: memory
    type: Resource
  currentReplicas: 3
  desiredReplicas: 3
  lastScaleTime: "2024-07-30T13:08:24Z"

Mahesh Goud Juvvadi 975 Reputation points Microsoft Vendor

2024-08-09T12:57:32.4333333+00:00

Hi Cloudy,

In your Kubernetes cluster, please verify whether the metrics server is operational for other workloads. Are the issues you're facing with the metrics server limited to this particular HPA, or are they impacting all workloads?

Thank you.
Cloudy 206 Reputation points

2024-08-12T07:15:19.4866667+00:00

Hi,

Yes it seems that this is the only one, what can i do to avoid that ? thanks
Mahesh Goud Juvvadi 975 Reputation points Microsoft Vendor

2024-08-12T11:28:56.53+00:00

Hi Cloudy,

Thank you for the response!

(HPA) is unable to get metrics for resource memory and CPU could be due to different reasons.

If the metrics server is not running or is not properly configured. The metrics server is responsible for collecting resource utilization data from the Kubernetes API server and making it available to other components, such as the HPA. You can check if the metrics server is running by running the command kubectl get pods -n kube-system | grep metrics-server.

The HPA is not properly configured to use the metrics server. You can check the HPA configuration by running the command kubectl describe hpa <hpa-name>

If it was helpful, please click "Upvote" on his post to let us know.

Thank You.
Cloudy 206 Reputation points

2024-08-12T12:07:18.8266667+00:00
Hi!

Metrics server is running but sometimes failing and the hpa config is present in the previous message, i don't know what i would be able to do because this is managed on AKS side:

metrics-server-64b6c75bb8-n2dgh 2/2 Running 0 3m4s metrics-server-64b6c75bb8-p5xml 2/2 Running 0 3m4s
Mahesh Goud Juvvadi 975 Reputation points Microsoft Vendor

2024-08-20T12:56:59.6566667+00:00

Hi Cloudy,

We are checking with our internal team and will update you soon.

Thank you.
Cloudy 206 Reputation points

2024-08-20T14:00:17.4066667+00:00

Thanks, waiting on your feedback, regards
Mahesh Goud Juvvadi 975 Reputation points Microsoft Vendor

2024-08-26T11:17:07.0633333+00:00

Hi Cloudy,

Thank you for your patience and understanding.

This issue needs deeper investigation and Azure technical support team will be able to help with this. I would recommend you open an azure support case. If you don't have the ability to open a technical support ticket, please let me know and I can help you further with this.

Share via

AKS 1.29, HPA unable to get metrics

Your answer