How to fix Azure App Gw Backend Health Alerts?

Murali R 225

Hi @KapilAnanth-MSFT I am seeking clarification regarding a recurring issue with unhealthy backend alerts. Despite the backend being healthy, I am consistently receiving these alerts. Upon investigation, I noticed that the alerts indicate deviations ranging from 3.33ms to 946.1ms, depending on the backend pool and settings.

I need to understand more about this deviation and the underlying reasons why these alerts are being triggered, especially since the primary condition for these alerts is supposed to be an unhealthy backend.

KapilAnanth-MSFT 38,971 Reputation points Microsoft Employee

2024-06-25T13:31:16.91+00:00
@Murali R ,

Welcome to the Microsoft Q&A Platform. Thank you for reaching out & I hope you are doing well.

Can you please provide more details on how this alert is configured?

What base condition are you using here to validate if the backend is unhealthy ?

What is the threshold value you are setting?

What Metrics signal are you using?

For the issue triggering time, under App gateway Metrics - what are you seeing for Unhealthy threshold count?

Is it dipping compared to average?

Cheers,

Kapil
Murali R 225 Reputation points

2024-06-25T13:46:41.4533333+00:00
Hi @KapilAnanth-MSFT We have used Azure Monitor to Configure this Application Gateway Backend Health Alerts

What base condition are you using here to validate if the backend is unhealthy ? - Health Probe and the backend health will be checked, if the backend shows unhealthy this alert needs to be triggered

What is the threshold value you are setting? - Threshold value has been set to '0'

What Metrics signal are you using? - UnhealthyHostCount

For the issue triggering time, under App gateway Metrics - what are you seeing for Unhealthy threshold count? - It varies from 3.46m to 946.1m which depends on the backend pool and backend settings

Is it dipping compared to average? - We have used Static Alert
KapilAnanth-MSFT 38,971 Reputation points Microsoft Employee

2024-06-25T16:54:04.3966667+00:00
@Murali R ,

Thanks for sharing the details.

Can you share a screenshot of the all the conditions you have defined in the Alerts logic?

I can see for "UnhealthyHostCount",

You might have followed something similar to Alert for average Unhealthy Host Count

And setting the condition to "Less than or equal to 0" would trigger an alert.

Also, when you say "It varies from 3.46m to 946.1m"

"UnhealthyHostCount" is an average value

But you mention difference in time

But there should be difference in number of unhealthy backend units

Please share a screenshot of the entire configuration

But I am not sure how you have configured for "Health Probe"

As I see, a single condition can have only one single Signal

So, do you have multiple conditions?

And the second condition validates health probe?

Please share a screenshot of the entire configuration, and query used if any.

Is there any document you followed to set up this alert?

If so, please share the link as well

Cheers,

Kapil
Murali R 225 Reputation points

2024-06-25T17:42:35.7066667+00:00

Hi @KapilAnanth-MSFT I have followed Alert Configuration as per MS Suggestion. Kindly find the below link for your reference
https://video2.skills-academy.com/en-us/answers/questions/1605218/azure-monitor-alert-automitigate-issue. Regards to Health Probe the configuration, we have configured with Single condition with Single Signal only, kindly find the below screenshot FYR

Alert Configuration:
KapilAnanth-MSFT 38,971 Reputation points Microsoft Employee

2024-06-26T15:01:50.4766667+00:00
@Murali R ,

Thanks for sharing the details.

From this alert, where are you getting the values "3.46m to 946.1m"

I don't see it being configured in the Alerts

Cheers,

Kapil
Murali R 225 Reputation points

2024-06-26T15:10:21.25+00:00

Hi @KapilAnanth-MSFT Kindly find the below fired alert FYR
KapilAnanth-MSFT 38,971 Reputation points Microsoft Employee

2024-06-26T16:34:24.26+00:00
Thanks @Murali R ,

It is strange that the value jumps from 0 to such a huge value.

As next steps,

You should validate that the above metric is actually correct.

You can do this by navigating to the App Gw ----> Metrics and setting the below for the alert triggered/fired time

If you see the same graph spike, this indicates that the alert fired for correct reason.

Similarly, you can do the same for "HealthyHostCount"

It should be opposite for what you saw in "UnhealthyHostCount"

i.e., there should be a dip in the HealthyHostCount.

Can you please share screenshots for "HealthyHostCount" and "UnhealthyHostCount"

Also, please note that Diagnostic logs for Application Gateway does not currently include healthprobe logs.

So if the metrics is actually correct (HealthyHostCount dips and UnhealthyHostCount raises), you will be required to create a Support Incident to troubleshoot this further

Cheers,

Kapil

Share via

How to fix Azure App Gw Backend Health Alerts?