Azure Application Gateway to AKS internal load balancer using native Kubernetes NGINX controller

Hi@Duncan House,

We are looking into this issue to get more information related to your query and will get back to you as soon as we have an update.

Hi @Duncan House

Welcome to Microsoft Q&A Platform. Thank you for reaching out & hope you are doing well.

A 502 bad gateway indicates that the Application Gateway is not communicating with the Backend Pool:

1. Verify the IP address configured in the backend pool of the Application Gateway which matches the Internal Load balancer. Verify the Port number configured in the backend pool of the Application Gateway is the correct port that the Internal Load balancer using.
2. Verify the configuration of the health probe, whether the backend Internal Load balancer is responding to the health probe or not.
3. Verify the NSG whether it is blocking the access to the backend and also check the NSG's which are associated to the Application Gateway subnet and make sure it is not blocking the access to the backend.
4. Make sure that the backend pool of the Internal Load balancer contains the IP address of the NGINX.
5. Make sure that the listener on the Application Gateway is configured with the correct frontend port and protocol.
6. Verify the routing rules which are correctly pointing to the backend pool.
7. Please try to increase the request time out, which might resolve the issue. You can increase the time out in BackendHttpSetting of the Application Gateway or else you can use the PowerShell scripting New-AzApplicationGatewayBackendHttpSettings -Name 'Setting01' -Port 80 -Protocol Http -CookieBasedAffinity Enabled -RequestTimeout 60. For your reference: https://video2.skills-academy.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502#solution-3
8. The Request timeout maximum to private backend is 24 hours. For your reference: https://video2.skills-academy.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits#application-gateway-limits
9. Finally try to enable and review the AGW logs, you might get a reason for failing the request.Kindly let us know if the above helps or you need further assistance on this issue.

Thankyou for responding....

Going through the points 1 - 9 above...

1. BEP is set as the internal LB IP
2. Backend health is not responding; not sure how to fix this?
3. Image below.
4. Checked and aligned with point 1.
5. First test setup HTTP port 80
6. Checked. Seems OK
7. Increased, see below.

Further notes...

The AKS cluster is within it's own private vnet.
The internal LB is an IP within the same vnet.

The AppGW is a Public IP, trying to reach backend pool of internal LB.

I'm missing something very simple, but sadly I just cannot spot it.... any more help gratefully received.

Hi @Duncan House,

Looks like you have implemented App Gateway->Load Balancer->AKS

You can use AGIC (Application Gateway Ingress Controller) directly instead of Load Balancer here.

As documented here

 https://video2.skills-academy.com/en-us/azure/application-gateway/ingress-controller-overview

AGIC helps eliminate the need to have another load balancer/public IP address in front of the AKS cluster and avoids multiple hops in your DataPath before requests reach the AKS cluster. Application Gateway talks to pods using their private IP address directly and doesn't require Node Port or Kube Proxy services. This capability also brings better performance to your deployments.

Could you please tell us the reason why you are using App Gateway->Load Balancer->AKS in this way as you can use AGIC->AKS instead.

And as you mentioned Backend health is not responding,

502 errors can also be frequent indicators that the default health probe can't reach backend VMs.

If you are using a default health probe, you can refer to the below document and do the troubleshooting steps.

https://video2.skills-academy.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502#problems-with-default-health-probe

Please confirm whether you have created the health probe for backend pool. If not, please create HTTP settings first and then create a health probe by using the HTTP settings, after that you can test the backend pool health with the probe.

For your reference:

https://video2.skills-academy.com/en-us/azure/application-gateway/application-gateway-create-probe-portal#createprobe

If you are using a custom health probe, you can refer to the below document and do the troubleshooting steps.

https://video2.skills-academy.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502#problems-with-custom-health-probe

For Additional reference, please go through this document related to 502 gateway errors.

https://video2.skills-academy.com/en-us/azure/application-gateway/application-gateway-troubleshooting-502#overview

Kindly let us know if the above helps or you need further assistance on this issue.

Hi @Duncan House,

Following up to see if the above suggestion was helpful. And, if you have any further query kindly do let us know.