Troubleshoot connection issues to pods or services within an AKS cluster (internal traffic)

Article
09/05/2024

This article discusses how to troubleshoot connection issues to pods or services as internal traffic from within the same Microsoft Azure Kubernetes Services (AKS) cluster.

Prerequisites

The Kubernetes kubectl tool, or a similar tool to connect to the cluster. To install kubectl by using Azure CLI, run the az aks install-cli command.
The apt-get command-line tool for handling packages.
The Client URL (cURL) tool, or a similar command-line tool.
The Netcat (nc) command-line tool for TCP connections.

Troubleshooting checklist

Step 1: Set up the test pod and remote server port

Set up the test pod and make sure that the required port is open on the remote server. From within the source pod (or a test pod that's in the same namespace as the source pod), follow these steps:

Start a test pod in the cluster by running the kubectl run command:

kubectl run -it --rm aks-ssh --namespace <namespace> --image=debian:stable

After you gain access to the pod, run the following apt-get commands to install the DNS Utils, cURL, and Netcat packages:
```
apt-get update -y
apt-get install dnsutils -y
apt-get install curl -y
apt-get install netcat-openbsd -y
```
After the packages are installed, run the following cURL command to test the connectivity to the IP address of the pod:
```
curl -Iv http://<pod-ip-address>:<port>
```
Run the Netcat command to check whether the remote server opened the required port:
```
nc -z -v <endpoint> <port>
```

Step 2: View operational information about pods, containers, the Kubernetes services, and endpoints

Using kubectl and cURL at the command line, follow these steps to check that everything works as expected:

Verify that the destination pod is up and running:

kubectl get pods -n <namespace-name>

If the destination pod is working correctly, the pod status is shown as Running, and the pod is shown as READY.

NAME           READY   STATUS    RESTARTS   AGE
my-other-pod   1/1     Running   0          44m
my-pod         1/1     Running   0          44m

Search the pod logs for access errors:

kubectl logs <pod-name> -n <namespace-name>

Search the pod logs for an individual container in a multicontainer pod:
```
kubectl logs <pod-name> -n <namespace-name> -c <container-name>
```
If the application that's inside the pod restarts repeatedly, view pod logs of a previous container instance to get the exit messages:
```
kubectl logs <pod-name> --previous                      
```
For the multicontainer case, use the following command:
```
kubectl logs <pod-name> -c <container-name> --previous  
```
Check whether there are any network policies that might block the traffic:
```
kubectl get networkpolicies -A
```
You should see output that resembles the following table.
```
NAMESPACE     NAME                 POD-SELECTOR             AGE
kube-system   konnectivity-agent   app=konnectivity-agent   4d1h
```
If you see any other network policy that's custom-created, check whether that policy is blocking access to or from the pods.
Check whether you can reach the application from the service IP address. First, show details about the service resource, such as the external IP address and port, by running the kubectl get services command:
```
kubectl get services -n <namespace-name>
```
```
NAME         TYPE           CLUSTER-IP   EXTERNAL-IP      PORT(S)        AGE
my-service   LoadBalancer   10.0.21.43   20.119.121.232   80:31773/TCP   28s
```
Then, run cURL by using the service IP address and port to check whether you can reach the application:
```
curl -Iv http://20.119.121.232:80
.
.
.
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
```

Get more verbose information about the service:

kubectl describe services <service-name> -n <namespace-name>

Check the pod's IP address:

kubectl get pods -o wide

NAME            READY   STATUS        RESTARTS   AGE   IP            NODE                                
my-pod          1/1     Running       0          12m   10.244.0.15   aks-agentpool-000000-vmss000000

Verify that the pod's IP address exists as an endpoint in the service:

kubectl describe services my-cluster-ip-service

Name:              my-cluster-ip-service
Namespace:         default
Selector:          app=my-pod
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.0.174.133
IPs:               10.0.174.133
Port:              <unset>  80/TCP
TargetPort:        80/TCP
Endpoints:         10.244.0.15:80     # <--- Here

Verify the endpoints directly:

kubectl get endpoints

NAME                      ENDPOINTS           AGE
my-cluster-ip-service     10.244.0.15:80      14m

If the connection to a service doesn't work, restart the kube-proxy and CoreDNS pods:

kubectl delete pods -n kube-system -l component=kube-proxy
kubectl delete pods -n kube-system -l k8s-app=kube-dns

Verify that the node isn't overused:
```
kubectl top nodes
```
Note

You can also use Azure Monitor to get the usage data for the cluster.

Contact us for help

If you have questions or need help, create a support request, or ask Azure community support. You can also submit product feedback to Azure feedback community.

Share via