Azure Container Registry
An Azure service that provides a registry of Docker and Open Container Initiative images.
461 questions
Private ACR with AKS: Failed to pull image... failed to resolve reference... unexpected status from HEAD request... 403 Forbidden
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 65%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELD%3C/text%3E%3C/svg%3E)
LyTien Dung
5
Reputation points
The kubelet try to pull an image from a private ACR (role assignment ArcPull granted for the -agentpool managed identity).
It gets an error like Failed to pull image... failed to resolve reference... unexpected status from HEAD request... 403 Forbidden
I think the network is ok, but if there is something missing about the permission on ACR private endpoint configuration.
In the cluster, using a pod to test network with ACR FQDN and private IP. I get these information:
nc -zv tempregistrydev.azurecr.io 443
Connection to tempregistrydev.azurecr.io (10.5.144.6) 443 port [tcp/https] succeeded!
telnet tempregistrydev.azurecr.io 443
Connected to tempregistrydev.azurecr.io
But, this ICMP ping is not passed
ping 10.5.144.6 PING 10.5.144.6 (10.5.144.6) 56(84) bytes of data.
Here are results after running Troubleshoot private link DNS configuration and connectivity issues
Have you provided a valid private endpoint resource?
Yes
Is your private endpoint in status 'Succeeded'?
Yes
What's the status of the private endpoint connection?
Approved
How are you trying to connect to the private endpoint from the client resource?
FQDN
What result do you get when resolving the FQDN?
Private IP
Run TCP pings to the FQDN or the private IP address and the corresponding TCP port. Is the TCP ping working?
Yes
Need further assistance If you have followed this troubleshooting guidance and reached this point, everything seems properly configured.
{count} votes
-
Prrudram-MSFT 24,916 Reputation points2024-08-02T13:31:41.99+00:00 Hello @LyTien Dung
Thank you for reaching out to the Microsoft Q&A platform.It seems like you are trying to pull an image from a private Azure Container Registry (ACR) using a Kubernetes cluster, but you are getting a "403 Forbidden" error. You have tested the network connection to the ACR using a pod and it seems to be working fine, but you are not able to ping the ACR's private IP address.
Based on the information you have provided, it seems like the private endpoint for the ACR is properly configured and the network connection to the ACR is working fine. However, you are still getting a "403 Forbidden" error when trying to pull the image from the ACR. One possible reason for this issue is that the managed identity used by the AKS cluster to access the ACR does not have the necessary permissions to pull the image. To resolve this issue, you can try granting the
AcrPullrole to the managed identity used by the AKS cluster.Here are the steps to grant the
AcrPullrole to the managed identity used by the AKS cluster:- Open the Azure portal and navigate to the ACR resource.
- Click on "Access keys" in the left-hand menu.
- Click on "Add role assignment" and select the "AcrPull" role.
- In the "Select" field, search for the name of the managed identity used by the AKS cluster.
- Click on "Save" to grant the role assignment. After granting the
AcrPullrole to the managed identity used by the AKS cluster, try pulling the image again and see if the issue is resolved.
I hope this helps! Let me know if you have any further questions.
Sign in to comment
Sign in to answer