connect to azure keyvault that has a private endpoint and the firewall enabled

We are trying to read and update values in keyvault from a container that both are connected to the same virtual network. The keyvault is connected using a private endpoint and the other services in the network can interact fine with the service. we have setup the private dns zones, a records etc to make sure the keyvault urls resolve to the local ip address.

an overview of what we are trying to achieve is here:

when we run the commands we are getting an error back stating that the ip used by the container isn't allowed by the network rules of keyvault. When we look at the IP it is a public azure IP address (we didn't assign a public ip address to the container instance). Because the containers are connected to a vnet we expected the ip address to be something internal. i have seen this documentation:
https://video2.skills-academy.com/en-us/azure/container-instances/container-instances-egress-ip-address but that is for a public address.

but this is for a public IP address and deploying a firewall just for internal network routing seems a bit overkill to me. is there any guidance to get this working?