APIM in Internal VNet Mode with User Assigned Managed Identity and Access to Key Vault

Hi Guys

I'm dealing with a requirement to deploy Azure API Management instance in "Internal" VNet mode through Bicep. It is assigned a User Assigned Managed Identity during the deployment. The APIM needs to access a SSL Certificate from a Key Vault that has Public Access disabled and Private Endpoint attached to it. Service Endpoint is enabled for Azure Key Vault, Azure Storage and Event Hubs on the APIM Subnet object. APIM's User Assigned MI is configured with the required key vault roles (Key Vault Certificate User) as described here--> https://video2.skills-academy.com/en-us/azure/api-management/api-management-howto-use-managed-service-identity

However, the deployment throws the following error message:

ERROR: {"status":"Failed","error":{"code":"DeploymentFailed","target":"/subscriptions/<Subscription ID>/resourceGroups/<RG Name>/providers/Microsoft.Resources/deployments/deployment1","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.","details":[{"code":"InvalidOperation","message":"Failed to access KeyVault Secret https://<KV Name>.vault.azure.net/secrets/<Secret Name>/<Secret ID> using Managed Service Identity of Api Management service. Check if Managed Identity of Type: UserAssigned, ClientId: <client ID> and ObjectId: <Object ID> has GET permissions on secrets in the KeyVault Access Policies."}]}}##[error]Script failed with exit code: 1

I see others reporting the exact same scenario and same issue here:

https://stackoverflow.com/questions/70924786/azure-apim-how-to-obtain-ssl-certificate-from-key-vault-with-user-assigned-mana

I've referenced these articles :

https://video2.skills-academy.com/en-us/azure/key-vault/general/network-security

States that "When you enable the Key Vault Firewall, you'll be given an option to 'Allow Trusted Microsoft Services to bypass this firewall.'"

https://video2.skills-academy.com/en-us/azure/key-vault/general/overview-vnet-service-endpoints

Suggests that Azure APIM is considered as "Trusted" service.

Question 1: From the deployment results so far, it seems like APIM is only considered as "Trusted" when it is deployed using System Assigned MI and not User Assigned MI. Is this the correct understanding?

Question 2: As we're doing the APIM deployment and SSL cert configuration (through Azure Key Vault) using Bicep and System Assigned MI for APIM will only be created after it is deployed. Wondering what is the best way (and sequence of events) to deploy APIM in such a configuration? Of course, we want to keep the Azure Key Vault Firewall ON?

Is this the right sequence (as suggested by the stackoverflow article)?

1.     Create the APIM with System Assigned MI with no hostnameConfigurations [] i.e., no custom domain name configuration.

2.     Assign System Assigned MI the correct Key Vault role to retrieve the SSL Certificate

3.     Re-run the APIM deployment with hostnameConfigurations [] i.e., no custom domain name configuration.

Is there a potential of anything being overwritten / deleted when we re-deploy the APIM at step # 3?

Can someone provide some guidance on this please?

Thanks in advance

Taranjeet Singh