Issue with using Azure API Management Connection Manager to retrieve tokens

We are encountering a BackendConnectionFailure Error while trying to use the policy named ‘get-authorization-context’ with the Azure API Management (APIM) Connection Manager to retrieve tokens. We have verified the firewall & NSG connectivity between APIM and Credential manager on port 443, still it is failing with connectivity issue.

Here is the error message we receive:

get-authorization-context (49.623 ms)
{
    "messages": [
        "An error occurred while sending the request.",
        "The underlying connection was closed: An unexpected error occurred on a send.",
        "Error occured while calling backend service.",
        "Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.",
        "Error occured while calling backend service.",
        "An existing connection was forcibly closed by the remote host"
    ]
}