Requests from Azure appService to containerApps get blocked on custom VNET with NSG

I asked the sam equestion on stack overflow. Pls remove if this is not the place for ist: https://stackoverflow.com/questions/75243527/requests-from-azure-appservice-to-containerapps-get-blocked-on-custom-vnet-with

I am trying to set up an Infrastructure that "freely" allows traffic between all subnets of the vnet but block all external ingress traffic other than from the IP associated to our VPN / office.

The (reduced) setup consists of:

1. A NetworkSecurityGroup with a single allow rule of our office IP
2. A VNET with 2 subnets, each of which has the networkSecurityGroup assigned to it
3. An AppService that hosts a REST API in a DockerContainer (is assigned to subnet1)
4. A ManagedEnvironment for containerApps (is assigned subnet2)
5. A ContainerApp associated with the ManagedEnvironment

These are the relevant parts in bicep:

Vnet & Network Security Group (in full)

var myIpSecRule = {
  name: 'allowMe'
  properties: {
    access: 'allow'
    direction: 'Inbound'
    protocol: '*'
    sourceAddressPrefix: 'XX.XXX.XXX.XXX' // our IP
    sourcePortRange: '*'
    destinationAddressPrefix: '*'
    destinationPortRange: '*'
    priority: 900
  }
}

resource netSecurityGroup 'Microsoft.Network/networkSecurityGroups@2021-05-01' = {
  location: location
  name: '${baseName}-netsecgrp'
  properties: {
    securityRules: [
      myIpSecRule
    ]
  }
}

resource virtualNetwork 'Microsoft.Network/virtualNetworks@2021-05-01' = {
  name: '${baseName}-vnet1'
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.0.0.0/8'
      ]
    }
    subnets: [
      {
        name: '${baseName}-subnet1' // subnet for AppService
        properties: {
          addressPrefix: '10.0.0.0/16'
          serviceEndpoints: [
            {
              service: 'Microsoft.Web'
            }
          ]
          networkSecurityGroup: {
            id: netSecurityGroup.id
          }
          delegations: [
            {
              name: 'Microsoft.Web/serverFarms'
              properties: {
                serviceName: 'Microsoft.Web/serverFarms'
              }
            }
          ]
        }
      }
      {
        name: '${baseName}-subnet2' //subnet for containerAppEnv
        properties: {
          addressPrefix: '10.1.0.0/16'
          networkSecurityGroup: {
            id: netSecurityGroup.id
          }
        }
      }
    ]
  }

  resource subnet1 'subnets' existing = {
    name: '${baseName}-subnet1'
  }
  resource subnet2 'subnets' existing = {
    name: '${baseName}-subnet2'
  }
}

AppService (parts relating to vnet/subnet/routes only)

resource appServicePlan 'Microsoft.Web/serverfarms@2021-03-01' = {
  name: serviceHostingPlanName
  location: location
  ... // sku etc
}

resource serviceApp 'Microsoft.Web/sites@2022-03-01' = {
  name: serviceAppName
  location: location
  properties: {
    siteConfig: {
      appSettings: [
        ... // more stuff
        {
          name: 'WEBSITES_PORT'
          value: '8000'
        }
      ]
      linuxFxVersion: 'DOCKER|${dockerRegistryHost}/${dockerImageName}'
      vnetRouteAllEnabled: true
    }
    serverFarmId: appServicePlan.id
    virtualNetworkSubnetId: virtualNetwork::subnet1.id
    vnetRouteAllEnabled: true
    vnetContentShareEnabled: true
  }
}

ContainerApp part (parts relating to vnet/subnet/routes only)

resource env 'Microsoft.App/managedEnvironments@2022-03-01' = {
  name: envName
  location: location
  properties: {
    appLogsConfiguration: {
     ...
    }
    vnetConfiguration: {
      internal: false
      infrastructureSubnetId: subnetId
    }
  }
}

resource containerApp 'Microsoft.App/containerApps@2022-03-01' = {
  name: containerAppName
  location: location
  properties: {
    managedEnvironmentId: envId
    configuration: {
      ... // secrets, registries
      ingress: {
        external: true
        targetPort: 8000
        allowInsecure: true // really just to make sure this is not the issue
      }
    }
    template: {
      containers: [...]
      }
    }
  }
}

The routes I am using are:

• appServiceRoute = 'https://${serviceApp.properties.defaultHostName}/'
• containerAppRoute = 'https://${containerApp.properties.configuration.ingress.fqdn}/'

What I observe is this:

• When issuing requests to any of the routes from my local machine (using the IP defined in myIpSecRule) everything works fine
• Issuing requests from any other IP times out (as expected)
• Issuing requests to appServiceRoutefrom within my containerApp works fine
• Issuing requests to containerAppRoute from within my AppService times out

I assumed that due to the automatically created rule that allows vnet-vnet traffic communicationn between the two would not pose a problem.

To investigate further I:

• Started a ssh console for the AppService and curled the containerAppRoute -> still timeout
• Switched to http:// -> still timeout
• Set everything I could find to route all traffic from the AppService through the Vnet/subnet (WEBSITE_VNET_ROTE_ALL=1 in AppSettings, vnetRouteAllEnabled: true, vnetContentShareEnabled: true) -> still timeout
• Added an ingress rule to allow all traffic from anywhere to the containerApp -> works as expected
• Removed the IP whitelisting NSG from the VNET (but dont actively add an ingress-allow-all rule) -> works as expected

From that I take that either the requests from the appService are not recognized as coming from within the vnet or something in the whole context of managedEnvironment/customVnet/NSG blocks traffic from subnets in the same vnet as well.

I am quite sure that I am missing something fundamental concerning VNETs in Azure given that thats far from my everyday profession but I have been trying for days and I am out of ideas... Any help would be greatly appreciated!