How to retrive APIM eventhub log from policy : log-to-eventhub

Amy Z 291 Reputation points
2021-01-18T11:06:52.727+00:00

Hi, we've followed this post to add request/response log to eventhub then capture to storage account, it works.
Related content can be found in *.avro from storage account, but the format is not user-friendly so we try to leverage existing Apache Nifi to directly read data from eventhub.

It's weird that we can read Gatewaylogs, metrics by Nifi , but no APIM request/response in it, so we try to inspect log analytics.

Since in policy setting we log request in partition 0 and response in partition 1, we use following script to check, but nothing in 0 and 1 partition.

SELECT
    *
INTO
    [OutputAlias]
FROM
    [evh-xxx]
WHERE PartitionId = 1

Really confused how the request/response log captured in storage account, we cannot read them from log analytics and Nifi.

Here's the config about event-hub and its namespace

//Event-hub namespace
{
    "sku": {
        "name": "Standard",
        "tier": "Standard",
        "capacity": 5
    },
    "id": "/subscriptions/12345678/resourceGroups/RG_xxx/providers/Microsoft.EventHub/namespaces/evhns-xxx",
    "name": "evhns-xxx",
    "type": "Microsoft.EventHub/Namespaces",
    "location": "West US 2",
    "tags": {},
    "properties": {
        "isAutoInflateEnabled": true,
        "maximumThroughputUnits": 20,
        "kafkaEnabled": true,
        "provisioningState": "Succeeded",
        "metricId": "12345678-3976-49e0-b333-389a5fdb7f7b:evhns-xxx",
        "createdAt": "2021-01-07T10:31:38.057Z",
        "updatedAt": "2021-01-07T10:32:21.077Z",
        "serviceBusEndpoint": "https://evhns-xxx.servicebus.windows.net:443/",
        "status": "Active"
    }
}
-------------------
//Event-hub 

{
    "id": "/subscriptions/12345678/resourceGroups/RG_xxx/providers/Microsoft.EventHub/namespaces/evhns-xxx/eventhubs/evh-xxx",
    "name": "evh-xxx",
    "type": "Microsoft.EventHub/Namespaces/EventHubs",
    "location": "West US 2",
    "properties": {
        "messageRetentionInDays": 7,
        "partitionCount": 20,
        "status": "Active",
        "createdAt": "2021-01-07T10:49:15.263",
        "updatedAt": "2021-01-18T08:28:26.973",
        "partitionIds": [
            "0",
            "1",
            "2",
            "3",
            "4",
            "5",
            "6",
            "7",
            "8",
            "9",
            "10",
            "11",
            "12",
            "13",
            "14",
            "15",
            "16",
            "17",
            "18",
            "19"
        ],
        "captureDescription": {
            "enabled": true,
            "encoding": "Avro",
            "destination": {
                "name": "EventHubArchive.AzureBlockBlob",
                "properties": {
                    "storageAccountResourceId": "/subscriptions/12345678/resourceGroups/RG_xxx/providers/Microsoft.Storage/storageAccounts/storagexxx",
                    "blobContainer": "evh-xxx",
                    "archiveNameFormat": "{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}"
                }
            },
            "intervalInSeconds": 300,
            "sizeLimitInBytes": 314572800
        }
    }
}
Azure API Management
Azure API Management
An Azure service that provides a hybrid, multi-cloud management platform for APIs.
1,913 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
591 questions
{count} votes