Push Large Quantity of events to Event Hub

Chandan B 1 Reputation point
2020-11-20T08:27:59.757+00:00

A service which is generating 7.2 B events in 30 mins Duration. Need to store these events
in Azure Data Explorer. To send the data we are using EventHub since we can stream the data in
realtime using the EventHubs. We are not able to send these many records to the event hub through
AMQP and event hub client SDK. Is there any way to push huge records in nominal
Duration(~30mins).
We are able to push 2.5 Million Events to HUB in batches(Each bach contains 400 Events)
with the following code:

public static async Task EventHub(string processors
 {     
     var producerOptions = new EventHubProducerClientOptions();
     producerOptions.ConnectionOptions.TransportType = EventHubsTransportType.AmqpWebSockets;
   var producer = new EventHubProducerClient(connectionstring, hubName,    producerOptions);
         using EventDataBatch eventBatch = await producer.CreateBatchAsync();
            _ = eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes(processLogs)));
      await producer.SendAsync(eventBatch);
}

and the event data follows below:
EVENT Data:
{
"Event_End_Timestamp": "2020-11-06T06:00:01.8700000Z",
"Event_Duration": 0.16361287236213684,
"Scrap": 5.0,
"OperationCycle": 0.19,
"WorkOrder_ID": "123450",
"Sku": "98764",
"Sequence": "10",
"Process_ID": "H200",
"Machine_ID": "M1503",
"Parent_Event_ID": 10000002,
"Event_ID": 10000000,
"Event_Type_Counter": 10,
"Event_Type": "OpercationCycle",
"Event_Start_Timestamp": "2020-11-06T06:00:01.7060000Z"
}

Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
502 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
591 questions
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,051 Reputation points
    2020-11-20T22:53:41.037+00:00

    Hello @Chandan B and @Ankarao . Welcome to Microsoft Q&A.

    You are describing a difficulty in throughput. To better serve you, can you describe whether the bottleneck is in:

    1. The throughput of your publisher to send events
    2. The throughput of Event Hubs to accept the events send by your publisher. (the publisher will see a server busy exception)
    3. Getting the data from Event Hubs to Azure Data Explorer

    In the case 2. one solution is to increase the Throughput Units. You can try the Auto-Inflate feature to increase based on demand. To grow beyond the maximum Throughput Units, you can raise a support request to increase the maximum.

    Another possibility in case 2. is to switch from basic or standard tier of Event Hub to Dedicated tier of Event Hub.

    1 person found this answer helpful.
    0 comments No comments