Azure function event hub trigger executes after delay of about 5-10 seconds

Question

I have an Azure Function v4 .NET 6 isolated app with an HTTP trigger function that writes events to an event hub. Another event hub triggered function in the same app listens to those events, performs some manipulation, and publishes them again to another hub, which is listened to by a different function app.

I've noticed through Application Insights logs and some custom audit logs I have written that there's a delay of approximately 5-10 seconds between when an event is published and when it's received by the event hub trigger function. Since there are two event hub listeners involved, the total delay is 10-20 seconds, which is unacceptable for my use case.

Package references:

Event hub trigger:

[Function(_functionName)]
public async Task RunAsync([EventHubTrigger(eventHubName: $"% 
{nameof(EnvironmentVariables.HubName)}%",
    Connection = $"{nameof(EnvironmentVariables.HubNamespaceReadWriteConnection)}",
    ConsumerGroup = "$Default")] string[] messages)
{
    //Biz logic
}

Configuration in host.json for the event hub is:

"eventHubs": {
  "batchCheckpointFrequency": 1,
  "maxEventBatchSize": 200,
  "maxWaitTime": "00:00:01",
  "minEventBatchSize": 2,
  "clientRetryOptions": {
    "mode": "exponential",
    "tryTimeout": "00:05:00"
  }
}

As you can see maxWaitTime is set to 1 second, so it should wait for a maximum of 1 second to create a batch. However, it's not behaving according to this configuration.

I've also tried removing batching from the trigger altogether and listening to single events only, but the delay remains the same. Here's an example of the code for that:

[Function(_functionName)]
public async Task RunAsync([EventHubTrigger(eventHubName: $"% 
{nameof(EnvironmentVariables.HubName)}%",
    Connection = $"{nameof(EnvironmentVariables.HubNamespaceReadWriteConnection)}",
    ConsumerGroup = "$Default", IsBatched = false)] string message)
{
    //Biz logic
}

My app is hosted in the consumption tier. I've also tried the premium tier, but the behavior remains the same. Additionally, this should not be a cold start issue because this is happening for several events sent at intervals of 1 minute.

I suspected that delay was originating from within the function app only and not from event hub, and to validate this I have tested this with normal .NET API hosted in Azure app service. There is no such delay, event is getting received with latency for few milliseconds.

In the meantime, are there any further changes I can implement to potentially reduce this delay?

[Edit] 05-06-24

I don't know how, but most of the detail from my question got removed. I have added it again now.

Answer

Hello Akshay Chauhan,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that your Azure function event hub triggers after delay of about 5-10 seconds and you are seeking what to changes or implement to potentially reduce this delay.

Best practices solution

Based on your explanations, the code and the work you posted. I must commend you for a good work, you have already implemented several optimizations and configurations to reduce the latency in your Azure Functions. However, there are a few additional aspects to consider that might help you further diagnose and potentially reduce the delay.

Validate the delay source with a simple .NET listener application.
Fine-tune the host.json configuration further.
Ensure all resources are in the same region to reduce network latency.
Log function execution times to identify internal delays.
Monitor and possibly adjust scaling settings.
Use Application Insights for detailed diagnostics.
Ensure sufficient throughput units are allocated for Event Hubs.

By addressing each of these areas, you should be able to diagnose and reduce the latency in your Azure Functions.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions or about to do any of the above.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam

Answer

This has been resolved. Actual delay was in the event hub publisher. I was using buffered event hub publisher with wait time of 5 seconds. Reducing the time solved the problem.

Share via

Azure function event hub trigger executes after delay of about 5-10 seconds

2 answers

Best practices solution

Accept Answer