Incoming and Outgoing messages mismatch in Event Hubs

dotnet_guy 10

Hello,

We have an Azure function triggered by Event Hubs. There is over 7-15min varying gap between each function execution(s). We see about 1k messages coming in constantly but the outgoing spikes every 7-15min are over 8k messages and sometimes around 10k as well. The overall incoming and outgoing messages nearly match as soon as soon as the spike is done. But there are some missing, will tackle that later. There are no errors in App Insights and Log Analytics.

Now I am wondering why the spikes and not have a uniform incoming and outgoing as soon as the messages come in, the memory for the spike is high (~350MB). The function is always on, so no cold start. I did not any additional config, for example to fire the function after a certain time or number of events. We are using 32 partitions and have standard, not premium tier.

Please let me know if I am missing something and how the spikes can be evened out.

Thanks.

Sander van de Velde | MVP 32,011 Reputation points MVP

2024-07-30T19:28:40.39+00:00

Hello @dotnet_guy ,

32 partitions feels like a lot. why did you chose 32 partitions?

How many streaming units have you set on the EventHub?

Does the Azure Function scale up automatically?
dotnet_guy 10 Reputation points

2024-07-30T19:33:42.2733333+00:00

By streaming units, I think you mean Throughput units, which are 5 and auto-inflates to 10. We have a lot more logs coming in later after this proof of concept, so use 32. We will be fine tuning later.

Scaling up should not be the issue here, as in between the spikes, the function memory and executions are zero and no logging/metrics as well.

Share via

Incoming and Outgoing messages mismatch in Event Hubs

Your answer