Watermark Delay Issues + Streaming Ingestion

Nathan Sommer 0 Reputation points
2024-06-26T21:19:25.6033333+00:00

Hello,

I'm using the azure stream analytics to filter the devices in

the event hub and output them in different tables. The issue I'm having is that it's not streaming the data on dashboards that it's getting. For example, it should show (stream) new data every 10 seconds, but it's showing the new data every 5-10 minutes. I'm not sure what's causing this - I did change the policy to streamingingestion and turned off batching policy. Any help would be greatly appreciated. User's image

Azure Stream Analytics
Azure Stream Analytics
An Azure real-time analytics service designed for mission-critical workloads.
342 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sander van de Velde | MVP 30,711 Reputation points MVP
    2024-07-01T19:45:24.38+00:00

    Hello @Nathan Sommer,

    welcome to this moderated Azure community forum.

    You experience large delays when streaming data from Azure Stream Analytics to Azure Data Explorer.

    Using the documentation you provide, it seems the delay is coming from Azure Data Explorer because the ASA watermark delay is much less.

    Azure Data Explorer uses queued ingest by default: 5 minutes, 1000 items, or a total size of 1 GB.

    You say to enabled streaming ingest.

    You need to do this both on cluster level and per database/table:

    .alter table TestTable policy streamingingestion enable

    Perhaps there could be a delay before this is actually working so stay tuned.

    Personally, I use an EventHub between ASA and ADX so I'm fully in control of the data being exchanged. Think about having a preview of the data inside the eventhub, being able to pause eventhub egress and the eventhub data connection settings.

    If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

    0 comments