Clarification on how to calculate watermark for Azure Streaming Analytics

Russell Lindell 1 Reputation point
2020-07-02T14:47:38.423+00:00

I'm working on understanding watermarks in Azure Streaming Analytics. Per MS documentation, there are two ways that watermarks can be calculated.

https://video2.skills-academy.com/en-us/azure/stream-analytics/stream-analytics-time-handling#how-time-progresses-in-azure-stream-analytics

The second method states: "When there's no incoming event, the watermark is the current estimated arrival time minus the late arrival tolerance window. The estimated arrival time is the time that has elapsed from the last time an input event was seen plus that input event's arrival time."

Questions:

  • What is meant by "No incoming events"? Does this mean that the source (ex. Event Hub) is verified to be empty?
  • What is meant by "The last time an input event was seen."? Does this mean when it has exited the processing engine to the source?

Currently, this is how I interpret the calculation:
11262-01.png

(7) Watermark = (5) [Estimated Arrival Time] - (6) [Late Arrival Tolerance Window]

----------

(5) Estimated Arrival Time =
(1) Elapsed Time = Time that elapsed from the last time an input event was last seen (2) and the current time (3) 
Plus
(4) Last Arrival Time

Azure Stream Analytics
Azure Stream Analytics
An Azure real-time analytics service designed for mission-critical workloads.
354 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. HimanshuSinha-msft 19,476 Reputation points Microsoft Employee
    2020-07-06T22:38:51+00:00

    Hello @rwlindell,

    Thanks for the question .

    Yes "No incoming events" means that ideally there is nothing to process on the Eventhub .

    For the second part , i think you have already gone through the doc which states

    "When there's no incoming event, the watermark is the current estimated arrival time minus the late arrival tolerance window. The estimated arrival time is the time that has elapsed from the last time an input event was seen plus that input event's arrival time." .

    If we are in the 0:45 mark and if you see that the expected events was at :40 unit ( lets assume that the
    event is coming in every 5 unit of time ) . so the watermark will be (45-15 (6) in your example)

    Thanks Himanshu

    Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.