Is there a better way to stream data?

Nathan Sommer 40 Reputation points
2024-08-07T01:38:29.9633333+00:00

Hello Support,

We currently have a system where there are 10 devices per iot hub, and each of those devices are routed to their own event hub. From there they are ingested and streamed in ADX, each having there own table in the database. I tried using stream analytics to filter, but the watermark delay is too high and defeats the purpose of streaming. Is there a better way to do this? Maybe another way to filter by device id?

Thank you for your support,

Nathan

Azure IoT Hub
Azure IoT Hub
An Azure service that enables bidirectional communication between internet of things (IoT) devices and applications.
1,176 questions
Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
521 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
627 questions
0 comments
Accepted answer
  1. Sander van de Velde | MVP 32,011 Reputation points MVP
    2024-08-07T18:34:51.61+00:00

    Hello @Nathan Sommer ,

    welcome to this moderated Azure community forum.

    The obvious way would be enriching the messages in the IoT Hubs while being routed to the EventHub.

    You can add the device identifier as enrichment if you repeat it eg. in the tags of a device.

    When messages are enriched, the key/value pairs are added to the application settings, outside the body.

    In the past, the Azure Data Explorer data connection did not understand/support application settings. Azure Stream Analytics could fix this but that is out of scope.

    I noticed this line in the documentation regarding application properties:

    • In Event Hubs, when creating the event data object, embed the custom properties as part of the data in the body section of the object.

    I am not sure what this means... I think it references the EventData class and askes you to reprogram the logic to discard application properties...

    Another solution is ingesting the IoT Hub streaming events directly in Azure Data Explorer via the Database data connection for IoT Hub.

    Then you have access to the system properties where the 'iothub-connection-device-id' is available.

    These can be referenced in the mapping and added to the table.

    As an alternative, you could check the system properties of the Database data connection for EventHub. This one could supports the 'iothub-connection-device-id' to.

    Please let us know which solution works for you.

    If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Wilko van de Velde 2,226 Reputation points
    2024-08-07T13:31:26.8133333+00:00

    Hi Nathan,

    I think you have several possibilities, here are some that you can try:

    Kind Regards,

    Wilko

    Please do not forget to "Accept the answer” wherever the information provided helps you, this can be beneficial to other community members. If you have extra questions about this answer, please click "Comment".

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.