How can I store data from IoT Hub/Function App/Azure Digital Twins into a storage account in CSV format?

Jayashri Goddanti 45 Reputation points
2024-08-20T10:58:53.5933333+00:00

I’m currently receiving live data from equipment and have set up IoT Hub, Function App, and Azure Digital Twins, successfully viewing the data across these layers. Now, I want to store this data in Azure Blob Storage or another storage solution in CSV or text file format, rather than in JSON, as the data from IoT Hub is in JSON format. Could you suggest the most effective method for achieving this? I previously tried using Stream Analytics but found it to be quite expensive. Any advice would be greatly appreciated. Thanks for your help!

Azure Digital Twins
Azure Digital Twins
An Azure platform that is used to create digital representations of real-world things, places, business processes, and people.
231 questions
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
5,081 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,215 questions
0 comments
Accepted answer
  1. Sander van de Velde | MVP 32,811 Reputation points MVP
    2024-08-20T15:45:26.1866667+00:00

    Hello @Jayashri Goddanti ,

    welcome to this moderated Azure community forum.

    There are a few options to consider.

    If you only want to forward raw IoT Hub messages to a storage account, you normally use a IoT Hub route towards an custom blob storage endpoint. Unfortunately, you want to use the CSV format, this is not supported by the IoT Hub routing endpoint (only AVRO and JSON).

    If you still want CSV, you could create an Azure Function triggered when a file is created on the storage account, converting the file to CSV and deleting the original file.

    If you are interested in events coming from the Azure Digital Twins Explorer, these are normally sent one-by-one to an Eventhub. So, batching them into separate CSV file will be a complex procedure.

    A possible solution would be forwarding ADT message to an Azure Function, triggered by the EventHub. The function puts each new message in a storage queue. If there are enough messages collected, the queue is emptied andthe messages are saved in a CSV file.

    An Alternative would be using Azure Data Factory, able to transform files too.

    As you can see, your wish to use CSV makes it all a bit complex.

    Why do you want to have it in CSV/text format?

    Do you know JSON is also just another text format?

    I want to propose a different approach.

    You can forward all ADT twin changes and/or telemetry, and/or IoT Hub raw messages to EventHubs.

    Have you considered the free Azure Data Explorer already?

    It feel you want to do some data engineering on the incoming data by storing it for a longer period.

    Normally Azure Data Explorer would be the perfect tool for this but an ADX cluster is not cheap if you have a simple project.

    Microsoft provides a free ADX cluster, capable of ingesting streaming data coming from an Event Hub so you could try this out too.

    If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.