Using Event trigger Azure function, how to convert csv file to parquet file

Amey Pimpley 1 Reputation point
2020-12-11T03:26:04.387+00:00

I want to get a file in azure blob storage and using event trigger function, I want to convert this csv file into parquet format and upload it to another blob storage. I tried to use choparquet reader, but it takes a lot of time to convert the file to parquet format and when a large files comes, few events are missed from the queue and not all files are this converted in to parquet format. I am using c#

Can anyone help in simplified approach to this problem?

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,919 questions
Azure Event Grid
Azure Event Grid
An Azure event routing service designed for high availability, consistent performance, and dynamic scale.
390 questions
C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
10,996 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. deherman-MSFT 37,411 Reputation points Microsoft Employee
    2020-12-11T17:43:39.143+00:00

    @Amey Pimpley
    Unfortunately I do not have experience working with parquet. Since you are working with c# your options are somewhat limited. I was able to find the Parquet.Net package which should be able to accomplish this. I found a blog post which shows how you can use the package to convert data. Please review this and see if it works for you.

    Hope this helps! Let us know if you have further questions or issues.

    -------------------------------

    Please don’t forget to "Accept the answer" and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.