Bulk records to download from azure data explorer

Soumya Banerjee 126 Reputation points
2022-01-21T09:46:25.063+00:00

Hi,

I want to download/exporttoCSV logs/records from azure data explorer for past 3 months for one particular table for firewall.
The criteria is pretty wide because of which the number of records for just 1 day is around 1Million for the required criteria.
I need it for 3 months .

I have used "set truncationmaxsize" and "set truncationmaxrecords" also tried with set notruncation, increased connection time to 60 minutes, also kept time range for a maximum of 3 days at a time. Its completing run for 60 minutes and then failing .

Please let me know how can I download such bulk logs/record from adx.

Thanks in Advance.

Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
507 questions
0 comments
Accepted answer
  1. ShaikMaheer-MSFT 38,326 Reputation points Microsoft Employee
    2022-01-24T16:33:54.927+00:00

    Hi @Soumya Banerjee ,

    Thank you for posting query in Microsoft Q&A Platform.

    I will suggest to leverage Azure data factory for your requirement.

    Azure Data Factory is Azure's cloud ETL service for scale-out serverless data integration and data transformation. It offers a code-free UI for intuitive authoring and single-pane-of-glass monitoring and management.

    We can utilize Azure data explorer connector in ADF, to copy data to or from Azure Data Explorer. Click here to know more about same.

    Hope this will help. Please let us know if any further queries.

    --------------

    Please consider hitting Accept Answer button. Accepted answers helps community as well.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest