Pre-copy script in Copy Activity is not available in Azure Data Factory when Sink is ADX Azure Data Explorer

Anonymous
2023-11-16T11:14:39.32+00:00

Data is ingested using Copy Activity of Azure Data Factory into Azure Data Explorer table.

Need to clear/truncate the data from ADX table before load.

But Pre-copy script feature not available when ADX is used as sink in copy Activity.

Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
502 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,015 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. KranthiPakala-MSFT 46,437 Reputation points Microsoft Employee
    2023-11-17T00:44:04.2933333+00:00

    @Anonymous Thanks for using Microsoft Q&A forum and posting your query.

    Unlike other database connectors the Pre-copy script feature is not available when ADX is used as a sink in Copy Activity. Hence you will have to take an alternative approach to clear or truncate data from the ADX table before loading new data.

    Possible workaround: One possible approach is to use the lookup activity before your copy activity and execute a query that clears or truncates the table data (Note: Please ensure that your query returns a valid json response to your lookup activity).

    Here's an example query that clears a table: .clear table data command

    Please refer to this document where there is an sample on how to execute a Kusto query using lookup activity: Use Azure Data Factory command activity to run Azure Data Explorer management commands

    In case if you have any feedback for product team to implement pre & post script feature for ADX connector, please feel free to share your feedback in IDEAS forum: https://feedback.azure.com/d365community/forum/1219ec2d-6c26-ec11-b6e6-000d3a4f032c Please do share the feedback link once it is submitted so that other users with similar feedback can up-vote and comment on your feature request to help increase the priority of the request.

    I hope this info helps!

    Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

    0 comments