Data factory storage event trigger support rename event

Joe Zeng 26 Reputation points
2024-10-16T05:23:00.34+00:00

Is it possible to add storage event trigger to support blob rename?

Currently it only support create and delete activity. But we have another platform called SAP DataSphere, it's using BlobRenamed event type and RenameFile api to push data into storage account.

So the storage event trigger dosen't work in Data Factory.

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,217 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,803 questions
Azure Event Grid
Azure Event Grid
An Azure event routing service designed for high availability, consistent performance, and dynamic scale.
390 questions
0 comments
Accepted answer
  1. Sina Salam 11,991 Reputation points
    2024-10-16T10:36:06.88+00:00

    Hello @Joe Zeng

    Welcome to the Microsoft Q&A and thank you for posting your questions here.

    In addition to solution above by @Pinaki Ghatak if you would like to have scalability and security solution, these are improvement solution and best practices:

    1. To maintain state and handle retries more robustly if there are failures when triggering the ADF pipeline and if the rename events are frequent, consider using Azure Durable Functions to handle the webhook logic - https://video2.skills-academy.com/azure/azure-functions/durable/durable-functions-overview
    2. To minimize the overhead of processing unrelated events, ensure that your Event Grid subscription is set up with appropriate filters to capture only the relevant events (BlobRenamed). - https://video2.skills-academy.com/azure/event-grid/event-schema-blob-storage
    3. Secure your webhook endpoint using authentication methods like Azure Active Directory (AAD) or API Management, ensuring that only authorized services can trigger the webhook - https://video2.skills-academy.com/azure/api-management/api-management-howto-protect-backend-with-aad
    4. When triggering the Data Factory pipeline from Azure Functions or Logic Apps, you can use Managed Identity to authenticate without storing credentials - https://video2.skills-academy.com/azure/data-factory/using-azure-managed-identities
    5. Implement logging within the webhook (Logic App or Function) for tracing failures or bottlenecks and leverage Azure Monitor and Application Insights to track events and performance - https://video2.skills-academy.com/azure/azure-monitor/overview

    I hope this is helpful! Do not hesitate to let me know if you have any other questions.

    Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

    0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Pinaki Ghatak 4,535 Reputation points Microsoft Employee
    2024-10-16T08:10:19.6966667+00:00

    Hello @Joe Zeng

    Azure Data Factory supports BlobCreated and BlobDeleted events for storage event triggers.

    However, it does not support BlobRenamed events. If you want to use BlobRenamed events to trigger your Data Factory pipelines, you can use Azure Event Grid to route BlobRenamed events to a webhook endpoint, which can then trigger your Data Factory pipeline.

    Here are steps of how you can use Azure Event Grid to route BlobRenamed events to a webhook endpoint:

    1. Create a webhook endpoint that triggers your Data Factory pipeline. You can use Azure Logic Apps or Azure Functions to create the webhook endpoint.
    2. Create an Event Grid subscription that filters for BlobRenamed events and routes them to your webhook endpoint. You can use the Azure portal or Azure CLI to create the subscription.
    3. Once you have set up the Event Grid subscription, any BlobRenamed events will be routed to your webhook endpoint, which can then trigger your Data Factory pipeline.

    I hope this helps

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.