How to do file deletion tracking for Azure AI Search and Azure File Storage

Support Boris Shabash 0 Reputation points
2024-06-27T19:24:33.8366667+00:00

I have an Azure AI Search resource as well as an Azure Storage resource that has files uploaded to a File Share

I've managed to connect an Index to the File Share, but deletion tracking seems to be a major pain point. I have to use the SoftDeleteColumnDeletionDetectionPolicy as a deletion policy, but there I do not know which column or metadata aspect to track.
I saw the documentation here https://video2.skills-academy.com/en-us/azure/search/search-howto-index-changed-deleted-blobs?tabs=portal#soft-delete-strategy-using-custom-metadata

But it seems to suggest I add a new metadata field to every file I have (~1,800 files) and then when I want to delete a file or files, I must first edit that metadata field, which doesn't sound like a scalable or serious method for a deletion of 10-100 files at a time

Is there documentation for which property of a file can change upon deletion? or some other way of achieving deletion tracking for a File Share

Azure Files
Azure Files
An Azure service that offers file shares in the cloud.
1,207 questions
Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
823 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sina Salam 6,496 Reputation points
    2024-06-29T00:50:33.17+00:00

    Hello Support Boris Shabash,

    Welcome to the Microsoft Q&A and thank you for posting your questions here.

    Problem

    I understand that you are having issues with file deletion tracking for Azure AI Search and Azure File Storage.

    Solution

    To answer your questions.

    Is there documentation for which property of a file can change upon deletion

    When it comes to tracking file deletions in Azure File Shares, there is not specific documentation on which property of a file changes upon deletion. But you can infer deletions through other properties and mechanisms such as:

    • Last Modified Timestamp.
    • File Metadata.
    • Activity Log and Azure Monitor.
    • Using soft delete for Azure AI Search
    • In Azure Storage, add a custom metadata key-value pair to the file to indicate it’s flagged for deletion. You can create a property like "IsDeleted" and set it to true when you want to delete the file.

    https://stackoverflow.com/questions/57500954/automatically-delete-files-in-storage.

    https://techcommunity.microsoft.com/t5/itops-talk-blog/azure-monitor-detecting-and-alerting-on-file-changes/ba-p/286327.

    https://video2.skills-academy.com/en-us/azure/architecture/ai-ml/architecture/search-blob-metadata.

    some other way of achieving deletion tracking for a File Share

    Managing deletion tracking for Azure File Share is essential, especially when dealing with a large number of files.

    • You can find out who deleted a file share from your Azure storage account by checking the Activity log in the Azure portal
    • Azure Files indexer (preview) in Azure AI Search is one of the best.
    • To set up alerts specifically for file deletions, you can create an alert rule using Azure Monitor.
    • Create a time-triggered Azure Function app and use an Azure CLI script to periodically delete files that haven't been modified within a specified time frame using Schedule Function App.

    https://stackoverflow.com/questions/72710134/how-to-set-alert-on-edit-delete-of-a-file-or-folder-in-azure-portal.

    https://stackoverflow.com/questions/68500637/how-to-delete-from-files-from-azure-file-share-periodically-after-say-x-days.

    https://video2.skills-academy.com/en-us/azure/storage/files/storage-files-monitoring.

    References

    Use above links as the source to read more and for detail steps.

    Accept Answer

    I hope this is helpful! Do not hesitate to let me know if you have any other questions.

    ** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

    Best Regards,

    Sina Salam

    0 comments