Intermittent Trigger Issues with Event Grid for Blob Upload and Deletion in Azure Function

SuMyat Hlaing 0 Reputation points
2024-06-12T01:03:14.7566667+00:00

I am using an Event Grid subscription to trigger an Azure Function for handling blob uploads and deletions in Azure Blob Storage. However, I'm encountering intermittent issues with the triggers not firing consistently.

Setup:

  1. Storage Account (Event Grid)
  2. Event Grid Subscription subscribed to Microsoft.Storage.BlobCreated and Microsoft.Storage.BlobDeleted events.
  3. Azure Function Event Grid Trigger

Issue: The problem is that the Event Grid trigger is not firing every time a blob is uploaded or deleted. When the trigger does fire, both the upload and delete actions work correctly. However, the trigger itself is inconsistent.

Steps Taken:

  • Subscribed to both Microsoft.Storage.BlobCreated and Microsoft.Storage.BlobDeleted events.
  • Implemented function code to handle different event types and execute the necessary business logic.

Despite this setup, the triggers do not consistently fire for every blob upload or deletion event.

Request for Help: Could someone help me troubleshoot and resolve these intermittent issues with the Event Grid triggers for blob uploads and deletions? Any insights or guidance would be greatly appreciated.

Thank you!
User's image

User's image

const { app } = require('@azure/functions');
const { BlobServiceClient, StorageSharedKeyCredential } = require('@azure/storage-blob');
const { SearchClient, AzureKeyCredential } = require('@azure/search-documents');
const pdf = require('pdf-parse');
const path = require('path');
// Azure Blob Storage configuration
const accountName = "";
const accountKey = "";
const containerName = 'content';
// Azure Cognitive Search configuration
const searchServiceName = "";
const indexName = "demoindex2";
const apiKey = "";
const blobServiceClient = new BlobServiceClient(
    `https://${accountName}.blob.core.windows.net`,
    new StorageSharedKeyCredential(accountName, accountKey)
);
const containerClient = blobServiceClient.getContainerClient(containerName);
const searchClient = new SearchClient(
    `https://${searchServiceName}.search.windows.net/`,
    indexName,
    new AzureKeyCredential(apiKey)
);
async function indexBlob(blobName) {
    try {
        const blobClient = containerClient.getBlobClient(blobName);
        const downloadResponse = await blobClient.download();
        const encodedName = Buffer.from(blobName).toString('base64');
        const properties = await blobClient.getProperties();
        const pdfBuffer = await streamToBuffer(downloadResponse.readableStreamBody);
        const pdfText = await pdf(pdfBuffer);
        const blobContent = pdfText.text;
        const document = {
            id: encodedName,
            content: blobContent,
            metadata_storage_content_type: properties.contentType || null,
            metadata_storage_size: properties.contentLength || null,
            metadata_storage_last_modified: properties.lastModified ? new Date(properties.lastModified).toISOString() : null,
            metadata_storage_content_md5: properties.contentMD5 ? Buffer.from(properties.contentMD5).toString('base64') : null,
            metadata_storage_name: blobName,
            metadata_storage_path: blobClient.url,
            metadata_storage_file_extension: path.extname(blobName),
            metadata_content_type: properties.contentType || null,
            metadata_language: null,
            metadata_author: null,
            metadata_creation_date: properties.creationTime ? new Date(properties.creationTime).toISOString() : null,
        };
        await searchClient.uploadDocuments([document]);
        console.log(`Document "${document.id}" has been indexed`);
    } catch (error) {
        console.error(`Error indexing document: ${error}`);
    }
}
async function deleteDocument(blobName) {
    try {
        const encodedName = Buffer.from(blobName).toString('base64');
        await searchClient.deleteDocuments([{ id: encodedName }]);
        console.log(`Document "${encodedName}" has been deleted from the index`);
    } catch (error) {
        console.error(`Error deleting document from the index: ${error}`);
    }
}
app.eventGrid('process-event-grid', {
    handler: async (context, eventGridEvent) => {
        try {
            const event = eventGridEvent.eventType;
            const blobUrl = eventGridEvent.data?.url;
            
            if (!blobUrl) {
                console.error("Event data does not contain 'url':", eventGridEvent);
                return;
            }
            
            const blobName = blobUrl.substring(blobUrl.lastIndexOf('/') + 1);
            if (event === 'Microsoft.Storage.BlobCreated') {
                console.log(`Blob created: ${blobName}`);
                await indexBlob(blobName);
            } else if (event === 'Microsoft.Storage.BlobDeleted') {
                console.log(`Blob deleted: ${blobName}`);
                await deleteDocument(blobName);
            } else {
                console.log(`Unhandled event type: ${event}`);
            }
        } catch (error) {
            console.error(`Error processing event: ${error}`, eventGridEvent);
        }
    }
});
async function streamToBuffer(readableStream) {
    return new Promise((resolve, reject) => {
        const chunks = [];
        readableStream.on("data", (data) => {
            chunks.push(data instanceof Buffer ? data : Buffer.from(data));
        });
        readableStream.on("end", () => {
            resolve(Buffer.concat(chunks));
        });
        readableStream.on("error", reject);
    });
}
Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,567 questions
Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
831 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,576 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Silvia Wibowo 3,411 Reputation points Microsoft Employee
    2024-06-12T03:07:51.6333333+00:00

    Hi @SuMyat Hlaing , I understand that you are having issues with Storage Blob event trigger for Azure Functions.

    Please review the Best practices for consuming events, especially these points:

    • Check the topic of the message to ensure that it comes from the storage account you are expecting.
    • Similarly, check that the eventType is one you are prepared to process, and do not assume that all events you receive will be the types you expect.
    • Storage events guarantees at-least-once delivery to subscribers, which ensures that all messages are outputted. However due to retries between backend nodes and services or availability of subscriptions, duplicate messages may occur. To learn more about message delivery and retry, see Event Grid message delivery and retry.
    • Use the blobType field to understand what type of operations are allowed on the blob, and which client library types you should use to access the blob. Valid values are either BlockBlob or PageBlob.
    • If you want to ensure that the Microsoft.Storage.BlobCreated event is triggered only when a Block Blob is completely committed, filter the event for the CopyBlob, PutBlob, PutBlockList or FlushWithClose REST API calls. These API calls trigger the Microsoft.Storage.BlobCreated event only after data is fully committed to a Block Blob. To learn how to create a filter, see Filter events for Event Grid.

    When you say "intermittent" - do you mean it's not triggering when it should, or triggering multiple times for one event?