I have an Azure Function written in Python that's meant to process messages from an Azure Storage Queue. In my host.json
file, I have set "batchSize": 1
and "newBatchThreshold": 0
under the queues settings. This should mean that the function dequeues and processes one message at a time. The function execution time will take minimum 1 minuets as per the business logic. This arrangement works just fine locally.
The function code
import azure.functions as func
import logging
import json
import os
import time
app = func.FunctionApp()
@app.queue_trigger(arg_name="msg", queue_name=os.getenv("REQUEST_QUEUE"), connection="AzureWebJobsStorage")
def ustorescraper(msg: func.QueueMessage):
received_message = msg.get_body().decode('utf-8')
json_data = json.loads(received_message)
request_type_str = json_data["requestType"]
logging.info('Process started : %s', request_type_str)
time.sleep(65)
logging.info('Processing end : %s', request_type_str)
host.json
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
},
"logLevel": {
"default": "Information"
}
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[4.*, 5.0.0)"
},
"extensions": {
"queues": {
"batchSize": 1,
"newBatchThreshold": 0,
"maxPollingInterval": "00:05:00",
"visibilityTimeout": "00:00:30",
"maxDequeueCount": 3,
"messageEncoding": "base64"
}
}
}
However, when I deploy to Azure, the function seems to be processing multiple messages at a time, which is not what I want.
In an attempt to override the settings on Azure, I added "AzureFunctionsJobHost__extensions__queues__batchSize": "1"
and "AzureFunctionsJobHost__extensions__queues__newBatchThreshold": "1"
as environment variables to the Function App settings in the Azure portal, as per Microsoft's documentation.
Some of Environment variables opened as Advanced Edit
{
"name": "AzureFunctionsJobHost__extensions__queues__batchSize",
"value": "1",
"slotSetting": true
},
{
"name": "AzureFunctionsJobHost__extensions__queues__newBatchThreshold",
"value": "0",
"slotSetting": true
},
{
"name": "REQUEST_QUEUE",
"value": "request-queue",
"slotSetting": false
},
{
"name": "AzureWebJobsStorage",
"value": "xxx",
"slotSetting": false
}
Additionally, I've also disabled the scale count of my function app, to prevent Azure from scaling out instances.
Despite all these configurations, my function app continues to dequeue and process multiple messages concurrently.
Looks like the application not reading from the host.json file on azure environment.
Has anyone encountered this ?