Azure SQL Database
An Azure relational database service.
5,347 questions
Azure Function concurrency issues
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 93%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Rafay Khan
20
Reputation points
I have multiple azure functions running at the same time and creating multiple instances of each.The parameters are also passed in parallel that means at one time all the azure function urls will be hit at once.Below is an example of how I am running
Input
https://function1..azurewebsites.net/api/datagenerator/{parameter]
https://function2..azurewebsites.net/api/datagenerator/{parameter]
https://function3..azurewebsites.net/api/datagenerator/{parameter]
Paramters passed
parameter = 1
parameter = 2
Output
https://function1..azurewebsites.net/api/datagenerator/{1]
https://function1..azurewebsites.net/api/datagenerator/{2]
https://function2..azurewebsites.net/api/datagenerator/{1]
https://function2..azurewebsites.net/api/datagenerator/{2]
https://function3..azurewebsites.net/api/datagenerator/{1]
https://function3..azurewebsites.net/api/datagenerator/{2]
The execution isnt sequential its parallel.My workflow is this
Azure Functions (all running at once on multiple parameter in parallel) -> Azure Stream Analytics -> Azure SQL DB
My Stream Analytics query is simple.I have tried parallelization and also increased the streaming units.I am getting 2 issues
Issue 1 : The problem I am getting is that if I run my functions for time longer than 10 seconds I dont get all the outputs I should be getting in the database.
Issue 2: The backlog input events when run for 10 seconds is still high.I have increased streaming units by which only The CPU utilzation,SU utilization and Watermark delay have been positively affected but no the backlog input events
{count} votes
-
MayankBargali-MSFT 69,946 Reputation points2023-12-27T02:48:55.1866667+00:00 @Rafay Khan Thanks for reaching out. When you mentioned "*my functions for time longer than 10 seconds I dont get all the outputs". *Did your function get executed successfully or was there any error/exception?
Your workflow Burst Load (using HTTP call) --> multiple azure function process --> sending the output to Azure stream Analytics --> Stream analytics saving the data to Azure SQL DB.
We need to first understand in which component/service you are facing the bottleneck/issue. I will first suggest you validate if there was any error/exception in any of the function app that couldn't send the request forward to stream analytics. Similarly, I will suggest you validate the same further for your other system/services so we can find the actual service that is causing the issue.
-
Rafay Khan 20 Reputation points
2023-12-27T11:49:11.84+00:00 Hello @MayankBargali-MSFT thankyou for replying.Yes my functions all got executed there was no error as such not in the Azure Function metrics and not even in the Function App Live metrics. The problem is at the multiple Http calls side I think. From what I assume there is a timeout and maybe after a certain time when the timeout expires the function is dropped.
The second issue is at the Stream Analytics side where irrespective of increasing the streaming units the backlog events are not being affected.
-
MayankBargali-MSFT 69,946 Reputation points
2023-12-27T12:01:35.45+00:00 @Rafay Khan Thanks for your response. If you don't see any error/exception at the function app end, then all the requests are processed successfully at the function app. I will suggest you to validate your HTTP request application whether the request was successfully sent or not. Did your application receiving timeout error/gateway error while making the call to your function app? Please validate if there is any port limit at your client where you are initiating the request. The function should be capable of handling a large number of requests.
Regarding your second issue I will wait for the stream analytics expert to comment.
-
Rafay Khan 20 Reputation points
2023-12-27T12:34:27.12+00:00 @MayankBargali-MSFT thankyou for replying. I have also checked the HTTP request application and its sending the requests successfully.
Did your application receiving timeout error/gateway error while making the call to your function app? No I did not receive any error
Please validate if there is any port limit at your client where you are initiating the request. The function should be capable of handling a large number of requests.
Can you please guide on how to validate a port limit?
-
MayankBargali-MSFT 69,946 Reputation points
2023-12-27T15:06:39.2466667+00:00 @Rafay Khan Your application would have thrown the error if there is any port exhaustion issue. Further if there is no error/exception either at the function app or your client application then I don't think there would be any issue with the function app. If you have logs in to your function app then you can manually validate the request logs to validate the same. Did you see any error/exception at the stream analytics end which would cause the behavior that you are seeing that the event not logged at the DB. Now the troubleshooting should be between your stream analytics and SQL db.
-
Rafay Khan 20 Reputation points
2023-12-29T11:25:35.64+00:00 Thankyou for your answer @MayankBargali-MSFT . No I dont find any error also I am confident that there is no problem between the Azure Stream Analytics and the SQL DB. When I trigger the functions I also open the IoT Hub metrics. So for instance if I run the functions for 10 seconds and expect 10 events I verify them in the metrics but as I increase it to 15 seconds or more and expect for example 15 events I receive less than 15 events.
I still think the error lies in Azure Function.
Case1:
I am running 2 azure functions in parallel without waiting for their response.The http requests are fired in parallel. If I run the below functions for 10 seconds with both functions per second
Time period = 10 seconds
Input
https://function1..azurewebsites.net/api/datagenerator/{parameter]
https://function2..azurewebsites.net/api/datagenerator/{parameter]
Paramters passed
parameter = 1
parameter = 2
Expected Output
https://function1..azurewebsites.net/api/datagenerator/{1] x 10
https://function2..azurewebsites.net/api/datagenerator/{1] x 10
https://function1..azurewebsites.net/api/datagenerator/{2] x 10
https://function2..azurewebsites.net/api/datagenerator/{2] x 10
(execution is parallel)
Recieved Output
https://function1..azurewebsites.net/api/datagenerator/{1] x 10
https://function2..azurewebsites.net/api/datagenerator/{1] x 10
https://function1..azurewebsites.net/api/datagenerator/{2] x 10
https://function2..azurewebsites.net/api/datagenerator/{2] x 10
(execution is parallel)
Case2:
I am running 2 azure functions in parallel without waiting for their response.The http requests are fired in parallel. If I run the below functions for 50 seconds with both functions per second
Time period = 50 seconds
Input
https://function1..azurewebsites.net/api/datagenerator/{parameter]
https://function2..azurewebsites.net/api/datagenerator/{parameter]
Paramters passed
parameter = 1
parameter = 2
Expected Output
https://function1..azurewebsites.net/api/datagenerator/{1] x 50
https://function2..azurewebsites.net/api/datagenerator/{1] x 50
https://function1..azurewebsites.net/api/datagenerator/{2] x 50
https://function2..azurewebsites.net/api/datagenerator/{2] x 50
(execution is parallel)
Recieved Output
https://function1..azurewebsites.net/api/datagenerator/{1] x 30
https://function2..azurewebsites.net/api/datagenerator/{1] x 50
https://function1..azurewebsites.net/api/datagenerator/{2] x 40
https://function2..azurewebsites.net/api/datagenerator/{2] x 20
(execution is parallel)
One thing to add is that I am using the consumption tier of the azure function can this be an error?
-
MayankBargali-MSFT 69,946 Reputation points
2023-12-29T11:37:37.4+00:00 @Rafay Khan I will suggest you engage us over one on one support to review it further as these needs live troubleshooting and looking into backend logs. In case you don't have the support plan then please let me know.
-
Rafay Khan 20 Reputation points
2024-01-02T11:37:44.65+00:00 Thankyou for replying @MayankBargali-MSFT .We are a direct CSP and have advanced support contract with Microsoft.
Sign in to comment