adf data flow acitivity stuck in queued status when the pipelines run

I have the same issue with dataflow in ADF, all my pipelines get stuck in queue and in-progress status. When i run the diagnostics, this is what I see.

I already cancelled all the queue/in-progress pipelines and re-run only one pipeline but can not, it's still stuck with in-progress status.

Already tried with SSIS service, AutoResolveIntegrationRuntime (default IR) with simple dataflow, it's still the same.

Could you pls suggest the solution to solve it?

Thanks

Hi @Burcu

Thanks for the question and using MS Q&A platform.

When an Azure Data Factory (ADF) pipeline is stuck in the "Queued" status, it usually means that there are not enough resources available to run the pipeline. This can happen if there are too many pipelines running at the same time, or if the resources allocated to the ADF instance are insufficient.

Here are some steps you can take to resolve the issue:

1. Check the ADF instance resource utilization: Go to the ADF instance in the Azure portal and check the resource utilization metrics. If the resource utilization is high, you may need to increase the resources allocated to the ADF instance.
2. Check the pipeline concurrency settings: Go to the pipeline settings and check the concurrency settings. If the concurrency limit is set too low, it may be causing the pipeline to get stuck in the "Queued" status. Increase the concurrency limit if necessary.
3. Check the data flow settings: Go to the data flow settings and check the data flow concurrency settings. If the data flow concurrency limit is set too low, it may be causing the pipeline to get stuck in the "Queued" status. Increase the data flow concurrency limit if necessary.
4. Check the integration runtime settings: Go to the integration runtime settings and check the number of nodes allocated to the integration runtime. If the number of nodes is too low, it may be causing the pipeline to get stuck in the "Queued" status. Increase the number of nodes if necessary.
5. Check the pipeline dependencies: Go to the pipeline dependencies and check if there are any dependencies that are not met. If there are dependencies that are not met, it may be causing the pipeline to get stuck in the "Queued" status. Resolve the dependencies if necessary.

If none of these steps resolve the issue, you may need to contact Microsoft support for further assistance.

You can also refer to the Azure documentation on troubleshooting common issues in Azure Data Factory

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.