Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,028 questions
Why are so many Workflow Orchestration Manager configurations locked? Especially AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION is annoying.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 19%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
aeteq
0
Reputation points
What is the reason that so many configuration settings are locked, some to non-default values? For example, we would like to turn AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION to True but it is currently locked to False. This greatly hinders our workflow: We want to be able to publish DAGs independently from when they are actually started.
But since new DAGs now automatically start right away, we have to release them without a schedule, then turn them off in the UI, and the change their schedule to what it actually should be in a second release. This is really not ideal.
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT 28,766 Reputation points Microsoft Employee2024-07-03T17:42:24.2733333+00:00 Hello aeteq
The AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION setting is locked to False by default to ensure that new DAGs start running as soon as they are created. This is the default behavior of Airflow and is intended to ensure that DAGs are executed as soon as possible.
If need more control over this setting, Please submit a feature request in the ADF feedback channel:
All feedback shared in this forum is actively monitored and reviewed by the ADF engineering team. They prioritize requests based on the number of submissions.
Additionally, please share the link to your feedback once it is posted so that we can pass it along to the respective product owners for further review.
Hope this info helps.
-
aeteq 0 Reputation points
2024-07-05T11:29:50.7466667+00:00 This is the default behavior of Airflow and is intended to ensure that DAGs are executed as soon as possible.
That's not true, in Airflow this setting defaults to True, which means the DAGs are paused by default:
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#dags-are-paused-at-creationFor some reason, in WOM this option is locked to False, and you have not given me an explanation of why you cannot even turn this option on manually if you want to.
-
BhargavaGunnam-MSFT 28,766 Reputation points Microsoft Employee
2024-07-05T15:40:09.9466667+00:00 I mean, this is the default behavior of Airflow in Azure Data Factory (ADF).
You are correct; this option is locked to "False."
As I mentioned, please submit a feature request to change this behavior in the ADF feedback channel:
-
aeteq 0 Reputation points
2024-07-08T07:14:21.0566667+00:00 I've posted a request: https://feedback.azure.com/d365community/idea/2a79bf70-f93c-ef11-a296-6045bdbc5330
-
BhargavaGunnam-MSFT 28,766 Reputation points Microsoft Employee
2024-07-08T15:11:40.3033333+00:00 Hello aeteq,
Thank you for submitting the feedback item. I will pass this on to the respective product group. I will let you know if I hear anything from them.
Sign in to comment