Working with Airflow, one often requires access to the Airflow CLI/API or even access to the underlying meta database. For example: to import or export connections start backfills manage Variables (using scripts) manage DB items that are not accessible from the UI How would you go about doing that in WOM?

Hi @aeteq Thank you for providing more context on what you're trying to achieve with the Airflow CLI, API, or database access in WOM. I understand that you need to perform various operations such as importing/exporting connections, starting backfills, managing variables, and accessing the database. Unfortunately, WOM doesn't provide direct shell access to the Airflow instance, and the Airflow CLI is not directly accessible. Additionally, the Airflow API is not openly available for authentication. However, I'd like to explore alternative solutions with you. Since WOM is a managed service, we can look into using WOM's built-in features or APIs to achieve your goals. Regarding database access, WOM provides a managed database service, and direct database access is not possible. However, you can use WOM's built-in features to export and import database data. If there are specific use cases that are not covered by WOM's built-in features, you can also share the feedback on our feedback channel Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature. Hope this helps. Do let us know if you any further queries. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

I found a workaround for the CLI part: You can use a specialized DAG with a BashOperator to execute airflow CLI commands. You can find an example backfill DAG here: https://github.com/mozilla/telemetry-airflow/blob/main/dags/backfill.py

Hi @aeteq I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that " The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer . Ask: Working with Airflow, one often requires access to the Airflow CLI/API or even access to the underlying meta database. For example: to import or export connections start backfills manage Variables (using scripts) manage DB items that are not accessible from the UI How would you go about doing that in WOM? Solution: I found a workaround for the CLI part: You can use a specialized DAG with a BashOperator to execute airflow CLI commands. You can find an example backfill DAG here: https://github.com/mozilla/telemetry-airflow/blob/main/dags/backfill.py If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information. I hope this helps! If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue. Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

How do you access the Airflow CLI/API/DB in Workflow Orchestration Manager?

aeteq 0

Working with Airflow, one often requires access to the Airflow CLI/API or even access to the underlying meta database. For example:

to import or export connections
start backfills
manage Variables (using scripts)
manage DB items that are not accessible from the UI

How would you go about doing that in WOM?

Smaran Thoomu 15,765 Reputation points Microsoft Vendor

2024-07-03T12:18:09.3433333+00:00
Hi @aeteq

Thanks for the question and using MS Q&A platform.

To access the Airflow CLI/API or the underlying metadata database in Workflow Orchestration Manager (WOM), you can use the following steps:

Once you are logged in, navigate to the Airflow installation directory. This is typically located at /opt/workflow-orchestration-manager/airflow.

To access the Airflow CLI, you can run the airflow command followed by the desired command and options. For example, to list all available DAGs, you can run the following command:
airflow list_dags

To access the Airflow API, you can use the API endpoint provided by WOM. The endpoint is typically located at https://<wom-instance-name>/airflow/api/v1/. You can use this endpoint to access the Airflow REST API and perform various operations such as starting backfills, managing variables, etc.

To access the Airflow API, you can use the API endpoint provided by WOM. The endpoint is typically located at https://<wom-instance-name>/airflow/api/v1/. You can use this endpoint to access the Airflow REST API and perform various operations such as starting backfills, managing variables, etc.
psql -h <database-host> -p <database-port> -U <database-username> <database-name>
Replace <database-host>, <database-port>, <database-username>, and <database-name> with the appropriate values from the Airflow configuration file. Once you are connected to the database, you can use SQL commands to manage DB items that are not accessible from the UI.

For more details refer the below docs:

REST APIs for the Workflow Orchestration Manager

How does Azure Workflow Orchestration Manager work

I hope this helps! Let me know if you have any further questions.
Smaran Thoomu 15,765 Reputation points Microsoft Vendor

2024-07-04T09:49:32.4233333+00:00

@aeteq We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
aeteq 0 Reputation points

2024-07-05T07:15:14.97+00:00

Hello @Smaran Thoomu

Once you are logged in, navigate to the Airflow installation directory.

Logged in where? What do you mean by navigate? How would I even access a shell on the airflow instance?

The docs you linked are not related to your answer.

Regarding the API, how would even authenticate with that? Are you sure the API is just openly available?
Smaran Thoomu 15,765 Reputation points Microsoft Vendor

2024-07-08T12:41:15.45+00:00

Hi @aeteq

Thank you for your feedback. I apologize if my previous response was not clear.

Regarding navigating to the Airflow installation directory, I understand that it might not be straightforward in a Workflow Orchestration Manager (WOM) setup. Since WOM is a managed service, you don't have direct access to the underlying Airflow instance. My previous instructions were incorrect, and I appreciate you bringing this to my attention.

To access the Airflow CLI, API, or database in WOM, you'll need to use alternative methods. Unfortunately, WOM doesn't provide direct shell access to the Airflow instance.

To better assist you, could you please provide more context on what you're trying to achieve with the Airflow CLI, API, or database access in WOM? This will help me provide more tailored guidance or alternative solutions.

Again, I apologize for any inconvenience caused, and I'm here to help.
aeteq 0 Reputation points

2024-07-10T14:53:29.72+00:00
I mentioned it in my original question, but here it is again along with some additional use cases

to import or export connections: airflow connections

start backfills: airflow dags backfill

start test DAG-runs (maybe even from CI/CD) using airflow dags test

manage Variables (using scripts): airflow variables

manage DB items that are not accessible from the UI: airflow db shell

backup the Airflow-DB: airflow db export-archived

import the DB (when migrating an on-premise airflow instance into WOM): would probably require direct DB access or custom import tool

4 answers

Deleted

This answer has been deleted due to a violation of our Code of Conduct. The answer was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

Comments have been turned off. Learn more
Smaran Thoomu 15,765 Reputation points Microsoft Vendor

2024-07-11T07:46:02.18+00:00

Hi @aeteq

Thank you for providing more context on what you're trying to achieve with the Airflow CLI, API, or database access in WOM. I understand that you need to perform various operations such as importing/exporting connections, starting backfills, managing variables, and accessing the database.

Unfortunately, WOM doesn't provide direct shell access to the Airflow instance, and the Airflow CLI is not directly accessible. Additionally, the Airflow API is not openly available for authentication.

However, I'd like to explore alternative solutions with you. Since WOM is a managed service, we can look into using WOM's built-in features or APIs to achieve your goals. Regarding database access, WOM provides a managed database service, and direct database access is not possible. However, you can use WOM's built-in features to export and import database data.

If there are specific use cases that are not covered by WOM's built-in features, you can also share the feedback on our feedback channelWhich would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.
Smaran Thoomu 15,765 Reputation points Microsoft Vendor

2024-07-17T12:22:26.2633333+00:00

@aeteq Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

aeteq 0 Reputation points

2024-07-17T14:08:47.17+00:00

However, you can use WOM's built-in features to export and import database data.

How? What data exactly?
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
aeteq 0 Reputation points

2024-07-17T14:05:59.9+00:00

I found a workaround for the CLI part: You can use a specialized DAG with a BashOperator to execute airflow CLI commands. You can find an example backfill DAG here: https://github.com/mozilla/telemetry-airflow/blob/main/dags/backfill.py
Please sign in to rate this answer.
Smaran Thoomu 15,765 Reputation points Microsoft Vendor

2024-07-17T14:34:43.21+00:00

@aeteq Glad to know your issue has been resolved. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "I'll repost your solution in case you'd like to accept the answer.

Smaran Thoomu 15,765 Reputation points Microsoft Vendor

2024-07-18T09:38:35.94+00:00

@aeteq Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Smaran Thoomu 15,765 Reputation points Microsoft Vendor

2024-07-17T14:34:16.23+00:00
Hi @aeteq

I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Ask: Working with Airflow, one often requires access to the Airflow CLI/API or even access to the underlying meta database. For example:

to import or export connections

start backfills

manage Variables (using scripts)

manage DB items that are not accessible from the UI

How would you go about doing that in WOM?

Solution: I found a workaround for the CLI part: You can use a specialized DAG with a BashOperator to execute airflow CLI commands. You can find an example backfill DAG here: https://github.com/mozilla/telemetry-airflow/blob/main/dags/backfill.py

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

I hope this helps!

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How do you access the Airflow CLI/API/DB in Workflow Orchestration Manager?

4 answers

Your answer