What Azure service should I use to deploy a complex Python program on the cloud?

kman-1604 0 Reputation points
2024-06-05T13:06:56.45+00:00

Background<br>

I have developed a Python program that fetches data from three different REST APIs, processes it, and inserts it into a database. The program also queries the database to identify which values to fetch from the APIs, so there is constant communication beteen the database and the APIs.

The data processing involves significant preprocessing, cleaning, and normalisation. My codebase is quite complex, utilising object-oriented programming with multiple modules, including database management classes, data preprocessing classes, OCR for reading PDFs, XML parsers, etc. I also use Poetry for dependency management.

Problem<br>

I need to migrate my program to the cloud. My organisation uses Azure, and I am new to cloud computing. I have been advised to use Azure Databricks, but it seems more like a Jupyter Notebook environment, which might not be suitable for my program's architecture (see example structure below).

Questions<br>

Is Azure Databricks suitable for this kind of task?

Are there better Azure products for deploying a complex Python program with heavy preprocessing and database interactions?

I don't mind refactoring my code or even changing its structure to make it fit with Azure Databricks.

Any guidance on the best approach for migrating my application to Azure would be greatly appreciated.

Example program structure


project_directory/

|

├──.github/

|   └── .keep

|

├── docs/

│   ├── CODE_OF_CONDUCT.md

│   ├── CONTRIBUTING.md

│   ├── db_schema.md

│   └── README.md

|   

├── img/

│   └── db_mockup.jpg

|

├── src/

|   ├── api

│   |   ├── api_base_class.py

│   |   ├── api_subclass_1.py

│   |   ├── api_subclass_2.py

│   |   ├── api_subclass_3.py

|   |   └── api_helper.py

|   |

|   ├── db

│   |   ├── db_manager_base.py

│   |   ├── db_sub_postgress.py

|   |   └── db_sub_mysql.py

|   |    

|   └── preprocessing

│   |   ├── preprocessor_base.py

│   |   ├── preprocessor_sub_1.py

│   |   ├── preprocessor_sub_1.py 

|   |   └── normaliser.py

|   |

|   ├── config.py

|   ├── build.py

|   ├── run.py

|   └── main.py

│

├── .gitattributes

├── .gitignore

├── poetry.lock

├── pyproject.toml

└── README.md
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,045 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 84,381 Reputation points Microsoft Employee
    2024-06-06T06:48:15.95+00:00

    @kman-1604 - Thanks for the question and using MS Q&A platform.

    Azure Databricks is a powerful tool for data engineering, data science, and machine learning tasks. It provides a collaborative environment for working with big data and allows you to scale your workloads easily. However, it might not be the best fit for your use case, as you mentioned that your program involves heavy preprocessing and database interactions.

    Azure provides several services that can help you deploy your Python program on the cloud. One option is to use Azure App Service, which is a fully managed platform for building, deploying, and scaling web apps. You can deploy your Python program as a web app and use Azure Database for PostgreSQL or Azure Database for MySQL as your database.

    Another option is to use Azure Functions, which is a serverless compute service that allows you to run your code on demand without having to manage any infrastructure. You can write your Python code as a function and trigger it based on events such as HTTP requests or messages from a queue.

    If you need to run your program on a virtual machine, you can use Azure Virtual Machines, which provides scalable compute capacity in the cloud. You can choose from a variety of preconfigured virtual machine images that include Python and other tools you might need.

    In summary, Azure Databricks might not be the best fit for your use case. Instead, you can consider using Azure App Service, Azure Functions, or Azure Virtual Machines to deploy your Python program on the cloud.

    Hope this helps. Do let us know if you any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    0 comments