We are a data team of 4 people. To make the process easy and more productive. Can we separate dev/prod environments at Databricks catalogue level rather than the workspace level? Can anyone share any thoughts on this? Thanks

@vicman - Thanks for the question and using MS Q&A platform. You could of course setup such environment. 2 seperate catalogs with save table names within. Catalogname would be then a parametername to your job definition. You have then a staging of your data assets but not your infrastructure. If you change/have an incident in dev environment, your prod would be done too (databricks and storage account resource). Depends on how risk affine you are. What would be your benefit? I dont see any benefit of doing this. For more details, refer to the below articles: Functional Workspace Organization on Databricks Isolation of Environments on the Databricks Data Intelligence Platform Hope this helps. Do let us know if you any further queries.

Databricks Dev/Prod setup

Accepted answer

PRADEEPCHEEKATLA-MSFT 89,296 Reputation points Microsoft Employee

2024-06-07T09:46:54.2766667+00:00

@vicman - Thanks for the question and using MS Q&A platform.

You could of course setup such environment. 2 seperate catalogs with save table names within. Catalogname would be then a parametername to your job definition. You have then a staging of your data assets but not your infrastructure. If you change/have an incident in dev environment, your prod would be done too (databricks and storage account resource). Depends on how risk affine you are. What would be your benefit? I dont see any benefit of doing this.

For more details, refer to the below articles:
Functional Workspace Organization on Databricks

Isolation of Environments on the Databricks Data Intelligence Platform
Hope this helps. Do let us know if you any further queries.
Please sign in to rate this answer.

1 person found this answer helpful.
vicman 20 Reputation points

2024-06-10T02:28:52.0166667+00:00

Thank you @PRADEEPCHEEKATLA-MSFT , can we achieve set-up simplicity and cost-saving by having 2 separate catalogues rather than 2 workspaces? Hence, we want to understand the benefits and drawbacks of each solution. Sounds like most people would recommend multiple workspace solutions.

vicman 20 Reputation points

2024-06-10T02:29:10.44+00:00

Those two links have been amazing.

Functional Workspace Organization on Databricks

Isolation of Environments on the Databricks Data Intelligence Platform

PRADEEPCHEEKATLA-MSFT 89,296 Reputation points Microsoft Employee

2024-06-10T05:09:46.5166667+00:00

@vicman -It is generally recommended to use separate workspaces for different purposes, such as separating SOC and non-SOC data, to avoid double ingestion costs. However, having separate catalogs within a single workspace can also be a viable solution depending on your specific needs.

Using separate catalogs within a single workspace can simplify setup and potentially save costs, but it may also limit scalability and flexibility in the long run. For example, if you need to scale up your resources for one catalog, it may affect the performance of the other catalog. Additionally, if you need to grant different levels of access to different teams or users, it may be more difficult to manage within a single workspace.

On the other hand, using separate workspaces can provide more flexibility and scalability, as each workspace can be optimized for its specific purpose. It also allows for easier management of access and permissions for different teams or users. However, it may require more setup and potentially higher costs.

Ultimately, the decision between using separate catalogs within a single workspace or separate workspaces should be based on your specific needs and priorities.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Databricks Dev/Prod setup

0 additional answers

Your answer