Edit: I am rewriting this to clarify the ask. Relatively new to Databricks. I am trying to understand how outbound traffic from clusters is determined. It seems to differ if SCC is enabled vs when it's not. With no SCC: VMs start up with a dedicated public IP. Traffic is sourced from it. With SCC: VMs start with no public IP. Outbound traffic appears to come from ranges identified at https://video2.skills-academy.com/en-us/azure/databricks/resources/supported-regions#outbound I have an existing workspace that cannot have SCC enabled at this time, but I need to whitelist the IPs that these VMs would be coming from. I just can't seem to find any information as to what configures these dedicated public IPs, and from what ranges. TIA

@McDonald, Matthew - Thanks for the question and using MS Q&A platform. Based on the information you provided, it seems that your Databricks workspace clusters are not using the documented outbound NAT ranges. This could be because SCC is disabled and the clusters are using their own dedicated public IP addresses. In this case, you can try the following steps to determine the public IP addresses/ranges that your clusters are using: Check the Azure portal for the public IP addresses associated with the virtual machines (VMs) that are running your Databricks clusters. You can find this information by navigating to the VMs in the Azure portal and looking at the "Public IP address" field. Once you have the public IP addresses, you can use a tool like IP2Location to determine the IP ranges that they belong to. This will give you an idea of the IP ranges that your clusters are communicating on. If the IP ranges you find are not the same as the documented outbound NAT ranges, you can try whitelisting the specific IP addresses instead of the entire range. Alternatively, you can try enabling SCC to use the documented ranges. Hope this helps. Do let us know if you any further queries. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

How do I figure out what public IP ranges my Databricks workspace clusters are coming from?

Accepted answer

PRADEEPCHEEKATLA-MSFT 89,376 Reputation points Microsoft Employee

2024-05-20T05:42:40.1833333+00:00
@McDonald, Matthew - Thanks for the question and using MS Q&A platform.

Based on the information you provided, it seems that your Databricks workspace clusters are not using the documented outbound NAT ranges. This could be because SCC is disabled and the clusters are using their own dedicated public IP addresses.

In this case, you can try the following steps to determine the public IP addresses/ranges that your clusters are using:

Check the Azure portal for the public IP addresses associated with the virtual machines (VMs) that are running your Databricks clusters. You can find this information by navigating to the VMs in the Azure portal and looking at the "Public IP address" field.

Once you have the public IP addresses, you can use a tool like IP2Location to determine the IP ranges that they belong to. This will give you an idea of the IP ranges that your clusters are communicating on.

If the IP ranges you find are not the same as the documented outbound NAT ranges, you can try whitelisting the specific IP addresses instead of the entire range. Alternatively, you can try enabling SCC to use the documented ranges.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.
McDonald, Matthew 186 Reputation points

2024-05-24T22:19:01.9233333+00:00

Perhaps my original post was not clear. I am trying to find documentation that explains the outbound behavior when SCC is enabled vs not. Additionally, when SCC is not enabled and the VMs get public IPs assigned directly to them, what controls that? What range are those IPs expected to come from? Is this documented anywhere?

These VMs spin up and down as part of workflows. Every time they restart they get a new public IP, so trying to identify ranges by working backwards from the VM's pov will not work.

PRADEEPCHEEKATLA-MSFT 89,376 Reputation points Microsoft Employee

2024-05-27T04:15:56.2+00:00

@McDonald, Matthew - I apologize for the confusion. Let me clarify my previous response and address your additional questions.

When SCC is enabled on your Databricks workspace, VMs start with no public IP and outbound traffic appears to come from ranges identified at https://video2.skills-academy.com/en-us/azure/databricks/resources/supported-regions#outbound. These IP address ranges are the ones that you need to whitelist to allow outbound traffic from your Databricks workspace clusters.

When SCC is not enabled and the VMs get public IPs assigned directly to them, the public IP addresses are assigned by Azure. The IP addresses are assigned from a pool of public IP addresses that are owned by Microsoft. The range of IP addresses that are used for this purpose is not documented, as it is subject to change.

Since the VMs spin up and down as part of workflows and get a new public IP every time they restart, it may not be practical to identify the IP address ranges by working backwards from the VM's point of view. In this case, you may need to whitelist all outbound traffic from your Databricks workspace clusters to the internet.

I hope this helps clarify the outbound behavior when SCC is enabled vs not, and how public IP addresses are assigned when SCC is not enabled. Let me know if you have any further questions.

McDonald, Matthew 186 Reputation points

2024-05-27T19:52:22.22+00:00

you may need to whitelist all outbound traffic from your Databricks workspace clusters to the internet.

How would I do that if I do not know the IP ranges this traffic comes from? I guess I will have to submit a support ticket to see if they can provide more detailed information.

PRADEEPCHEEKATLA-MSFT 89,376 Reputation points Microsoft Employee

2024-05-28T02:06:14.0233333+00:00

@McDonald, Matthew - . If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?

For more details, refer to How to create an Azure support request.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How do I figure out what public IP ranges my Databricks workspace clusters are coming from?

0 additional answers

Your answer