How do I use Data Factory with private resources in an Azure Hub and Spoke topology

Miles Barnard 0

I'm trying to set up a Data Factory pipeline copy job between an Azure file share hosted on a storage account and azure blob storage. The file share and the blob storage are on separate subscriptions and are connected to two separate VNETs. These VNETs are then peered to a hub VNET in another subscription. The private DNS zones are hosted in the same subscription as the hub VNET.

I've configured Azure Data Factory to use Azure Integration runtime with managed VNET and then configured the linked service to the Azure File Share with a private endpoint. However when I try to browse my file share when I add the linked service to the pipeline I get 'The remote name could not be resolved:myfileshare.file.core.windows.net' - is this due to the location of the DNS zones or am I missing some piece of information to allow it to resolve?

Many thanks

ChaitanyaNaykodi-MSFT 25,841 Reputation points Microsoft Employee

2024-08-06T22:42:12.82+00:00

@Miles Barnard

Thank you for reaching out.

Based on your statement above.

These VNETs are then peered to a hub VNET in another subscription. The private DNS zones are hosted in the same subscription as the hub VNET.

Can you please confirm you have linked these Vnets to the private DNS zone?

After you create a private DNS zone in Azure, you'll need to link a virtual network to it. Once linked, resources hosted in that virtual network can access the private DNS zone.

This set-up can help you understand how private DNS resolution works in peered VNets.

You can also deploy a test VM in the VNET of your ADF and try the DNS resolution. It will help isolate the issue.

Thank you!
Miles Barnard 0 Reputation points

2024-08-07T09:11:05.88+00:00

@ChaitanyaNaykodi-MSFT

Thanks for getting back to me! I can confirm that the VNETs are linked to the private DNS zone via Virtual Network Link. All our privately networked resources work correctly in other scenarios.

What seems to be the issue is Data Factory specifically, where when I add one of our privately networked resources such as a storage account file share as a data source in the pipeline, it fails and I can't browse to it with the error "The remote name could not be resolved: 'myfileshare.file.core.windows.net'"
Bhargava-MSFT 30,816 Reputation points Microsoft Employee

2024-08-09T16:58:52.36+00:00

To isolate the issue, can you test the DNS resolution and connectivity from a VM within the same VNET as the ADF managed VNET to check if myfileshare.file.core.windows.net resolves to the private IP correctly

also try using the IP address of the file share instead of the hostname. This can help to bypass any DNS resolution issues
Miles Barnard 0 Reputation points

2024-08-14T09:03:48.4833333+00:00

Yes it resolves to that. The issue is specifically with the Azure Data Factory integration.

Share via

How do I use Data Factory with private resources in an Azure Hub and Spoke topology

Your answer