Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,566 questions
Azure Data Factory (ADF) dataflow seems to pick the wrong endpoint when Data Preview
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 94%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKP%3C/text%3E%3C/svg%3E)
Karydakis, P (Panagiotis)
5
Reputation points
I have created a storage account/blob storage StorageV2 in Azure. Inside it I have created a container. Storage account automatically creates endpoints for all the provided services
Then in ADF I have created a linked service (user assigned managed identity/ private endpoint) which is pointing to azure blob storage/ file.csv
The connection is tested and works fine.
Later on, I create a dataset using this linked service and I capable of previewing the data just fine
Finally, inside a new dataflow I create a source activity when I specify the above mentioned dataset.
When I hit the Data Preview tab I get the below error
This happens only inside a dataflow
From the error message I realize that ADF send the request to the wrong endpoint (data lake instead of blob service).
Why does this happen?
How can I change it?
{count} vote
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Bhargava-MSFT 30,816 Reputation points Microsoft Employee2024-08-28T17:10:28.9966667+00:00 Hello Karydakis, P (Panagiotis),
Can you please confirm if the Vnet was whitelisted in the firewall?
If Vnet was not whitelisted, then most likely the issue is due to the request went to public endpoint instead of private endpoint (because of the firewall blocking the request)
Resolution:
Can you add the subnet where the request is coming from to the Storage Firewall, this will help prevent Firewall from blocking the requests coming from the public endpoint. -
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 59%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EX%3C/text%3E%3C/svg%3E)
xlve 0 Reputation points2024-08-28T18:23:17.1366667+00:00 @Bhargava-MSFT I am having the same issue, as you said opening the firewall fixes the issue with the data preview, but why is that i need to open the firewall for data preview when i am using a manage private end point? Also the preview works correctly from the data set with the storage public network access set to disabled.
Trying to understand the issue as the requirement is to disable public network access.
-
xlve 0 Reputation points
2024-08-28T18:26:32.04+00:00 @Bhargava-MSFT why is that we need to change the firewall if we have a managed private end point and the preview works under the data source? I have the same issue but I don't want to enable public access, I want to use private managed end points.
-
Bhargava-MSFT 30,816 Reputation points Microsoft Employee
2024-08-28T19:13:53.7566667+00:00 Hello xlve,
As I pointed out, the request went to public endpoint instead of private endpoint.
Are you using any custom DNS server at your end?
Please loop your Network admin and architect to investigate why the request went to public endpoint instead of private endpoint.
Are you seeing the same behavior in other environments too?
-
xlve 0 Reputation points
2024-08-28T22:11:51.7833333+00:00 reviewing DNS setting now, but why it would fail only during data preview in the data flow but works when doing preview at the data set level?
-
Bhargava-MSFT 30,816 Reputation points Microsoft Employee
2024-08-28T23:34:37.0966667+00:00 I don't have the visibility into the internals of the backend system when the 'Data Preview' button is clicked. However, the root cause is that the request went to public endpoint instead of private endpoint.
I hope this helps.
-
Karydakis, P (Panagiotis) 5 Reputation points
2024-08-29T07:25:42.8766667+00:00 @Bhargava-MSFT thanks for your answers. Just for my understanding, if for some reason the requests goes through the public network then it automatically picks the data lake endpoint instead of the blob service endpoint.
Any workarounds (Tbh i dont think that my request goes through the public network but I ll investigate it) or ideas on how we can clarify it even more?
BTW (for other people): https://stackoverflow.com/questions/78923674/azure-data-factory-adf-dataflow-seems-to-pick-the-wrong-endpoint-when-data-pre
-
Bhargava-MSFT 30,816 Reputation points Microsoft Employee
2024-08-29T13:52:22.1766667+00:00 I suggest you open a support case and involve your network engineer and architect in a call with the Microsoft support representative. This will help them better understand your network architecture and provide the appropriate next steps.
I hope this helps.
-
Bhargava-MSFT 30,816 Reputation points Microsoft Employee
2024-09-03T21:33:50.9166667+00:00 I am checking to see if you have any further questions here.
Sign in to comment
Sign in to answer