Getting Socket error code timed out while connecting to a SFTP server via Azure data factory

Ridhima Verma 25 Reputation points
2024-09-04T04:18:44.96+00:00

I am very new to Azure and

My Use Case is that, I want to move the file from my Azure blob storage to a third party SFTP server.

I got to know that Azure data factory will help me to move the files out by creating linked services from source and target through pipelines.

I am able to create a linked service from the Azure blob storage but when I am trying to create a SFTP linked service, I am getting below error.

"Meet network issue when connect to Sftp server '185.204.122.2', SocketErrorCode: 'TimedOut'.A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond"

I believe the IP needs to be whitelisted at the SFTP server.

THerefore I started to find my ADF IPs. I got two ways to do that and got problems in both:

  1. I created a self hosted Integration Runtime and copied its IP under the heading 'Nodes'. But when I got this whitelisted, I found the IP changed the next day magically. Hence my connection is not yet established.
  2. I downloaded the JSON file containing list of IP ranges in Azure and I checked my storage account region and picked up the IPs under 'AzuredataFactory.westeurope'. But I don't see the IP which I got from self hosted IR node in the IP ranges I found from JSON. I am not sure if this is the correct way of doing it.

Can you please suggest , which IPs should I get whitelisted to get this connection working ?

Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,787 questions
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
1,175 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,566 questions
Accepted answer
  1. Vinodh247 18,101 Reputation points
    2024-09-04T05:22:48.06+00:00

    Hi Ridhima Verma,

    Thanks for reaching out to Microsoft Q&A.

    The issue you're experiencing is likely related to the dynamic nature of the IP addresses used by Azure services, like ADF when using SHIR. The discrepancy you're observing between the IP address from your Self-hosted Integration Runtime (IR) and the IP ranges listed under AzureDataFactory.westeurope in the JSON file is because the IP ranges in the JSON file correspond to Azure services' public IP addresses, particularly when using Azure IR rather than SHIR.

    Key Points to Consider:

    1. Self hosted IR vs. AzureIR:
      • SHIR runs on your on-premises infrastructure or a virtual machine within your network, and it uses the outbound IP address of that machine or network, not the IP ranges listed in the JSON file.
      • Azure IR uses a set of public IP addresses provided by Azure, which are listed in the JSON file under your region (ex., AzureDataFactory.westeurope).
    2. IPs in SHIR:
      • The IP address you see from your SHIR is the public IP of the environment where the IR is hosted. If this environment's IP is dynamic (ex., if it’s hosted in a cloud environment with dynamic IPs), it can change, which is why it's not listed in the static ranges provided in the JSON file.
    3. Whitelisting Approach:
      • If you are using SHIR, you need to whitelist the public IP of the machine or environment where the SHIR is running. You might have to configure your infrastructure to use a static public IP if the current setup changes IP addresses dynamically.
      • For Azure IR, you would whitelist the range of IPs provided in the JSON file under AzureDataFactory.<region>.

    Using Azure IR:

    • If feasible, switch to using Azure IR. This will allow you to use the IP ranges from the JSON file for whitelisting. These IP ranges are static and tied to the region of your Azure Data Factory.

    Ensure StaticIP for SHIR:

    • If you must use SHIR, ensure that the environment where it is running uses a static public IP address. You can then provide this static IP to the SFTP server administrators for whitelisting.

    Verify with Network Setup:

    • Double-check your network setup to make sure that the Self-hosted IR's IP address remains consistent, or use a service like Azure Virtual Network NAT, which can assign a consistent outbound IP address for all traffic from your virtual network.

    Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Ridhima Verma 25 Reputation points
    2024-09-04T18:17:58.57+00:00

    Thank you for the answer Vinodh, this was really helpful to make my connection worked.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.