Creating ADF linked service to connect to a storage account on private endpoint using service principal

Shashwat Tiwary 40 Reputation points
2024-06-10T10:28:27.29+00:00

HI,

Currently my storage account is open and allows public access.

I disabled public network access to storage and followed this article for creating a secured SFTP connection through a firewall - https://video2.skills-academy.com/en-us/azure/firewall/firewall-sftp for a future production setup. (I will use the client provided IP address ranges later based on this learning)

I was able to successfully perform this tutorial and connect to storage on pvt endpoint using SFTP client. My use case is that there are many external data provides who will share some data with us from their PC. I had to make this secure through a firewall.

However there are other things that I am unable to do now which I was able to do previously (by selecting option 'Enabled from all networks' and option 'Enabled from selected virtual networks and IP addresses' by allowing internet IP). These are listed below -

  1. Unable to access containers from portal. Image attached. So, to access the container from portal via internet which is outside VNet I created a network rule in firewall allowing port 443 from all sources to destination firewall public IP. But it did not worked. (I probably did something wrong).
  • Also, if I have to allow accessing the containers from portal from inside client VM which is on another VNet, then how do we do this?
  1. Unable to access container by creating a linked service using service principal. I followed this article - https://video2.skills-academy.com/en-us/azure/databricks/connect/storage/tutorial-azure-storage.
  2. I also have to access these storage account from databricks, KeyVault, Snowflake, etc. I am sure these would also cause an issue. So, how do I allow access from these trusted services? If I select the option 'Enabled from selected virtual networks and IP addresses', this had an option to select trusted services. So, how do I allow access from these trusted services now?

How do I go about these? I need to setup test environment by end of next week so would need a solution soon.

Thanks and Regards,

Shashwat

Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,863 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,990 questions
0 comments
Accepted answer
  1. Anand Prakash Yadav 7,700 Reputation points Microsoft Vendor
    2024-06-11T10:56:13.1+00:00

    Hello Shashwat Tiwary,

    Thank you for posting your query here!

    I understand that you’ve taken steps to secure your Azure Storage account by disabling public network access and configuring a private endpoint for SFTP connections. Let’s address the specific issues you’re facing:

    Accessing Containers from the Portal:

    When using the “Enabled from selected virtual networks and IP addresses” option, ensure that you’ve added the correct virtual network and subnet to the storage account’s firewall rules.

    To access containers from the portal via the internet (outside the VNet), you’ll need to allow traffic from your client VM’s public IP address. Make sure you’ve added the VM’s IP to the firewall rules.

    Accessing Containers from Inside a Client VM on Another VNet:

    Ensure the VNets are peered correctly to allow communication between the client VM's VNet and the VNet containing the storage account's private endpoint. Then route tables are set up to route traffic from the client VM's VNet to the storage account’s private endpoint through the peered VNet.

    Or else to allow access from a client VM in a different VNet, add the subnet of that VM to the storage account’s firewall rules.

    Access Container via Service Principal:

    · Ensure the service principal has the necessary RBAC roles (e.g., Storage Blob Data Contributor) on the storage account.

    · If the service principal operates from specific IP addresses, add these IP ranges to the storage account firewall rules.

    · When configuring linked services (e.g., in Databricks), use the storage account’s connection string or SAS token for authentication.

    Access from Trusted Services (Databricks, KeyVault, Snowflake):

    In your storage account, navigate to Networking. Enable the option to allow access from trusted Microsoft services.

    Also, for services like Databricks, KeyVault, and Snowflake, use managed identities to authenticate and access the storage account. Ensure managed identities have the necessary RBAC roles on the storage account.

    Do let us know if you have any further queries. I’m happy to assist you further.

    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest