Unable to connect to GCS from ADF

kks8589 101

Hi Team

When creating a linked service in ADF to connect to GCS, the connection is getting failed.

we were provided with the GCS service account access id and key and URI

Any other permissions need to be taken care at gcs side to establish the connection.Could anyone help us with this

Thanks & Regards

kks8589

HarithaMaddi-MSFT 10,136 Reputation points

2020-09-28T11:41:54.547+00:00

Hi @kks8589 ,

Welcome to Microsoft Q&A Platform. Thanks for posting the query.

You would need secret as well in addition to access key and URL. I have connected to my GCS from Azure data factory without additional permissions by creating a storage account and linked service works fine as below. Also, Please suggest if below prerequisites as mentioned in the documentation are taken care from GCS. If the issue persists, kindly share any snaps and additional details on region of data factory for us to investigate further.

Looking forward for your response.
kks8589 101 Reputation points

2020-09-28T12:34:58.447+00:00

Hi @HarithaMaddi-MSFT
Thanks for your response. Already provided the prerequisites steps that needs to be done at the GCS side to the GCS team from the ADF documentation.
After following those steps they have provided us the below things from the Interoperability tab
Access key Id: 62 letters
Secret Access key:41 letters
Service URL:https://storage.googleapis.com
with these things i am getting connection failed error

So the team is asking us to let them know any other setups / configurations needs to be done.

Regards
Kks
kks8589 101 Reputation points

2020-09-28T12:51:44.187+00:00

And also the GCS team mentioned they are able to connect with the credentials provided from the python script

GCS_API_ENDPOINT = 'https://storage.googleapis.com'
hmac_key = '****REDACTED****'
hmac_secret = bytearray ( '****REDACTED****', 'utf8' )
BUCKET_NAME = 'xxxxxxxxxxxxxxxxxxxx'
OBJECT_NAME = 'xxxxxxx.ndjson'
HarithaMaddi-MSFT 10,136 Reputation points

2020-09-28T13:18:08.407+00:00

Thanks @kks8589 for sharing the details. It is mentioned in google documentation as below that secret has to be 40 characters and the one I have for my account is for 40. Can you please re-check with them on that and let us know. Also, please confirm if required storage permissions are assigned to the service account from GCS IAM page.
kks8589 101 Reputation points

2020-09-28T14:36:50.91+00:00

Yeah, Access key id is of 61 characters and secret is of 40 characters.

@HarithaMaddi-MSFT Could you please help me with the screenshot of storage permission of the service account in GCS IAM page with permissions storage.buckets.get,storage.buckets.list , storage.objects.get so that will get the details or the confirmation from the GCS team
HarithaMaddi-MSFT 10,136 Reputation points

2020-09-29T08:27:14.957+00:00

Thanks @kks8589 for sharing the details. The permissions are grouped into roles in GCS. "Storage Object Creator" would be sufficient for the permissions mentioned in prerequisites as per document. Please also check this link for more details. Kindly let us know for more details.

Also, I will discuss with the internal product team to get more insights if this does not help.
HarithaMaddi-MSFT 10,136 Reputation points

2020-09-30T08:32:30.657+00:00

Hi @kks8589 ,

We have not received a response from you. Are you still facing the issue? If you found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members
kks8589 101 Reputation points

2020-09-30T15:12:52.633+00:00

Hi @HarithaMaddi-MSFT
We tried with Storage Object Viewer as well as Storage Object Admin Roles to the GCS Service Account
Still the connection from ADF is getting failed

However i tried with my own google account also now with your steps still i was unable to access the GCS with the Service account Access key id and secret.
Just sharing my google account screens that I followed

Thanks & Regards
kks
HarithaMaddi-MSFT 10,136 Reputation points

2020-10-06T08:47:54.383+00:00

Hi @kks8589 ,

Apologies for delay in responding as I was on vacation. To investigate further by connecting with you, can you please send a mail to AzCommunity[at]microsoft[dot]com with subject "Attn:Haritha - Unable to connect to GCS from ADF".
HarithaMaddi-MSFT 10,136 Reputation points

2020-10-09T15:46:37.507+00:00

Hi @kks8589 ,

Thanks for sharing all the details over mail. I have reproduced and observed the same, shared the details with Product team. I am waiting for the response from the team on further details and will get back to you as soon as I hear from them.

Thanks for your patience!

Accepted answer

HarithaMaddi-MSFT 10,136 Reputation points

2020-10-13T09:07:32.51+00:00
Hi @kks8589 ,

Thanks for your patience. I connected with Product team and below are observations

The mapping from storage.buckets.get, storage.buckets.list, or storage.objects.get to roles in GCS IAM will be included in the documentation

storage.buckets.get, storage.buckets.list is not mapped to "Storage object viewer" role in GCS IAM and this is needed to enable root level permissions. "Test connection" in linked service is checking for permissions at root level and hence without this role, the connection shows failed. Ex: "Storage Admin" can be enabled for the account

Similar to "Amazon S3" linked service in ADF as below, ADF team is working on a backlog item to use this layout for GCS connection where specific file path (Bucket) can be tested in GCS linked service

Using only "storage.objects.get" permission i.e., storage object viewer role, will enable to read from specific bucket in dataset for which account has permissions and ADF still loads the data.

Additional details will be added to documentation, thanks for your contribution.

Hope this helps! Please let us know for more queries and we will be glad to assist further.
Please sign in to rate this answer.

1 person found this answer helpful.
kks8589 101 Reputation points

2020-10-14T07:10:15.333+00:00

HI Haritha
So are you telling like only storage admin option works as of now? however the admin access cant be given to the bucket in point of GCS team due to security issues. until and unless we know the bucket name we cant get the files from gcs as of view nor we can view files in the bucket .

HarithaMaddi-MSFT 10,136 Reputation points

2020-10-14T07:31:49.76+00:00

Hi @kks8589 ,

Thanks for the response. Yes, if we are limiting the permissions to object roles and due to security reasons, cannot enhance it to bucket roles, then we cannot browse the buckets. In this scenario, as account does not have permission to search through buckets, ADF product team said it is not possible without giving those roles today.

Please let us know if it clarifies the query.

HarithaMaddi-MSFT 10,136 Reputation points

2020-10-16T08:50:55.847+00:00

Hi @kks8589 ,

We have not received a response from you. Please let us know if above details clarified the question, if not, we will be glad to assist further.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

kks8589 101 Reputation points

2020-10-19T04:59:02.447+00:00

HI Haritha,
Thanks for the response..when previously Storage Admin permission is given for a service account even then the connection has failed in linked service. Anyways the GCS team is not ready to provide storage admin permisions. and as a confirmation with storage object viewer permission we can access the bucket but the linked service connection will befailing.

Using only "storage.objects.get" permission i.e., storage object viewer role, will enable to browse buckets from dataset for which account has permissions and ADF still loads the data.

from the above comment i can confirm that i cant browse the bucket jut i can provide its name and file name if i know them

Thanks

HarithaMaddi-MSFT 10,136 Reputation points

2020-10-19T08:45:59.117+00:00

Thanks @kks8589 for confirming the details. Unfortunately, without the higher permissions, it is not possible to browse buckets today. To give more clarity, I edited that line in the above answer, your understanding is right. I would recommend you also to post an idea in feedback forum which is closely monitored by data factory product team and will prioritize the implementation of new features.

---------------------------------------------------------------------------------------------------------

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

Want a reminder to come back and check responses? Here is how to subscribe to a notification.

HarithaMaddi-MSFT 10,136 Reputation points

2020-10-20T06:32:47.737+00:00

Hi @kks8589 ,

We have not received a response from you. Please let us know if above details clarified the question, if not, we will be glad to assist further.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

HarithaMaddi-MSFT 10,136 Reputation points

2020-10-21T08:29:30.447+00:00

Hi @kks8589 ,

We still have not heard back from you. Following up to check if above details clarified the question, if not, we will be glad to assist further.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

kks8589 101 Reputation points

2020-10-23T05:48:53.657+00:00

Thanks for the info provided.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Unable to connect to GCS from ADF

0 additional answers

Your answer