Azure Databricks fails

Catherine 刘 30

Hello,

In the databricks notebook which is provided by Microsoft training classes, when I tried to import => read a data (csv or json) like

path = source + "/wikipedia/pagecounts/staging_parquet_en_only_clean/"
files = dbutils.fs.ls(path)
display(files)

, there were"server failed" message each time:

shaded.databricks.org.apache.hadoop.fs.azure.AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

I do not quite understand this message and can not fix this by myself.

PS: I am using a Premium - 14 days Free DBUs for practicing.

Thank you in advance for your reply

PRADEEPCHEEKATLA-MSFT 84,051 Reputation points Microsoft Employee

2023-07-21T08:10:54.5+00:00
@Catherine 刘 - Thanks for the question and using MS Q&A platform.

It seems like you are encountering an authentication issue while trying to read data from Azure Databricks. The error message indicates that the server failed to authenticate the request, and it suggests that you check the value of the Authorization header to ensure that it is formed correctly, including the signature.

This error message usually occurs when the credentials used to access the storage account are incorrect or have expired. To resolve this issue, you can try the following steps:

Check that the credentials you are using to access the storage account are correct. You can verify this by checking the access key and account name in the Azure portal.

Ensure that the access key you are using has not expired. If it has, you can regenerate a new access key in the Azure portal.

Check that the storage account firewall is not blocking access to the storage account. You can verify this by checking the firewall settings in the Azure portal.

Ensure that the storage account is not in read-only mode. If it is, you will not be able to write data to the storage account.

If none of these steps resolve the issue, you can share the screenshot of the code which you are trying along with the stack trace of the error message?
PRADEEPCHEEKATLA-MSFT 84,051 Reputation points Microsoft Employee

2023-07-25T05:10:53.94+00:00

@Catherine 刘 - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Catherine 刘 30 Reputation points

2023-07-25T08:41:41.9533333+00:00

Actually, the probem is still here. In the "storage account" - "access keys", there is a check to detect common connectivity issues in order to find out why unable to load data. After i run it, i had a "DenyAssignmentAuthorizationFailed" with error code 403.

and i did not see the option for regenerate new access key in my account either.
PRADEEPCHEEKATLA-MSFT 84,051 Reputation points Microsoft Employee

2023-07-25T16:09:22.1133333+00:00

@Catherine 刘 - Could you please help us understand which storage account are you using? Is it the storage account available in the Managed Resource Group of your Databricks workspace as shown below:
Catherine 刘 30 Reputation points

2023-07-25T19:13:01.0333333+00:00

Yes

Accepted answer

PRADEEPCHEEKATLA-MSFT 84,051 Reputation points Microsoft Employee

2023-07-27T08:21:48.85+00:00
@Catherine 刘 - Thanks for the question and using MS Q&A platform.

There are two things regarding storage accounts & Databricks:

Databricks automatically creates a storage account for each workspace to hold so-called DBFS Root. This storage account is meant to be used to keep only temporary data, libraries, cluster logs, models, etc. It's not designed to keep the production data as this storage account isn't accessible outside of the Databricks workspace.

Databricks can work with storage accounts created outside of the workspace (documentation) - just create a dedicated storage account to keep your data, and access it using the abfss protocol as described in the documentation, or mount it into workspace (although it's not recommended anymore). And then you can access that storage account from Synapse & other tools as well.

The one which you are trying to access is an internal databricks storage account which you cannot modify any settings or you cannot assign any permissions on the storage account. If you try to modify you will experience this error message: DenyAssignmentAuthorizationFailed.

Note: You need to try to access the storage account other than default storage, if you wan t to access using dbutils.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.
Catherine 刘 30 Reputation points

2023-07-27T08:44:56.5533333+00:00

Thanks for the explaintation. I understood the differences between them. For my "authentication issue", shall I need to create an extra storage account for DataBricks? I created the Databricks by following the "Create an Azure Databricks workspace and cluster" from Coursera "Perform data science with Azure Databricks" instructions.

PRADEEPCHEEKATLA-MSFT 84,051 Reputation points Microsoft Employee

2023-07-31T08:12:01.4566667+00:00

@Catherine 刘 - Glad to know it helped. Going forward try to use MS Q&A platform for any question related to Azure.

Choi, Seung-Rak 0 Reputation points

2024-06-29T15:20:43.3166667+00:00

Hello, I am experiencing exactly the same issue with the same course.

and I do see the same screenshots you shared, but I am not sure exactly what the resolution is. From my observation and debugging, it seems that the materials are simply unavailable.

Choi, Seung-Rak 0 Reputation points

2024-06-29T15:25:30.5933333+00:00

In the code provided, the variable "source" is [wasbs://training@dbtraineastus.blob.core.windows.net]

In other words, the entire path is wasbs://training@dbtraineastus.blob.core.windows.net/wikipedia/pagecounts/staging_parquet_en_only_clean/

and I get the same error message as above.

There was no other instructions regarding any action that needs to be taken to gain access to the folder and its content.

In the code provided, the variable "source" is [wasbs://training@dbtraineastus.blob.core.windows.net]

In other words, the entire path is wasbs://training@dbtraineastus.blob.core.windows.net/wikipedia/pagecounts/staging_parquet_en_only_clean/

and I get the same error message as above.

There was no other instructions regarding any action that needs to be taken to gain access to the folder and its content.

Apparently, wasb became deprecated, but I found a link - https://video2.skills-academy.com/en-us/azure/databricks/archive/storage/wasb-blob

According to this link, it looks like I need to have something called a storage account access key, which was never provided.

I submitted this issue to Coursera a number of times, but they just seem to be too busy to take care of this.
Sign in to comment

Share via

Azure Databricks fails

0 additional answers