HD Insights - Script Action stored in ADLSGen2

Brendan Cassidy 26 Reputation points Microsoft Employee
2020-10-26T20:12:52.64+00:00

When trying to run Bash scripts as Script Actions on Cluster creation is there a way to use a ADLSGen2 location? From the current documentation it shows ADLGen1, and public HTTPS requests but does not discuss this possibility with Gen2.

In scripts within HDInsights to copy over Gen2 data can be done with abfs://<container>@<storageaccount>.dfs.core.windows.net/ (or abfs:/// if its the default file location).

However, am retrieving an error of 'abfs unsupported shema' in the script action property it seems.
I have also tried the other formats and all are giving Not Found errors -- my guess is because it isn't a public lake, which is intentional:
https://<storageAccount>.blob.core.windows.net/<fileSystem>/Script_action.sh
https://<storageAccount>.dfs.core.windows.net/<fileSystem>/Script_action.sh

Documentation I could find was here but unspecific to Gen2.
https://video2.skills-academy.com/en-us/azure/hdinsight/hdinsight-hadoop-customize-cluster-linux
https://video2.skills-academy.com/en-us/azure/hdinsight/create-cluster-error-dictionary

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
207 questions
Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 85,346 Reputation points Microsoft Employee
    2020-10-28T04:15:01.173+00:00

    Hello @Brendan Cassidy ,

    To use Azure Data Lake Gen2 path, you need to pass the URI with SAS token.

    Hope this helps. Do let us know if you any further queries.

    ------------

    • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification.
    0 comments

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest