Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,466 questions
Getting error when connecting to Azure Data Lake Storage Gen 2 ABFSS using spark from local
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 96%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKJ%3C/text%3E%3C/svg%3E)
K, Jansee
1
Reputation point
Error is Exception in thread "main" java.io.IOException: No FileSystem for scheme: abfss
Log is below
20/11/10 22:58:18 INFO SharedState: Warehouse path is 'file:/C:/sparkpoc/spark-warehouse'.
20/11/10 22:58:18 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
20/11/10 22:58:19 WARN FileStreamSink: Error while looking for metadata directory.
Exception in thread "main" java.io.IOException: No FileSystem for scheme: abfss
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2586)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2593)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2632)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2614)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:547)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:545)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft 19,476 Reputation points Microsoft Employee2020-11-13T01:09:27.937+00:00 Hello anonymous user,
Thanks for the ask and using the forum .
Not sure if you already gone through this , it does talk about the error , hopefully it helps .Thanks
Himanshu -
HimanshuSinha-msft 19,476 Reputation points Microsoft Employee
2020-11-16T22:50:12.763+00:00 Hello ,
We have not heard back from you on this and was just following up ,if the issue is resolved .In case if you have a resolution we request you to share that with the community .
Thanks
Himanshu -
K, Jansee 1 Reputation point
2020-12-03T08:00:06.98+00:00 Hi HimanshuSinha-MSFT ,
I am trying to connect Gen2Datalake using scala code from spark (IntelliJ) not using databricks jars. I am getting Exception in thread "main" java.io.IOException: No FileSystem for scheme: abfss
But When I try to connect from Spark shell from local, I could connect and read data from Gen2 Datalake. Pls help
Sign in to comment
Sign in to answer