pyspark data frame for json

Question

hello all ,
i am using MS-Azure databricks platform for python+spark ,

cluster = spark 3.1.1

kindly tell me how can i import json file in it

syntax that i have used now:

df = spark.read.json("FileStore/tables/sample1.json").show(truncate = False)

and it is giving me the error :

IllegalArgumentException: Path must be absolute: FileStore/tables/sample1.json

Answer

Hello @Abdul Sattar ,

Thanks for the ask and using the Microsoft Q&A platform .

I think you forgot to mount the container .

I have tested the below code and it is working . I am using the access key from the blob .

  dbutils.fs.mount(  
  source = "wasbs://yourcontainer@ yourstorageaccount.blob.core.windows.net/",  
  mount_point = "/mnt",  
  extra_configs = {"fs.azure.account.key.yourstorageaccount.blob.core.windows.net":"acccess key from the portal"})  
df = spark.read.json("/mnt/HimanshuTest.json").show(truncate = False)  
dbutils.fs.unmount("/mnt")

Output

Thanks
Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Share via

pyspark data frame for json

1 answer

Your answer