parsing json databricks / python

braxx 436 Reputation points
2021-04-27T14:28:02.383+00:00

I am very beginner in databricks and python, so it maybe basics for you but for me it is still new.

I am trying to parse the json to get all childitems form "valid" and "date_of_creation" sections as columns in table which I will later write to parquet:

{  
	"valid": [  
		{  
			"retailer_id": 11,  
			"retailer_name": "abc",  
			"country": "AA",  
			"point_of_sales": [  
				66,  
				68655  
			]  
		},  
		{  
			"retailer_id": 22,  
			"retailer_name": "def",  
			"country": "AA",  
			"point_of_sales": [  
				2067,  
				2068,  
				68690,  
				68691  
			]  
		},  
		{  
			"retailer_id": 33,  
			"retailer_name": "ghi",  
			"country": "AA",  
			"point_of_sales": [  
				70,  
				71,  
				68694  
			]  
		}  
	],  
	"invalid": {  
		"retailers": [],  
		"points_of_sale": []  
	},  
	"date_of_creation": "04/26/2021 14:11:16"  
}  

here is my code where I get "Cannot call display(<class 'method'>)" error

#read json file  
json = spark.read.option("multiline", "true").json(input_file)  
  
#parse json  
from pyspark.sql.functions import explode, col  
  
valid = json.select(explode("valid").alias("valid"))  
valid_childitems = valid.select  
(  
  col("valid.retailer_id").alias("retailer_id"),  
  col("valid.retailer_name").alias("retailer_name"),  
  col("valid.country").alias("country"),  
  explode("valid.point_of_sales").alias("point_of_sales")  
)  
  
display (valid_childitems)  

91804-capture33.png

I followed the post: https://adatis.co.uk/parsing-nested-json-lists-in-databricks-using-python/

Can someone point me what I am doing wrong?

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,162 questions
0 comments No comments
{count} votes

Accepted answer
  1. Saurabh Sharma 23,801 Reputation points Microsoft Employee
    2021-04-28T00:50:11.013+00:00

    Hi @braxx ,

    Thanks for using Microsoft Q&A !!
    You need to keep left parenthesis - '(' on the same line with the Select statement like below -

    91902-image.png

    You can get the results as shown below -
    91828-image.png

    ----------

    Please do not forget to "Accept the answer" wherever the information provided helps you to help others in the community.

    Thanks
    Saurabh

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.