ADF Schema validation for JSON

Daniel Berkes 1 Reputation point
2020-10-20T07:39:12.727+00:00

Hi Community,

I am implementing a feature which is validating a dataset schema, I know that in ADF there is a checkbox in the Mapping Dataflow section which does it automatically if columns or type of the columns do not match against the reference.

For CSV it works fine for me, but in case of JSON I noticed two things:

  • when I created a Dataset for the JSON its schema is different what I see in the Mapping Dataflow although I use the same dataset. I tried every option at both
    (Dataset & Data Flow) places import from sample file, import projection etc.. to have the same schema for dataset and mapping data flow.
    At the end in the Dataset I changed it manually in the code part by editing the JSON's JSON :)
  • lastly it does not solved the problem after having the same Schema it is still failing during schema validation. Basically the feature what I would like to have.
    However in the error msg itself it dispalys the same type if you see the screenshot:

Found: ArrayType(StructType(StructField(Description1,StringType,true),...etc Required: ArrayType(StructType(StructField(Description1,StringType,true),...etc

The workaround what I have is first I transform JSON to CSV and validating schema for CSV which is not ideal since I would not do transformation if the JSON is not correct.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,568 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.