ADF Schema validation for JSON
Hi Community,
I am implementing a feature which is validating a dataset schema, I know that in ADF there is a checkbox in the Mapping Dataflow section which does it automatically if columns or type of the columns do not match against the reference.
For CSV it works fine for me, but in case of JSON I noticed two things:
- when I created a Dataset for the JSON its schema is different what I see in the Mapping Dataflow although I use the same dataset. I tried every option at both
(Dataset & Data Flow) places import from sample file, import projection etc.. to have the same schema for dataset and mapping data flow.
At the end in the Dataset I changed it manually in the code part by editing the JSON's JSON :) - lastly it does not solved the problem after having the same Schema it is still failing during schema validation. Basically the feature what I would like to have.
However in the error msg itself it dispalys the same type if you see the screenshot:
Found: ArrayType(StructType(StructField(Description1,StringType,true),...etc Required: ArrayType(StructType(StructField(Description1,StringType,true),...etc
The workaround what I have is first I transform JSON to CSV and validating schema for CSV which is not ideal since I would not do transformation if the JSON is not correct.