Hello Zhuoqiong Mo,
Welcome to the Microsoft Q&A and thank you for posting your questions here.
Problem
I understand that you are having issue with your Copy Data Activity in Azure Data Factory.
Solution
To solve the issue, after reviewing the information provided. I saw that error message indicates an issue related to Parquet schema validation, which means that the schema is empty or incorrectly defined for the dynamic property you're using to pass the TempoKey
values to the Copy Data Activity.
Now, I saw that in your integration dataset settings, you've configured the dynamic property, to ensure that the TempoKey
values are correctly passed to the Copy Data Activity, kindly follow these steps:
- In your ForEach activity, make sure that the dynamic property expression is correctly set to
@item().TempoKey
. This will ensure that each value from the Lookup activity is passed to the Copy Data Activity. - In the Copy Data Activity, check the mapping settings for the source and destination datasets to ensure that the source dataset (CSV file) correctly maps the
TempoKey
column to the appropriate field. Also, verify that the destination dataset (API connector) maps theTempoKey
value to the correct field in the API request. - Before running the pipeline, use the data preview feature to verify that the
TempoKey
values are correctly passed from the Lookup activity to the Copy Data Activity and ensure that the data looks as expected in the preview. - Since you're encountering a Parquet-related error, double-check the format settings for the destination dataset. Meanwhile, if you're writing data to a Parquet file, ensure that the schema matches the expected structure (including the
TempoKey
column). - Finaly, make sure that the integration runtime (IR) associated with both the source and sink (destination) data stores is correctly configured and accessible.
References
Kindly use the additional resources provided by the right side of this page for more reading and clarifications.
Accept Answer
I hope this is helpful! Do not hesitate to let me know if you have any other questions.
** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.
Best Regards,
Sina Salam