Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,567 questions
Custom error handling Azure Data Factory
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 92%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPV%3C/text%3E%3C/svg%3E)
Poel van der, RE (Ron)
451
Reputation points
Hi
We want to create custom error messages in our pipelines like:
- File is missing a header record
- Flow abended: too many incomplete records will be added
- Too many error records
How are people handling these errors in a SQLDB and non-SQL (only Parquet) environment if the requirements are that:
- it should be visible which pipeline abended
- on which time it abended
- the error message
?
Does ADF provide standard handling of (custom) error messages? ADF's own error messages are written to the log. Can I extract these from the log?
I heard things like Azure Function, Logic App, setting Variables and writing to SQL etc. etc.
Looking forward to hear
Regards
Ron
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma 23,801 Reputation points Microsoft Employee2020-11-18T19:23:45.703+00:00 @Poel van der, RE (Ron) Thanks for posting your ask on Microsoft Q&A.
Can you please elaborate on your second point - Flow abended: too many incomplete records will be added ? What exactly you mean by flow abandoned ? -
Poel van der, RE (Ron) 451 Reputation points
2020-11-19T13:45:01.323+00:00 Hai @Saurabh Sharma
It's just a description of an error message I made up. Could also have been 'Too many incomplete records will be added'. With 'flow abended' I can mean 'data flow abended' or 'pipeline abended'
regards
Ron -
Saurabh Sharma 23,801 Reputation points Microsoft Employee
2020-11-20T22:44:45.28+00:00 @Poel van der, RE (Ron) Ok, I get it. Sorry, but custom error handling is not supported in ADF currently and you may need to go out of the box to have some logic in your ADF pipeline. You can find similar kind of feature requests in UserVoice -
You can ask to add additional capabilities to those features or you can submit a new feature request at the same place.
Sign in to comment
Sign in to answer