Invalid records failed in DQ checks

@Anshal - Thanks for the question and using MS Q&A platform.

When it comes to handling records that fail DQ checks, whether to use a soft stop or hard stop depends on your specific use case and business requirements. If you want to allow processing to continue even if a record fails multiple times, then a soft stop would be a better design. On the other hand, if you want to stop the processing completely if a record fails multiple times, then a hard stop would be a better design.

As for getting the count of a specific record that failed multiple times, adding an extra column to track the count is a logical approach. However, if you are dealing with a large volume of data, this approach may not be scalable. In that case, you can consider using a distributed computing framework like Apache Spark to process the data and get the count of specific records that failed multiple times.

Using Spark's groupBy and count functions, you can group the data by the record ID and count the number of times it failed. This approach is scalable and can handle large volumes of data.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.