SSIS 2019 - Identify and Redirect Duplicate Values in Data Flow

libpekin 86 Reputation points
2024-06-23T20:12:21.84+00:00

Hello,

To be clear this is not a requirement to identify and redirect or dedupe rows in the data flow task leveraging the Sort Transformation. Instead, the requirement is to identify the duplicate values in a column then handle affected rows differently. Using the attached mock-up table, ProductID 1002 will quality as duplicates, therefore rows 3 and 4 should flow down a different path. Likewise, ProductCode A01 resulting in rows 2 and 6 being redirected.

Any help is appreciated. Thanks!

User's image

SQL Server Integration Services
SQL Server Integration Services
A Microsoft platform for building enterprise-level data integration and data transformations solutions.
2,502 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. libpekin 86 Reputation points
    2024-06-26T20:08:02.1833333+00:00

    Having tested we learned that the sorts, required for deduping each column, will cause performance issues on large datasets. As a result, we've taken an approve to depude the data on the database instead of in the data flow.

    0 comments