How to ignore the records by applying an auditing filed column condition using ADF Data Flows

venkat rao 65 Reputation points
2024-05-30T09:28:46.6066667+00:00

Hi All

I am building a data transamination using mapping data flows ,I have a time stamp field Like TimeStampUpdated in the target table.
I want to lockup historical data with incremental data transamination and ignore the records coming in the incremental which are less than or equal to TimeStampUpdated .
I have added both source 1 and source 2 and joined using join to get both tables TimeStamp felids,But i am struck how to apply a condition to ignore those record's using Asserts or Conditional splits or filters .
Require your assistance

User's image

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,621 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,045 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,015 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Amira Bedhiafi 18,501 Reputation points
    2024-05-30T09:35:29.0866667+00:00

    I will start with the Join transformation (to join Source1 and Source2 based on the key fields)

    If you need to create a new column for easier filtering, add a Derived Column transformation. For instance, you can create a boolean column that indicates if the incremental record should be ignored:

    
shouldIgnore = iif(Source2.TimeStampUpdated <= Source1.TimeStampUpdated, true(), false())

    Then add a Filter transformation after the Join where you specify that you want to keep records where the TimeStampUpdated from the incremental data is greater than the historical data. 

    
Source2.TimeStampUpdated > Source1.TimeStampUpdated

    Then add a Sink transformation to write the filtered data to the target table.

    1 person found this answer helpful.