Remove Duplicate Columns in Azure Data Factory

Dinesh Prajapati 126 Reputation points
2022-12-29T07:05:14.477+00:00

In Azure ADF, if there are two duplicate columns. lets says address & address , what ADF does is it make first "address" as address 1 and second "address " as "address some suffix". This causes issue when the data goes to sink, because it does not recognise address1. In the database, the column name is "address".
This is the issue.

Could you please let me know when we can handle this in Azure Data factory.

The second duplicate column is getting removed, but since the first columnname "address" gets changed to "address1", it causes issue.

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,175 questions
Azure Lab Services
Azure Lab Services
An Azure service that is used to set up labs for classrooms, trials, development and testing, and other scenarios.
297 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. AnnuKumari-MSFT 32,906 Reputation points Microsoft Employee
    2023-01-02T05:45:13.103+00:00

    Hi @Dinesh Prajapati ,

    Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

    As I understand your query, you are trying to handle duplicate column names from dataset using Azure data factory pipelines. Kindly check if the below steps help resolving your issue:

    1. Ignore the column headers in the source dataset.

    275287-1.png

    1. Update the mappings

    275288-2.png

    1. Skip line count in the source so that headers are ignored while copying the data

    275299-3.png

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.