Copy Activity Not Recognizing an Existing Column in a CSV File

Question

Copy Activity Not Recognizing an Existing Column in a CSV File

Matt Tkachev 20

Hello,

We have a simple Azure Data Factory pipeline that takes a CSV file as input and copies its contents to a Snowflake table. This setup had been working reliably for several months. However, starting last week, the Copy activity has begun failing.

The failure occurs when the Copy activity attempts to map the CSV columns to the Snowflake table. The error message (included below) indicates that one of the columns (Adjustment Note) cannot be found in the source file — even though the column still exists in the CSV.

If we remove the Adjustment Note column from the mapping, the pipeline still fails, this time not finding the next column in sequence. We’ve also tested the pipeline with older CSV files that previously loaded successfully, but the same error occurs.

I’ve attached a screenshot showing that the column does exist in the CSV file, as well as the full error message below.

Error details (Azure Data Factory):

ErrorCode=UserErrorInvalidColumnMappingColumnNotFound, 
Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,
Message=Invalid column mapping provided to copy activity: '{...,"Adjustment Note":"ADJUSTMENT_NOTE", ...}', 
Detailed message: Column 'Adjustment Note' defined in column mapping cannot be found in Source structure. 
Check column mapping in table definition.
Source=Microsoft.DataTransfer.Common

It appears that the Copy activity is no longer recognizing the source schema correctly. Could this be related to a recent ADF update?

Please advise on possible causes or next troubleshooting steps. Let me know if any other details are needed.

Thank you! CSV_Column

Smaran Thoomu 32,255 Reputation points Microsoft External Staff Moderator

2025-10-30T12:54:00.24+00:00

Matt Tkachev Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer accepted by question author

0 additional answers

Your answer

Smaran Thoomu 32,255 Reputation points Microsoft External Staff Moderator

2025-10-30T12:54:00.24+00:00

Matt Tkachev Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Amira Bedhiafi 40,576 Volunteer Moderator

Hello Matt !

Thank you for posting on Microsoft Learn Q&A.

ADF isn’t seeing the header value exactly as you expect and a small change in the CSV can make the source schema different, so the name in your mapping no longer matches.

I can see that the header looks like Adjustment Note but the actual bytes are something like \ufeffAdjustment⍽Note (BOM with non breaking space) or have trailing spaces so try to go to source dataset then preview data and expand column list to see exact detected column names. If you see weird characters that’s it.

Try to export CSV again as UTF-8 without BOM and edit the header to plain ASCII and set first row as header to false, skip header line count to 1, and define the columns explicitly and then map by ordinal position instead of by name.

Matt Tkachev 20 Reputation points

2025-10-30T14:54:46.0866667+00:00

Hi Amira,

Thanks for your message!

Even if there were unexpected characters (such as a BOM or trailing spaces), which I couldn’t find in the latest file, it’s strange that the pipeline now produces the same error for files that were processed successfully just a few weeks ago. It seems like something might have changed in ADF because the existing setup used to work fine. I also confirmed that the latest file was exported in UTF-8, so encoding shouldn't be the issue.

I tried the workaround you recommended (using ordinal positions instead of column headers), and it resolved the issue - thank you for the suggestion!

That said, we’d love to understand what might have recently changed in ADF and caused the configuration to stop working as using column headers is much more reliable.

Kind regards,

Matt
Smaran Thoomu 32,255 Reputation points Microsoft External Staff Moderator

2025-11-03T11:46:03.69+00:00

Hi Matt Tkachev

Appreciate the update, and I’m glad to hear that mapping by ordinal position resolved the issue as suggested by Amira.

You’re correct that it can be surprising when a previously stable pipeline starts failing. Recently, there have been minor backend updates in ADF that slightly changed how CSV schema inference handles header parsing - particularly with files containing encoding markers (like BOM) or hidden whitespace in header names. This can cause ADF to interpret column names differently than before.

For long-term stability, I’d recommend:

Regenerating the dataset schema from a fresh sample file to align with the current detection behavior.

Ensuring all CSVs are saved in UTF-8 without BOM and that headers contain no extra spaces or special characters.

I’ve shared your feedback with the ADF product group to help improve consistency in schema recognition.

If the workaround provided by Amira resolved your issue, please consider marking her answer as Accepted - it helps others in the community find the right solution more easily.
Matt Tkachev 20 Reputation points

2025-11-04T23:42:07.55+00:00

Hi Smaran,

I accepted Amira's answer. We'll test the long-term approach you suggested and let you know if it works.

Thanks,

Matt

Share via

Copy Activity Not Recognizing an Existing Column in a CSV File

0 additional answers

Your answer