ADF Data Flow Mapping shows 0 rows written to Dataverse sink

Noah Gilbert 26

Running a simple set of pipelines with data flow mappings to Dataverse sinks. Data flow runs successfully, but Rows Written through interface consistently shows 0 rows. I can see the rows were written by going to the table in Dataverse.

phemanth 10,330 Reputation points Microsoft Vendor

2024-09-27T10:50:45.9833333+00:00
@Noah Gilbert

Thanks for reaching out to Microsoft Q&A

you’re encountering a known issue where the “Rows Written” metric in Azure Data Factory (ADF) shows 0, even though the data is successfully written to the Dataverse sink.

This can happen due to certain limitations or configurations in ADF.

Here are a few things you can check:

Known Limitations: There are known limitations with some sink types where the “Rows Written” metric might not be accurately reported.

For example, this issue is known to occur with Delta sinks.

Sink Transformation Settings: Ensure that your sink transformation settings are correctly configured. Sometimes, specific settings or schema projections might need to be overridden.

Error Handling: Check if there are any errors being logged that might affect the row count reporting. You can set the sink error row handling to “Continue on Error” to ensure that errors don’t stop the data flow.

Timeout Settings: If the operation times out while writing data, it might not report the rows correctly. Increasing the timeout value in the sink transformation settings can help.

If you’ve checked these and the issue persists, please share your data flow configuration and sink settings screenshots if possible
Noah Gilbert 26 Reputation points

2024-09-27T19:41:39.58+00:00

Screenshots below:
Note: df ran successfully, 0 Rows written, Stages shows "-" for rows
Not a inline table
No options to configure error handling to continue on error
Noah Gilbert 26 Reputation points

2024-09-27T19:44:21.4733333+00:00

Attempted to comment but comment isnt showing up despite no error showing, so trying again.

1 answer

Pinaki Ghatak 4,210 Reputation points Microsoft Employee

2024-09-27T18:36:57.54+00:00

Hello @Noah Gilbert

It's possible that the metrics for the sink are not being captured correctly. If a sink has zero rows written, it will not show up in metrics. Existence can be verified using the contains function. For example, contains(activity('dataflowActivity').output.runStatus.metrics, 'sink1') will check whether any rows were written to sink1.

To get the number of rows written to a sink named 'sink1' in an activity named 'dataflowActivity', use @activity('dataflowActivity').output.runStatus.metrics.sink1.rowsWritten.

To get the number of rows read from a source named 'source1' that was used in that sink, use @activity('dataflowActivity').output.runStatus.metrics.sink1.sources.source1.rowsRead.

If you are still having issues, it may be helpful to check the debug logs for any errors or warnings related to the sink. You can also try running the data flow from a pipeline and checking the output in the pipeline run details to see if the rows are being written correctly.

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.
Please sign in to rate this answer.
Noah Gilbert 26 Reputation points

2024-09-27T19:36:24.07+00:00

Hi Pinaki,

I am running the data flow via a pipeline. Once it completes (and shows success) I go to the Data Flow details under the Monitor tab. This is where it displays: Rows written: 0.

Is there a known issue with using Dataverse as a sink? Can you confirm that writing to the Dataverse as a sink in a data flow mapping correctly shows rows written?

Thanks
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

ADF Data Flow Mapping shows 0 rows written to Dataverse sink

1 answer

Your answer