azure data factory copy activity Uri string is too long

Han Shih 施學翰 146 Reputation points
2022-02-17T03:18:12.76+00:00

I try to load a JSON file from Azure storage into Azure data explorer via Azure data factory.
The file is in 'one-line', 'nested' JSON format. (about 5~10 MB)

{"value":[{"col1":1, "col2":2},{"col1":11, "col2":22} ... ]}

Since I am not familiar with the JSON mapping syntax in Azure data explorer, I just simply load the file in to a table.
Which leads to only 1 column exists in that table (with default mapping rule).

Here is my COPY activity setup:
175149-image.png

175183-image.png

However, I get this error: 【Type=System.UriFormatException,Message=Invalid URI: The Uri string is too long.,Source=System,'】
175222-image.png

[Update]

After some modification, I received new error message (I add 'database user' & 'database ingestor' for my Azure data factory instance)

input

{
"source": {
"type": "JsonSource",
"storeSettings": {
"type": "AzureBlobStorageReadSettings",
"recursive": false,
"wildcardFileName": "2022-02-17.json",
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "JsonReadSettings"
}
},
"sink": {
"type": "AzureDataExplorerSink",
"ingestionMappingName": ""
},
"enableStaging": false
}

ouput

{
"dataRead": 3416480,
"dataWritten": 0,
"filesRead": 1,
"sourcePeakConnections": 1,
"sinkPeakConnections": 1,
"rowsRead": 1,
"rowsCopied": 1,
"copyDuration": 7,
"throughput": 476.629,
"errors": [
{
"Code": 11204,
"Message": "ErrorCode=UserErrorKustoWriteFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Write to Kusto failed with following error: An error occurred for source: 'DataReader'. Error: 'CsvWriter: Cannot skip to the next line as the current line is still empty'.,Source=Microsoft.DataTransfer.Runtime.KustoConnector,''Type=Kusto.Ingest.Exceptions.IngestClientException,Message=An error occurred for source: 'DataReader'. Error: 'CsvWriter: Cannot skip to the next line as the current line is still empty',Source=Kusto.Ingest,''Type=Kusto.Cloud.Platform.Utils.UtilsInvalidOperationException,Message=CsvWriter: Cannot skip to the next line as the current line is still empty,Source=Kusto.Cloud.Platform,'",
"EventType": 0,
"Category": 5,
"Data": {},
"MsgId": null,
"ExceptionType": null,
"Source": null,
"StackTrace": null,
"InnerEventInfos": []
}
],
"effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (Japan East)",
"usedDataIntegrationUnits": 4,
"billingReference": {
"activityType": "DataMovement",
"billableDuration": [
{
"meterType": "AzureIR",
"duration": 0.06666666666666667,
"unit": "DIUHours"
}
]
},
"usedParallelCopies": 1,
"executionDetails": [
{
"source": {
"type": "AzureBlobStorage",
"region": "Japan East"
},
"sink": {
"type": "AzureDataExplorer"
},
"status": "Failed",
"start": "2022-02-17T04:02:13.0176604Z",
"duration": 7,
"usedDataIntegrationUnits": 4,
"usedParallelCopies": 1,
"profile": {
"queue": {
"status": "Completed",
"duration": 4
},
"transfer": {
"status": "Completed",
"duration": 3,
"details": {
"listingSource": {
"type": "AzureBlobStorage",
"workingDuration": 0
},
"readingFromSource": {
"type": "AzureBlobStorage",
"workingDuration": 0
},
"writingToSink": {
"type": "AzureDataExplorer"
}
}
}
},
"detailedDurations": {
"queuingDuration": 4,
"transferDuration": 3
}
}
],
"dataConsistencyVerification": {
"VerificationResult": "NotVerified"
},
"durationInQueue": {
"integrationRuntimeQueue": 0
}
}

error message

Failure type:
User configuration issue
Details:
ErrorCode=UserErrorKustoWriteFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Write to Kusto failed with following error: An error occurred for source: 'DataReader'. Error: 'CsvWriter: Cannot skip to the next line as the current line is still empty'.,Source=Microsoft.DataTransfer.Runtime.KustoConnector,''Type=Kusto.Ingest.Exceptions.IngestClientException,Message=An error occurred for source: 'DataReader'. Error: 'CsvWriter: Cannot skip to the next line as the current line is still empty',Source=Kusto.Ingest,''Type=Kusto.Cloud.Platform.Utils.UtilsInvalidOperationException,Message=CsvWriter: Cannot skip to the next line as the current line is still empty,Source=Kusto.Cloud.Platform,'

After some searching, I am still not able to find a solution.
Any suggestion?

Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
525 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,682 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Han Shih 施學翰 146 Reputation points
    2022-02-24T07:13:00.317+00:00

    Any suggestions?

    Can not copy csv file into azure data explorer table due to "," exists in column values

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.