How can I monitor and identify ADX ingestion fails at file level by using Queued file ingestion type with SDK library ?

Galgani, Stefano 205 Reputation points
2023-11-06T14:40:48.3566667+00:00

Hi community,

I have implemented ADX Queued ingestion ( QueueIngestionLink ) to ingest timeseries data passed by csv files. The java application is implemented by using SDK library as suggested from Microsoft.
I have left default ingestion policy as I showed below:
defaultIngestionPolicy

To understand better the Queued ingestion file behavior, I have tried to ingest a file per time to see the result on my Telemetry table. Being only a file for testing the policy batch was triggered by Time.

How I could see from the Ingestion Time field of the Telemetry table I have seen 4 batches for 4 different files. As I'm supposing in other situations we could have more files in a single batch.
ingestionTimebatches

I understood that this approach is asynchronous attributable to Producer and Consumer pattern, but in case of the batch ingestion failed, is there a way identify the file or files have caused the failure ?

In nutshell is there the possibility to know failure at file level ?

Otherwise do you have the suggestions ?

Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
501 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sander van de Velde | MVP 30,711 Reputation points MVP
    2023-11-06T15:34:13.5433333+00:00

    Hello @Stefano,

    the queued ingestion is following the Batching rules by default.

    The ingestion time is available via this Kusto statement:

    T | extend ingestionTime = ingestion_time() | top 10 by ingestionTime

    You can check ingestion using:

    .show commands | top 100 by StartedOn

    You can check the ingestion failures with

    .show ingestion failures

    There are multiple ways to check the behavior of ingestion. See also this blog post regarding the 'Drop by' tags and 'Ingest By' tags on extents, the ReportLevel and Report Method ingestion properties, and altering the ingestion time.

    The batching strategy can be overridden completely when the streaming ingestion policy is overruled.

    I recommend investing some time to play with these settings. This video gives you some pointers too.

    This answer is written by myself, not by some AI. If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

    2 people found this answer helpful.