Data ingestion is not working in Azure Data Explorer after I created a new data connection

Kim Strasser 1,036

I have created a new data connection in my Azure Data Explorer Cluster. I have chosen the connection type Blob storage and the event type Blob created. In addition, I have chosen Resources creation Automatic. I didn't get an error message when I created the data connection. But I get no results when I query my TestDatabase. I used this code to query:
TestTable
| count

The problem is that this code returns "Count": 0, but I have several files in my Blob storage container. I don't know why querying returns "Count": 0. Something is not working with my new data connection.

Why is my data connection not working?

My Blob storage container:

PRADEEPCHEEKATLA-MSFT 89,296 Reputation points Microsoft Employee

2021-09-02T08:22:15.65+00:00

Hello @KimStrasser-4518,

Welcome to the Microsoft Q&A platform.

After creating the data connection have you generating any sample date by creating a blob container because the event will be generated on Blob Created?

Meanwhile, you can checkout this Ingest blobs into Azure Data Explorer by subscribing to Event Grid notifications.
Kim Strasser 1,036 Reputation points

2021-09-02T09:01:51.873+00:00

Yes, I have generated the sample data folder named "31" after the data connection was created:

The files in my folders end with ".json.gz". I think that I have chosen the wrong settings when I created my data connection.
I have deleted my current data connection. I want to create a new data connection, but I don't know what Basics and Ingest properties that I should use if I want to query all the files in Data Explorer that are located in: mycontainer / PlayFabTitleAllEvents

Should I choose Blob created or Blob renamed in Basics for event type?
Are my filter settings correct?

Are my Ingest properties correct? I don't know if I should leave the field Data format empty.
PRADEEPCHEEKATLA-MSFT 89,296 Reputation points Microsoft Employee

2021-09-03T07:35:23.657+00:00

Hello @KimStrasser-4518,

Thanks for brining this to our attention. As per the repro, we are also observed the same behaviour.

We are reaching out to the internal team to get more details on this behaviour. I will be update you once I hear back from the team.

Accepted answer

PRADEEPCHEEKATLA-MSFT 89,296 Reputation points Microsoft Employee

2021-09-03T10:55:23.14+00:00

Hello @KimStrasser-4518,

Thanks for the question and using MS Q&A platform.

I'm able to successfully follow the tutorial: Ingest blobs into Azure Data Explorer by subscribing to Event Grid notifications.

In order to succefully ingest the blobs into ADX you need to modify the Data format to MULTILINE JSON as shown below:

Now succefully ingest the blobs into ADX.

In case, if you are still not receive the data with code returns "Count": 0, try running .show ingestion failures to get the details about the failures.

Note: This command returns a result set that includes any ingestion failures that occur when the data ingestion control commands run.

Hope this helps. Do let us know if you any further queries.

---------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Please sign in to rate this answer.
Kim Strasser 1,036 Reputation points

2021-09-04T09:21:14.88+00:00

It's not working. It still returns "Count": 0. In addition, .show ingestion failures returns no failures:

I don't know what I'm doing wrong.
Is it necessary to enter a suffix in Data connection Basics-->Filter settings? I have left this field empty.

My data connection settings and my storage container Blob files:

PRADEEPCHEEKATLA-MSFT 89,296 Reputation points Microsoft Employee

2021-09-06T07:12:01.213+00:00

Hello @KimStrasser-4518,

When are you uploading the data in to the blob storage account?

After configuring the data connection wait for 5 minutes and then upload the data to blob storage account?

After 10 minutes you can review the data in the ADX. In case you didn't get the data you should get the ingestion failures as shown above?

I would suggest you to follow the steps mentioned in the document: https://video2.skills-academy.com/en-us/azure/data-explorer/ingest-data-event-grid#create-an-event-grid-data-connection-in-azure-data-explorer

Kim Strasser 1,036 Reputation points

2021-09-07T09:52:37.107+00:00

Thanx. It works now. I think that I have not waited long enough until I uploaded the files to my storage account.

PRADEEPCHEEKATLA-MSFT 89,296 Reputation points Microsoft Employee

2021-09-08T10:06:42.897+00:00

Hello @KimStrasser-4518,

Glad to know that your issue has resolved.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.