Stream Analytics does not refresh the input reference blob data when it is updated on the blob storage account

zarcap 0

Hi Colleagues; I'm dealing with a problem with Stream Analytics jobs and their corresponding input reference blob data.

Basically Stream Analytics is not refreshing the input reference snapshot when a new blob file has been created within the storage account. The blob files pattern is as follows: "reference/xxx-xxxx/{date}/{time}/xxxx-xxxxxx.json" so we have a new json file comming when the reference data has to be refreshed. ( with a greater date-time than the previous one )

Sometimes I find in the log that Stream Analytics correctly re-create the snapshop:

"Reference Data full snapshot "reference/xxx-xxxx/2024/06/11/16/03/xxxx-xxxxxx.json" covering 06/11/2024 16:03:45 == 638537186250633356 to 06/11/2024 16:04:40 == 638537186800245033 with size 643578 bytes is being loaded into memory."

BUT in many other cases there is no reference Data full snapshot refresh at all in the log, then, then, reference data for the running jobs remains outdated.

for the previous example we have a new file:

reference/xxx-xxxx/2024/06/11/16/20/xxxx-xxxxxx.json

but Stream Analytics has never detect it.

Some of you had to deal with this problem?; don't known whether I'm doing something wrong...

Many thanks in advance.

PRADEEPCHEEKATLA-MSFT 84,376 Reputation points Microsoft Employee

2024-06-13T04:56:15.38+00:00

@zarcap - Thanks for the question and using MS Q&A platform.

It seems like you are having trouble with Azure Stream Analytics not refreshing the input reference snapshot when a new blob file has been created within the storage account.

According to the documentation, Stream Analytics automatically scans for refreshed reference data blobs at a one-minute interval. A blob with the timestamp 10:30:00 might be uploaded with a small delay, for example, 10:30:30. You'll notice a small delay in the Stream Analytics job referencing this blob.

To avoid such scenarios, upload the blob earlier than the target effective time, which is 10:30:00 in this example. The Stream Analytics job now has enough time to discover and load the blob in memory and perform operations.

If you have already followed these steps and are still experiencing issues, it may be helpful to check the logs to see if there are any errors or warnings related to the reference data. If you are still unable to resolve the issue, you may want to consider opening a support ticket for further assistance.

Hope this helps. Do let us know if you any further queries.
zarcap 0 Reputation points

2024-06-13T21:14:13.82+00:00

Many thanks PRADEEPCHEEKATLA for your comment; yes , we are having trouble with Azure Stream Analytics not refreshing the input reference snapshot when a new blob file has been created within the storage account; we did follow all the recommended steps but many times blob files input reference are not correctly refreshed by Stream Analytics, we will open a support ticket...
PRADEEPCHEEKATLA-MSFT 84,376 Reputation points Microsoft Employee

2024-06-14T06:51:13.7166667+00:00

@zarcap - The support team will be able to assist you in troubleshooting the issue and finding a resolution. Once you open a support ticket please do share the SR number here:

If you have any other questions or concerns, feel free to ask.
PRADEEPCHEEKATLA-MSFT 84,376 Reputation points Microsoft Employee

2024-06-18T05:47:23.18+00:00

@zarcap - Did you get a chance to open a support ticket?

Share via

Stream Analytics does not refresh the input reference blob data when it is updated on the blob storage account