Currently only hours are supported in the default time format, producing the result you are seeing. However, setting your path pattern to {date}/{datetime:HH}/{datetime:mm}/asa-piiCleaner/{MachineId}
should create the results you are expecting.
Stream Analytics Output to BlobStorage Folder Structure and Time
We have a stream analytics job that is simply processing json files and passing them into cosmos or to another event hub. On the event hub we have capture events to make avro files for later consumption. We are having an issue there and have thought to use stream analytics and blob storage output and simply make the json files ourselves. However we are running into a problem/concern and thought to bring here before a ticket is made.
Path Pattern: {date}/{time}/asa-piiCleaner/{MachineId}
Date Format: YYYY/MM/DD
Time Format: HH/mm
Format: Array
Minimum rows: 1
Maximum time: 1 minute
Given this setup I was expecting to see in blob storage something akin to the following
2021/03/31/16/00/asa-piiCleaner/00dc7a2a-dbe0-4b08-b939-acf073cb88ba/jsonfile
2021/03/31/16/01/asa-piiCleaner/00dc7a2a-dbe0-4b08-b939-acf073cb88ba/jsonfile
2021/03/31/16/02/asa-piiCleaner/00dc7a2a-dbe0-4b08-b939-acf073cb88ba/jsonfile
2021/03/31/16/03/asa-piiCleaner/00dc7a2a-dbe0-4b08-b939-acf073cb88ba/jsonfile
I have yet to run this for a full hour however at this time not seeing what we are expecting instead
2021/03/31/16/00/asa-piiCleaner/00dc7a2a-dbe0-4b08-b939-acf073cb88ba/jsonfile
is being created and that same file is being updated for now going on 10 minutes. Everything else feels like its working as designed except that the "mm" column is not changing
I did this right at the bottom of an hour so was able to see it change from 03/31/15 to 03/31/16. However the folder "00" seems to stay constant
Also anyone know how to just take the stream and make individual files?
Thoughts?
2 answers
Sort by: Most helpful
-
Samara Soucy - MSFT 5,051 Reputation points
2021-04-01T18:12:35.907+00:00 -
JS Azure 76 Reputation points
2021-04-02T16:43:37.663+00:00 Hi, can you confirm you have outputs going at the minute level (e.g. if you have a 10 minute tumbling windows, you will have output only every 10 minutes).
Also what is your out-of-order and late arrival settings in the settings? Out-of-order add some time before the first output, and late arrival events may adjust the events to another folder. You can also see if you have late events in the metrics.
Thanks,
JS (Stream Analytics)