Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,466 questions
Disappearing files from Blob Storage
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 40%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Scott
6
Reputation points
Hi everyone.
I have recently set up an Azure IoT Hub that is accepting telemetry data from an MQTT source.
I have also set up an Azure Stream Analytics Job which simply takes the input and writes it to a Gen 2 Storage account (Data Lake, Hierarchy enabled).
The script is very simple :
SELECT
*
INTO
[tg-stream-output]
FROM
[tg-hub-stream-input]
The output format is {date}/{time} - so everything is as basic as it can get.
The events arrived without a problem and started writing to storage. The first folder was 04, and a file was created. Then on hour 5, a new folder was created and the stream continued writing. I downloaded the file to have a look at it and everything seemed fine.
Hour 6 came, new folder, new file. Same with hour 7,when the events stopped (which was expected).
I came back after some time to download everthing and folder 4 and 7 have data, but the files in 5 and 6 were missing. I had absolutely seen the files in there, but now they are missing completely.
Any help would be greatly appreciated, as I am stumped as to what could have happened.
Cheers.
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft 19,476 Reputation points Microsoft Employee2021-01-21T23:38:50.053+00:00 Hey @Scott ,
Thanks for the ask and using the forum .I did tried to repro in the issue and I have not seen any files getting deleted . Are you able to repro this ?
Can to be possible and some other process is deleting this ?
-
Scott 6 Reputation points
2021-01-22T03:46:12.147+00:00 Thanks for getting back to me.
So I tried for an hour or so to investigate the problem after posting this last night without success. The job is still running. Eventually I gave up and went to sleep.
I've just looked at it again and several more folders have appeared as the data has started streaming back in.
I now have folders 04, 05, 06, 07 and 20, 21, 22, 23.
The files that are missing in 05 and 06 have magically returned. I have done nothing. HOWEVER, folders 21 and 22 simply won't open (using Azure Storage Explorer). It just spins. I noticed similar behaviour on the previous folders. When I look at the IoT hub, there were certainly events arriving during these times, but for whatever reason, I can't even get into those folders to see the files. Trying to open several of the newer folders just has azure explorer spin and it never opens.
I'm hoping things will catch up again, but am very confused as to this behaviour.
All help is appreciated.
-
Scott 6 Reputation points
2021-01-22T03:47:43.407+00:00 I'm now having a problem where I simply can't open the directories that are being created. I'm wondering if the files are in there and for some reason, I can't use Azure Storage Explorer to look at them? I'm not having a problem viewing anything else, but several directories just spin when I time to open them.
Any ideas on that?
-
Sander van de Velde | MVP 32,641 Reputation points MVP2021-01-23T00:13:20.263+00:00 Hello @Scott ,
I checked the Azure Storage Explorer troubleshooting page but there does not seems to be a related page.
I recommend posing an issue on the related GitHub project.
Have you tried to install the Azure Storage Explorer already on another device (laptop, pc) to check it's not your configuration?
-
Scott 6 Reputation points
2021-01-23T01:08:16.877+00:00 Thanks for that idea.
I'm very convinced it's actually something to do with Data Lake Storage. Azure Storage Explorer doesn't work on several machines, but neither does any other mechanism.
What's interesting is that as time goes on (quite a bit of time), some of the directories start working, and they seem to go in order. As in, folders that are 18 hours old will work, but folders that are 15 hours won't work UNTIL they hit about 18 hours. It seems as if it's some incredibly odd caching problem.
Not entirely sure where to go from here, but this is obvioulsy a problem that needs to be fixed.
-
Sander van de Velde | MVP 32,641 Reputation points MVP
2021-01-24T12:08:57.43+00:00 Hello @Scott ,
I have not seen this behavior before.
If you suspect the service itself, I recommend filing a Support Request.
This is available in the Azure Portal under "Help + support":
Please report back if and how the issue is resolved.
-
HimanshuSinha-msft 19,476 Reputation points Microsoft Employee
2021-01-27T18:56:23.05+00:00 Hello @Scott ,
Please let us know if you got a chance to file a support ticket . if you did , please do share the ticket number .
Thanks
Himanshu -
HimanshuSinha-msft 19,476 Reputation points Microsoft Employee
2021-02-02T17:21:26.987+00:00 Hello @Scott ,
Please let us know if you got a chance to file a support ticket . if you did , please do share the ticket number .
Thanks
Himanshu
Sign in to comment
Sign in to answer