How to know Which Job On Azure Data Analytics Read/Write Which File in Storage Account by Diagnostic Log

Ivy Feng 16

Hello,

I am using Azure Data Lake Analytics to perform job of reading/writing files in Storage Account(gen 2). I enable the diagnostic log for both Azure Data Lake Analytics and Storage Account.

I am trying to analyze the diagnostic logs of storage account to have a set of data of which job(Azure Data Lake Analytics job) read/write which file.

However, I have looked into both log with Azure Data Lake Analytics and Storage Account, I can not find out a way to know.

In storage account log, I can only know a file is read/written at specific time.
And in Azure Data Lake Analytics log as following, I still can not find there is a relation between jobId and file/clientRequestId

To be clear, I want a way to build a relation between jobId in Azure Data Analytics and fileName(be read/be written) or clientRequestId(since there is an existing relation between clientRequestId and fileName )
which means to know which job visit which file.

Can I get some help/idea on it?

Thank you in advance!!!!

KranthiPakala-MSFT 46,487 Reputation points Microsoft Employee

2021-07-23T00:14:07.123+00:00

Hi @Ivy Feng ,

Welcome to Microsoft Q&A forum and thanks for posting your query.

As per my analysis I'm not sure if this is possible but I have reached out to internal team to get additional insights on this requirement.
I'll keep you posted as soon as I have an update from the team.

We appreciate your patience.
Thank you
Ivy Feng 16 Reputation points

2021-07-23T08:42:21.577+00:00

Hi,

Thank you so much for your reply and I am looking forward to your reply!!!!

Kind regards,
Ivy
KranthiPakala-MSFT 46,487 Reputation points Microsoft Employee

2021-07-23T19:07:30.747+00:00

HI @Ivy Feng ,

When you say Storage Account(gen 2) do you mean Azure Data Lake Storage Gen2 ? The reason I'm asking this is because ADLA doesn't support ADLS Gen2 yet.

Thank you
Ivy Feng 16 Reputation points

2021-07-23T19:22:11.79+00:00

Hi, @KranthiPakala-MSFT

Sorry I think I have a typo here.

We are using ADLA to analyze the data stored in gen1.

And we want to build the relation between the job in ADLA and the file accessed(read/write) in gen1
by the log produced by ADLA and Storage account

Thank you in advance!

Kind regards,
Ivy
KranthiPakala-MSFT 46,487 Reputation points Microsoft Employee

2021-07-26T23:36:06.693+00:00

Hi @Ivy Feng ,

Thanks for clarifying o this ask. We have reached out to internal to get additional info about this requirement and will get back to you as soon as we have an update.

Thank you for your patience.
KranthiPakala-MSFT 46,487 Reputation points Microsoft Employee

2021-08-13T08:45:49.737+00:00

Hi @Ivy Feng ,

Apologies for the delay in response. As per my conversation with internal team there is no easy way to correlate both information as per your requirement.

I would recommend you to please submit a feature request in user voice feedback channel here : https://feedback.azure.com/forums/327234-data-lake. All the feedback shared in this forum are actively monitored and reviewed by product team.

Please let us know if you have further query
KranthiPakala-MSFT 46,487 Reputation points Microsoft Employee

2021-08-16T10:50:06.647+00:00

Hi there,

Just checking to see if you have further query regarding this requirement.

Thank you

Share via

How to know Which Job On Azure Data Analytics Read/Write Which File in Storage Account by Diagnostic Log

Your answer