Accessing data saved during notebook run

Vincent Russo 20

I have a notebook instance where I want to save data. I want to be able to save to the same directory in which my notebooks are stored however using, for instance os.listdir(), I am not able to find the file path for which the notebooks are stored.

tl;dr How do I save data to the "My notebooks" directory?

PRADEEPCHEEKATLA-MSFT 84,381 Reputation points Microsoft Employee

2024-04-16T07:51:22.7+00:00

@Vincent Russo - Thanks for the question and using MS Q&A platform.

Could you please let us know the Azure Service area which you are using the notebook instance?
Vincent Russo 20 Reputation points

2024-04-16T10:58:29.3466667+00:00

Hi @PRADEEPCHEEKATLA-MSFT

Thank you for the response. Sure, the Azure Service that I am using is "Azure Quantum".

Let me know if I can provide anything else, and thank you again!
Vincent Russo 20 Reputation points

2024-04-16T10:59:15+00:00

Hi @PRADEEPCHEEKATLA-MSFT ,

Thank you for the response. Yes, the Azure Service I am using is "Azure Quantum"
PRADEEPCHEEKATLA-MSFT 84,381 Reputation points Microsoft Employee

2024-04-17T03:01:21.8666667+00:00

@Vincent Russo - Thanks for the detailed. Added "Azure Quantum" tag to your question, so that would make sure that your issue has better visibility in the community.
Vincent Russo 20 Reputation points

2024-04-17T11:06:30.8466667+00:00

Thanks @PRADEEPCHEEKATLA-MSFT .

Is there any level of consistency in file structure amongst the Azure project? My thought is that it shouldn't matter what Azure Service is being used in this case as (I imagine, but maybe am wrong) that the way in which to access the filesystem would be the same on notebook server instances. Is there a general way in which to do this that works across services that you can suggest?
kobulloc-MSFT 25,731 Reputation points Microsoft Employee

2024-04-24T19:00:33.7233333+00:00

Hello, @Vincent Russo ! I reached out to the Azure Quantum team and because the Jupyter Notebook instance will live up to a maximum of 12 hours, an alternative that was suggested was to use a client such as VS Code and have your Jupyter Notebook saved locally.
Vincent Russo 20 Reputation points

2024-04-24T23:18:30.7866667+00:00

Hi @kobulloc-MSFT

Thanks for the follow-up and comment. Got it. Are there docs anywhere for how to connect to the Azure Quantum Workspace via a local notebook? I wasn't able to find any documentation around operating with Azure (but not within their notebook ecosystem).

Apologies for the (probably obvious) question, but are there docs for how to set this up? Thank you again!
kobulloc-MSFT 25,731 Reputation points Microsoft Employee

2024-04-25T22:05:13.6866667+00:00
No worries, @Vincent Russo !

You can upload Jupyter Notebooks to an Azure Quantum workspace and download notebooks as well. I would start off with a downloaded notebook from your Quantum Workspace and then work forward from there.

If this approach doesn't work, let me know so I can present your use case to the Azure Quantum team.

Upload notebooks

Select your Azure Quantum workspace in the Azure portal.

Select Notebooks.

Select My notebooks.

Select Upload new.

Drag and drop your notebook files, or select Browse for Files and select the notebook files you want to upload.

If you want to overwrite already existing files, select Overwrite if already exists.

Select Upload files.

Download (or manage) notebooks

Select your Azure Quantum workspace in the Azure portal.

Select Notebooks.

Select your notebook in My notebooks

Click on the context pane, which is ... to the right of your notebook's name.

Select Rename Notebook, Delete Notebook, Duplicate Notebook, or Download Notebook.
Vincent Russo 20 Reputation points

2024-04-27T12:12:00.25+00:00

Hi @kobulloc-MSFT

Thank you again for the response.

For the links you provided, this appears to apply to uploading and downloading notebooks. My question is not with respect to notebooks but in reference to the data that I would generate from running my notebook on the Azure Quantum Workspace.

That is, my initial question is: For data that is saved in the course of running the notebook on the Azure Quantum Workspace: How do I download and save that data. Does this make sense?
kobulloc-MSFT 25,731 Reputation points Microsoft Employee

2024-05-01T22:51:02.6833333+00:00

Hello, @Vincent Russo ! For clarification, you are talking about data generated outside of running a job?

https://video2.skills-academy.com/en-us/azure/quantum/how-to-submit-jobs?tabs=tabid-python&pivots=ide-qsharp#connect-to-azure-quantum-and-submit-your-job
Vincent Russo 20 Reputation points

2024-05-02T13:35:34.98+00:00

Hi @kobulloc-MSFT

To be clear, I am talking about data generated on the Azure Quantum Workspace instance when running the notebook.

My notebook runs -> generates data. This data is not accessible to me and I am not able to download this data. My main question has been: How can I acquire/download the data that is generated from running my notebook on Azure?

1 answer

kobulloc-MSFT 25,731 Reputation points Microsoft Employee

2024-05-02T23:13:14.61+00:00
Hello, @Vincent Russo ! I reached out to the Azure Quantum team and it looks like notebook data is in the associated storage account which you can access using the storage browser. I've included details and instructions below.

How do I access data that is generated when running a notebook?

First, you will need to make sure that you have the appropriate access to the resource:

https://video2.skills-academy.com/en-us/azure/storage/blobs/authorize-data-operations-portal#permissions-needed-to-access-blob-data

Once you have confirmed that you have access, you can navigate to the storage account:

Go to the workspace overview

Select Storage account: <ID>

In the left page, select Storage browser

Select the File shares card and select azure-quantum-notebooks

Drill down through the workspace_ID and each notebook is listed in its own folder.

Start with the storage account associated with your quantum workspace:

Navigate to the storage browser, and find your notebook under the file shares:

I hope this has been helpful! Your feedback is important so please take a moment to accept answers.

If you still have questions, please let us know what is needed in the comments so the question can be answered. Thank you for helping to improve Microsoft Q&A!
Please sign in to rate this answer.
Vincent Russo 20 Reputation points

2024-05-02T23:29:00.96+00:00

Hi @kobulloc-MSFT

I don't see any "Storage Browser" icon in my Quantum Workspace

Vincent Russo 20 Reputation points

2024-05-02T23:29:51.2333333+00:00

(Attaching image here in case the prior comment did not work)

Moderator edit: Removed PII

kobulloc-MSFT 25,731 Reputation points Microsoft Employee

2024-05-02T23:38:11.8866667+00:00

That is interesting. I was under the impression that a storage account was required to set up a workspace or use a notebook. Let me look through this again.

Vincent Russo 20 Reputation points

2024-05-03T13:28:09.1266667+00:00

Yeah, rather odd. Thank you for looking into it, and do let me know if you come across anything. Thank you, @kobulloc-MSFT !

kobulloc-MSFT 25,731 Reputation points Microsoft Employee

2024-05-03T19:31:36.17+00:00

Hello, @Vincent Russo !

It looks like the current process for creating a quantum workspace does require an associated storage account (for both quick create and advanced create). If possible, I would recommend recreating the quantum workspace to make sure that a storage account is associated.

Azure Quantum Workspace with quick create:

Azure Quantum Workspace with advanced create:

kobulloc-MSFT 25,731 Reputation points Microsoft Employee

2024-05-06T18:11:49.9566667+00:00

Hello, @Vincent Russo !

Just checking in to see if you were able to create a new workspace, and to see if that correctly associated a storage account for you.

Vincent Russo 20 Reputation points

2024-05-08T20:10:52.63+00:00

Hi @kobulloc-MSFT ,

I already have an associated storage space for my notebook (but for some reason, it's not accessible via the menu you showed in your screenshots).

In any case, I am able to navigate to the associated storage account for this, but data that is saved does not actually save in the file storage. Where it saves is not clear, but despite saving to the local directory where the notebook is stored, I do not see any resulting files in that directory.

kobulloc-MSFT 25,731 Reputation points Microsoft Employee

2024-05-17T16:52:41.39+00:00

Hello, @Vincent Russo ! The Quantum team has confirmed that it should be accessible in the menu shown in the screenshot which is why I suspect that there may be a legacy setting or something that didn't execute correctly during the initial setup.

I still think that creating a new workspace would be worth trying, but we can have someone take a look at your resource if you would prefer to go that route instead.

Vincent Russo 20 Reputation points

2024-05-17T18:31:44.0166667+00:00

Hi @kobulloc-MSFT

Thanks for your response. I think, at present, the usage hurdles might be a bit too high for us to consider Azure at this time. I'm happy to speak directly with a representative on the Quantum side, but I think, at present, the usage problems are quite large blockers. Thank you again for the help!

kobulloc-MSFT 25,731 Reputation points Microsoft Employee

2024-05-22T18:46:25.6966667+00:00

Sorry I couldn't be of more help, @Vincent Russo !

If you are interested in speaking with the Azure quantum team, you can contact AzureQuantumInfo@microsoft.com.

kobulloc-MSFT 25,731 Reputation points Microsoft Employee

2024-05-23T23:35:40.4233333+00:00

Hello, @Vincent Russo !

Before we close this out I want to make sure there's nothing else we can do to help. Hopefully you've reached out to the email alias with any additional usage issues.

Your feedback is important so please take a moment to accept answers.

If you still have questions, please let us know what is needed in the comments so the question can be answered. Thank you for helping to improve Microsoft Q&A!

Vincent Russo 20 Reputation points

2024-05-24T01:53:32.5333333+00:00

Hi @kobulloc-MSFT

Thanks, I appreciate the help. Unfortunately, I think this can be closed out as unresolved. Sadly, the takeaway on my end is that the Azure offering is not a viable approach for my purposes and has far too many usability hurdles for it to be a competitive offering. Thank you again for your help however, I sincerely appreciate it.

kobulloc-MSFT 25,731 Reputation points Microsoft Employee

2024-05-24T04:31:02.3633333+00:00

No worries, @Vincent Russo ! Please do let us know if there is anything we can do for you in the future.
Sign in to comment

Share via

Accessing data saved during notebook run

1 answer