Synapse Analytics - Cluster does not use requirements.txt file

Rory Tarnow-Mordi 1

I've been trying to include some additional Python packages in a Spark cluster, but when I put my requirements.txt file up (using any of the three methods here) and restart the cluster, the packages are not available (either when I try to import <package> or when I list them using:)

import pip #needed to use the pip functions
for i in pip.get_installed_distributions(local_only=True):
print(i)

The requirements.txt is attached.

20836-requirements.txt

Rory Tarnow-Mordi 1 Reputation point

2020-08-27T12:18:52.813+00:00

The only other Q&A related to this is here, maybe you can help here as well please, @Euan Garden ?
HimanshuSinha-msft 19,376 Reputation points Microsoft Employee

2020-08-27T23:03:24.517+00:00

Hello ,

Thanks for posting the question and welcome to Microsoft Q & A .

I think I was able to repro this , let me reach out to the in internal team and get help on this . We will update on this once we hear from them .

Thanks
Himanshu

1 answer

HimanshuSinha-msft 19,376 Reputation points Microsoft Employee

2020-08-31T21:24:34.163+00:00
Hello @Rory Tarnow-Mordi ,

Can you please try the below steps and let me knmow if that works .
.

Be sure auto pause is ON (whatever the time value)

Re-scale your pool (let’s say from 3nodes_to_6nodes) to a new value (3nodes_to_8nodes).

Check the “Force new settings” and apply.

This force your notebook to acquire a new nodes instance from the pool, where your modules will be loaded

Thanks
Himanshu
Please sign in to rate this answer.
HimanshuSinha-msft 19,376 Reputation points Microsoft Employee

2020-09-02T23:49:59.147+00:00

Hello ,

Since we have not heard back from you , so was checking back if the above resolution helped . In case if you have a different resolution , we request to share that with the community , so that others reading the thread can benefit from that .

Thanks
Himanshu

HimanshuSinha-msft 19,376 Reputation points Microsoft Employee

2020-09-08T18:00:11.757+00:00

Hello ,

Since we have not heard back yet , so was checking back if the above resolution helped . In case if you have a different resolution , we request to share that with the community , so that others reading the thread can benefit from that .

Thanks
Himanshu
Sign in to comment

Share via

Synapse Analytics - Cluster does not use requirements.txt file

1 answer