Every subscription in Azure as a core quota that you set. To run Databricks with a 4 core quota, shut down any other clusters or VMs in your subscription and start a smaller cluster. Databricks has Single Node Clusters which goes down to 4 cores for the whole cluster.
Quota limit hit on tutorial notebook
I'm attempting to launch a default cluster (min 2, max 8) on the premium trial account in order to run 01-The-Databricks-Environment. I haven't been able to run any operations in the notebook. Receiving this error:
Azure error code: OperationNotAllowed
Azure error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westus, Current Limit: 4, Current Usage: 4, Additional Required: 4, (Minimum) New Limit Required: 8. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/?#create/Microsoft.Support/Parameters/%7B%22subId%22:%2293a981b0-792f-4b4e-a538-662b956d1836%22,%22pesId%22:%2206bfd9d3-516b-d5c6-5802-169c800dec89%22,%22supportTopicId%22:%22e12e3d1d-7fa0-af33-c6d0-3c50df9658a3%22%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://video2.skills-academy.com/en-us/azure/azure-supportability/regional-quota-requests. (Azure request Id: ae4f41ce-cfec-4af8-8927-74ce0fbdb413)
Thank you for your time and any assistance!
2 answers
Sort by: Most helpful
-
-
PRADEEPCHEEKATLA-MSFT 88,381 Reputation points Microsoft Employee
2020-12-08T05:25:05.263+00:00 Hello @Zak Wear ,
Welcome to the Microsoft Q&A platform.
Currently, you can use Azure Free Trial subscription to create a Single node cluster that will have one Driver node with 4 cores.
A Single Node cluster is a cluster consisting of a Spark driver and no Spark workers. Such clusters support Spark jobs and all Spark data sources, including Delta Lake. In contrast, Standard clusters require at least one Spark worker to run Spark jobs.
Single Node clusters are helpful in the following situations:
- Running single node machine learning workloads that need Spark to load and save data
- Lightweight exploratory data analysis (EDA)
Reference: Azure Databricks - Single Node clusters
OR
To create a cluster with more than 4 cores, you need to have “Pay-As-You-Go” subscription.
Why can't I use Azure Free Trial subscription?
Azure Free Trial has a limit of 4 cores, and you cannot create Azure Databricks cluster using a Free Trial Subscription because to create a spark cluster which requires more than 4 cores i.e (Minimum 4 cores for Driver type and 4 cores for Worker type).
If you have a free account, go to your profile and change your subscription to pay-as-you-go. Then, remove the spending limit, and request a quota increase for vCPUs in your region. When you create your Azure Databricks workspace, you can select the Trial (Premium - 14-Days Free DBUs) pricing tier to give the workspace access to free Premium Azure Databricks DBUs for 14 days.
Hope this helps. Do let us know if you any further queries.
------------
- Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
- Want a reminder to come back and check responses? Here is how to subscribe to a notification.