Quota limit hit on tutorial notebook

Zak Wear 1 Reputation point
2020-12-07T20:09:56.14+00:00

I'm attempting to launch a default cluster (min 2, max 8) on the premium trial account in order to run 01-The-Databricks-Environment. I haven't been able to run any operations in the notebook. Receiving this error:

Azure error code: OperationNotAllowed

Azure error message: Operation could not be completed as it results in exceeding approved Total Regional Cores quota. Additional details - Deployment Model: Resource Manager, Location: westus, Current Limit: 4, Current Usage: 4, Additional Required: 4, (Minimum) New Limit Required: 8. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/?#create/Microsoft.Support/Parameters/%7B%22subId%22:%2293a981b0-792f-4b4e-a538-662b956d1836%22,%22pesId%22:%2206bfd9d3-516b-d5c6-5802-169c800dec89%22,%22supportTopicId%22:%22e12e3d1d-7fa0-af33-c6d0-3c50df9658a3%22%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://video2.skills-academy.com/en-us/azure/azure-supportability/regional-quota-requests. (Azure request Id: ae4f41ce-cfec-4af8-8927-74ce0fbdb413)

Thank you for your time and any assistance!

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,148 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. David Browne - msft 3,846 Reputation points
    2020-12-07T21:53:14.793+00:00

    Every subscription in Azure as a core quota that you set. To run Databricks with a 4 core quota, shut down any other clusters or VMs in your subscription and start a smaller cluster. Databricks has Single Node Clusters which goes down to 4 cores for the whole cluster.

    0 comments
  2. PRADEEPCHEEKATLA-MSFT 88,381 Reputation points Microsoft Employee
    2020-12-08T05:25:05.263+00:00

    Hello @Zak Wear ,

    Welcome to the Microsoft Q&A platform.

    Currently, you can use Azure Free Trial subscription to create a Single node cluster that will have one Driver node with 4 cores.

    A Single Node cluster is a cluster consisting of a Spark driver and no Spark workers. Such clusters support Spark jobs and all Spark data sources, including Delta Lake. In contrast, Standard clusters require at least one Spark worker to run Spark jobs.

    Single Node clusters are helpful in the following situations:

    • Running single node machine learning workloads that need Spark to load and save data
    • Lightweight exploratory data analysis (EDA)

    29653-image.png

    Reference: Azure Databricks - Single Node clusters

    OR

    To create a cluster with more than 4 cores, you need to have “Pay-As-You-Go” subscription.

    Why can't I use Azure Free Trial subscription?

    Azure Free Trial has a limit of 4 cores, and you cannot create Azure Databricks cluster using a Free Trial Subscription because to create a spark cluster which requires more than 4 cores i.e (Minimum 4 cores for Driver type and 4 cores for Worker type).

    alt text

    If you have a free account, go to your profile and change your subscription to pay-as-you-go. Then, remove the spending limit, and request a quota increase for vCPUs in your region. When you create your Azure Databricks workspace, you can select the Trial (Premium - 14-Days Free DBUs) pricing tier to give the workspace access to free Premium Azure Databricks DBUs for 14 days.

    Hope this helps. Do let us know if you any further queries.

    ------------

    • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.