Unable to create a cluster in Databricks getting quota exceeded error while everything is new
Azure Quota Exceeded Exception: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardDADSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location:…
Azure to AWS
Hello We need to transfer files from ADLS to AWS (S3 bucket) for a SAS application hosted in third party in batches. We need to ensure data security and best practices. My understanding, we can use ADF to create a linked service for AWS S3 but IT DOES…
![](https://techprofile.blob.core.windows.net/images/Z1PCM1zxm0SLa41PVP7B7g.png?8DA865)
Understand costs AZ Databricks job
I have an Azure Data Factory pipeline that executes multiple Databricks Notebooks using job clusters. I need to track the cost of these job clusters, including both the Databricks and the underlying VM costs, specifically for this set of jobs. Currently,…
Databricks integration to Azure Synapse Serveless SQL Pool
Hi, What is the way to connect to Databricks to Azure Synapse serverless SQL pool when I am trying to connect using serverless SQL endpoint getting an error as - Py4JJavaError: An error occurred while calling o388.load. :…
how to downgrade pricing tier from Premium (+ Role-based access controls) (Click to change) to Starndard
I am trying to downgrade pricing tier from Premium (+ Role-based access controls) (Click to change) to Starndard : But I am getting this error message : Downgrade Databricks Workspace databrics-name Failed. Error: InvalidSkuForDefaultCatalog, Default…
Questions on Azure Databricks prepurchase plan
Hello, Our customer had purchased Azure Databricks prepurchase plans but their utilization is not even 50% over 3 months and their expiry date is 15 months from today. Could you please clarify on following questions? Why Azure Databricks prepurchase…
How to run stored procedure in Azure Database for PostgreSQL using Azure Databricks Notebook
We have Stored Procedure available in Azure Database for PostgreSQL and we want to call or run or execute the postgreSQL stored procedures in Azure Databricks through Notebook We are attempting to run PostgreSQL stored procedures, through Azure…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Delete the file from SharePoint location
Hi All, I am trying to copy the files from Share Point to ADLS and referring to the below URL pipeline to achieve the copy functionality. https://www.syntera.ch/blog/2022/10/10/copy-files-from-sharepoint-to-blob-storage-using-azure-data-factory/ I need…
Why isn't code working as an expression for parameters in SQL Server Reporting with connection to Databricks SQL Warehouse using Simba Spark
Error: For more information about this error navigate to the report server on the local server machine, or enable remote errors ---------------------------- Query execution failed for dataset 'DataSet1'. (rsErrorExecutingCommand)…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Unable to downgrade Databricks workspace
When downgrading the Databricks Workspace, I receive the following message; However, none of the Enhanced Security options are currently enabled; Could you please help me identify the cause of the error?
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
how to resolve this error while getting data from databricks from power bi.Not able to load table in power bi
DataSource.Error: ODBC: ERROR [HY000] [Microsoft][DSI] (20039) Cannot store ""."".""."REMARKS" value in temporary table without truncation. (Column metadata implied a maximum of 512 bytes, while provided value is…
Is there a way to restrict multiple instances of an ADF pipeline on Same path of event based trigger?
I'm running Spark notebooks in Synapse using an event-based ADF pipeline. If any notebook gets triggered twice, I encounter a conflict error. I understand that I can avoid concurrent execution of the pipeline by setting the concurrency to 1. However, I…
Firewall Configuration for Custom Model Serving in Azure Databricks
Hi, I am encountering an error when trying to serve my custom LLM model endpoint. The error message reads: "Container image creation failed, see Build Logs for details. If there are no build logs, the failure may be due to storage firewall…
How do I figure out what public IP ranges my Databricks workspace clusters are coming from?
Edit: I am rewriting this to clarify the ask. Relatively new to Databricks. I am trying to understand how outbound traffic from clusters is determined. It seems to differ if SCC is enabled vs when it's not. With no SCC: VMs start up with a…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Invalid records failed in DQ checks
We are capturing the records that failed in DQ checks by using Databricks in the Blob storage for business owners to resolve inconsistencies, we have added an extra column as DQ checks failed reason. I have the following: What if the particular record…
Databricks SQL endpoint
Hi friends, where does the databricks SQL endpoint stand with comparison to other data warehousing technologies such as Synapse, snowflake, and google cloud? please provide metrics related comparison in terms of costs,scalability and performance. Which…
What is the difference between Databrick prepay and Databrick reservation in Azure ?
Hello, We are just considering ways to reduce Databrick cost in Azure other than buying RI for VMs behind Databrick clusters. What is the difference between Databrick prepay and Databrick reservation in Azure It seems Databrick reservation is named as…
Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you. (cluster-id 0524-002352-kk357210-v2n)
Hello good people, I am getting this error "Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you. (cluster-id 0524-002352-kk357210-v2n)" Please help. Thank You so much.
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
How to use a different version of a Spark Java library dependency (antlr4) in a Databricks notebook?
Hello. I need to use in a Databricks notebook a custom made Java library which depends on Drools v8.40.1.Final which depends on ANTLR4 v4.10.1. When I try to invoke a method in my Java library I get the following error: "ANTLR Tool version 4.10.1…