Azure Databricks

What is the difference between Databrick prepay and Databrick reservation in Azure ?

Hello, We are just considering ways to reduce Databrick cost in Azure other than buying RI for VMs behind Databrick clusters. What is the difference between Databrick prepay and Databrick reservation in Azure It seems Databrick reservation is named as…

Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you. (cluster-id 0524-002352-kk357210-v2n)

Hello good people, I am getting this error "Cluster Start-up Delayed. Please wait while we continue to try and start the cluster. No action is required from you. (cluster-id 0524-002352-kk357210-v2n)" Please help. Thank You so much.

I don't see the Data tab in my 14-day trial for Azure databricks.

How to use a different version of a Spark Java library dependency (antlr4) in a Databricks notebook?

Hello. I need to use in a Databricks notebook a custom made Java library which depends on Drools v8.40.1.Final which depends on ANTLR4 v4.10.1. When I try to invoke a method in my Java library I get the following error: "ANTLR Tool version 4.10.1…

How to parse nested json array of document in ADF data flow

Hi all I am trying to fitch the values from a nested josh array of document , I have used aggregate to convert into objects but not able to fitch the values of all child nodes like as below itOffer.item itOffer.item.SplOfr itOffer.item.buy …

Azure Databricks workflow job failure

We have a stream workflow job that run 24*7 and loads the data in delta table for say: raw.deltaTableA Now, the problem is in case we are trying to optimize this delta (optimize raw.deltaTableA) table while the table is getting loaded we get frequent…

CSV to XML conversion in databricks which have some blank values as well in csv

I am converting CSV data to xml and that CSV data has some blank values as well for a few columns let's take an example there are 4 columns in CSV and out of that for a row(record) 1 colom value is blank , so as an output in xml, I am getting a missing…

ADF | ADB Activity Execution Time on Job Clusters

Has anyone noticed adb notebooks running (on job clusters) faster in ADF ? we have sequential notebook activities and seeing the start up time of clusters to be as low as 2 minutes.

how to disable autoscaling local storage

I'm configuring the cluster with the 'enable_elastic_disk' parameter as 'false', using tfvars. ex: enable_elastic_disk = false. However, clustering in Databricks remains true. what to do?

Connecting Azure Databricks workspace to on-premises network - peering

I was following this tutorial to deploy a workspace for on prem database access. I created the VNET for Databricks as mentioned as well as the transit VNET. However, when I got to the option to peer the two VNETs the VNET peering option seems to be…

[Databricks] Clusters are failing to launch. Cluster launch will be retried.

Hi all, I am a complete newbie on Databricks Azure. I have encounterd the below issue which I think is stopping me from running query. Any help will be much appreciated. Thanks. Billy Clusters are failing to launch. Cluster launch will be…

How to access the Databricks manages resource group to rotate the access keys for a storage account under the managed resource group?

We want to rotate the access keys for a storage account under a Databricks managed RG. However, keep getting the below error message: "the access is denied because of the deny assignment with name System deny assignment created by Azure…

Azure Databricks fail to install Geospark libraries from Maven

Hi Team , I am attempting to add below two geospark Maven libraries to my Azure Databricks interactive cluster with Runtime Version 14.3 LTS . However , I am getting below error Library installation attempted on the driver node of cluster…

Results too large error

Hi, We have a databricks table for which the underlying data is in ADLS gen2. The table has a column named "data" (stringtype) which has very large JSON values. When we try to select the rows from the table it throws an error as "Results…

An error occurred when using odbc and managed identity to link azure databricks in the. net core

Environmental description: The app service has already been configured with managed identity, and the relevant permissions for this managed identity have been configured in Azure databricks. When the app service uses code to access Azure databricks, the…

Do you know how to install the 'ODBC Driver 17 for PostgreSQL' on a Azure Databricks cluster?

I am attempting to run postgreSQL stored procedures , through Azure Databricks notebook. We have stored procedure written in Azure Database for PostgreSQL and we want to run postgreSQL stored procedures through Azure Databricks Notebook (using…

py4j.security.Py4JSecurityException

Hello I am trying to run spark XGBoostRegression model on Databricks cluster with Databricks runtime: 14.3 LTS. I am getting the following error: Py4JError: An error occurred while calling o547.resourceProfileManager. Trace:…

How do I use the Script activity in ADF, so it uses Azure Databricks SQL Warehouse

I want to be able to use ADF Script activity to execute SQL statements on the Azure Databricks SQL warehouses (including the serverless kind). https://video2.skills-academy.com/en-us/azure/data-factory/transform-data-using-script Azure Databricks SQL…

wrong answer in dp 203 test exam question

Hello, I have test exam questions for dp 203 and i had the following question after the answer, it shows that it is wrong and gives a link to the documentation …

How to configure ADF pipeline run, linked service, so it uses Databricks serverless compute

Databricks has recently announced serverless compute for workflows: https://video2.skills-academy.com/en-us/azure/databricks/workflows/jobs/run-serverless-jobs I would like to be able to execute Azure Data Factory (ADF) jobs using this…