Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,042 questions
1 answer
Databricks Spark Scala: RaiseError throws type error
Hello, I am facing an issue in Databricks 14.3-LTS within a Scala notebook. When I try to raise an Exception using Spark Catalyst with the following Scala code: import org.apache.spark.sql.types.{StringType, DateType} val errorMessage =…
asked 2024-06-05T08:24:16.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 41%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
bn2302
0
Reputation points
commented 2024-06-07T04:19:55.0466667+00:00
PRADEEPCHEEKATLA-MSFT
84,051
Reputation points Microsoft Employee
2 answers
I am unable to mount containers using databricks and storage gen 2 ?
what is the issue?
asked 2024-06-05T15:57:33.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 14.000000000000002%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
smriti das
0
Reputation points
answered 2024-06-06T12:19:54.16+00:00
Luis Arias
5,901
Reputation points
1 answer
One of the answers was accepted by the question author.
Can a single instance of Microsoft Purview scan multiple Azure Databricks Unity Catalog instances that exist in different logical data domains?
I am researching a use case where a single instance of Microsoft Purview can be used to scan multiple instances of Azure Databricks Unity Catalogs hosted in multiple logical / geographical domains, including using OpenLineage to provide lineage data to…
asked 2024-06-04T21:03:01.6766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 38%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEH%3C/text%3E%3C/svg%3E)
Erich Huckschlag
20
Reputation points
accepted 2024-06-06T07:43:48.44+00:00
Erich Huckschlag
20
Reputation points
1 answer
Connecting Databricks to on prem sources
Is there a way to connect my Azure Databricks workspace to my local SQL Server database? I am trying to read data from my local SQL Server installed on my machine, but I am looking for a way to connect the two directly. I am aware we can use a SHIR with…
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,042 questions
Windows Network
Windows Network
Windows: A family of Microsoft operating systems that run across personal computers, tablets, laptops, phones, internet of things devices, self-contained mixed reality headsets, large collaboration screens, and other devices.Network: A group of devices that communicate either wirelessly or via a physical connection.
696 questions
asked 2024-05-29T11:03:25.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 28.000000000000004%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAH%3C/text%3E%3C/svg%3E)
Abdullah Humayun
40
Reputation points
commented 2024-06-05T21:35:50.7366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
28,526
Reputation points Microsoft Employee
2 answers
Enabled IP Access Control in Databricks workspace and no one can connect to the workspace
We setup Access control list with only one entry of the wrong IP address (outbound nat gateway IP ) Also Enforce IP access list on Compute Plane Requests toggle on . Now no one can access the workspace and there is no way I can toggle off. Is there a…
asked 2024-06-04T19:19:54.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 80%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIM%3C/text%3E%3C/svg%3E)
Inna Mednyk
1
Reputation point
commented 2024-06-04T22:54:45.2933333+00:00
BhargavaGunnam-MSFT
28,526
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure Databricks workflow that runs multiple jobs sequentially using the same job cluster
Newbie here as far as Azure Databricks and workflows are concerned. TL/DR version: Is there a way to configure the same job cluster for multiple jobs that are part of the same workflow? Long version: We have an ELT process that we have broken down into…
asked 2024-06-01T14:30:01.2+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 99%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
Aravind Raam
20
Reputation points
accepted 2024-06-04T19:16:12.37+00:00
Aravind Raam
20
Reputation points
1 answer
One of the answers was accepted by the question author.
ADF Data Flows Flatten nested json array values are being populated as null
Hi All I am building a data transamination with ADF data flow using a nested json array of objects , but after parse and flatten the json node itOffer.item.LeadOfer.zdeal.item[].dealNumber I am seeing that the column values are populated as null . I…
asked 2024-05-30T11:06:56.13+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 6%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVR%3C/text%3E%3C/svg%3E)
venkat rao
65
Reputation points
accepted 2024-06-04T12:37:27.4766667+00:00
venkat rao
65
Reputation points
1 answer
One of the answers was accepted by the question author.
Request to clarify Databricks workspace IP access controls change for existing workspaces on August 26th
Microsoft sent an email about a change in Databricks workspace IP access controls that will impact existing workspaces on August 26, 2024. Our organization uses vnet injected Databricks and secure cluster connectivity, and we have some doubts: Do we…
asked 2024-06-03T12:21:43.8266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 80%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
Said Mohammed
20
Reputation points
accepted 2024-06-04T11:17:29.78+00:00
Said Mohammed
20
Reputation points
1 answer
One of the answers was accepted by the question author.
Are High Concurrency clusters deprecated or renamed in UC databricks worskpace
Hello Team, Is the High concurreny clusters deprecated. Even I don't see Custom Access mode in UC enabled databricks workspace UI. I went through this article https://video2.skills-academy.com/en-us/azure/databricks/archive/compute/cluster-ui-preview but I am…
asked 2024-05-28T10:02:40.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 24%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
110
Reputation points
commented 2024-06-04T09:53:40.45+00:00
PRADEEPCHEEKATLA-MSFT
84,051
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
azure data factory's databricks runtime
Hello, i am trying to ingest data from SAP to fabric lakehouse table using dataflow and copy activity, however because my sink table has change data feed enabled, the pipeline throws the following error: Operation on target Data flow1 failed:…
asked 2024-06-03T14:24:46.9933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 56.00000000000001%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EOT%3C/text%3E%3C/svg%3E)
Osama Tarek
20
Reputation points
accepted 2024-06-04T06:26:36.36+00:00
Osama Tarek
20
Reputation points
1 answer
Can we execute a Stored procedure(Azure SQL DB) through Databricks Notebook??
I already have a stored Procedure which has Merge Statement logic, I would like to execute this SP through Databricks notebook. Is there a way to do this?
asked 2022-04-20T08:16:44.697+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 16%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAR%3C/text%3E%3C/svg%3E)
Anusha Rudra
6
Reputation points
commented 2024-06-04T05:25:20.0466667+00:00
Vedant Desai
651
Reputation points
2 answers
Seeking Expertise in Spark SQL CTE Recursive Queries in Azure Databricks
I'm currently diving deep into Spark SQL and its capabilities, and I'm facing an interesting challenge. I'm eager to learn how to write CTE recursive queries in Spark SQL, but after thorough research, it seems that Spark doesn't natively support…
asked 2024-05-30T04:27:35.4+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 57.99999999999999%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Anuj, Singh (Cognizant)
50
Reputation points
edited a comment 2024-06-03T16:54:22.8866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
12,090
Reputation points Microsoft Vendor
1 answer
insert record from azure datafactory into a delta table generated in databricks
Hi team, i'm trying to insert value into a delta table from datafactory, the table has been created in databricks and in the table properties i setted "delta.enableChangeDataFeed=true", for this reason the protocol minWriterVersion of my delta…
asked 2022-03-02T12:08:37.73+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 61%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAN%3C/text%3E%3C/svg%3E)
Antonio Nuzzo
11
Reputation points
commented 2024-06-03T14:13:46.6433333+00:00
Osama Tarek
20
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Databricks with Key vault backed secret using RBAC
Azure Key vault had launched the RBAC access model in 2021. This allows finer granular access to a particular secret or key or certificate. The previous model was access policies which doesn't allow this granular access. From my knowledge, Azure keyvault…
asked 2023-05-17T14:10:08.1633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 11%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECA%3C/text%3E%3C/svg%3E)
Chand, Anupam SBOBNG-ITA/RX
461
Reputation points
commented 2024-06-03T13:22:57.4466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 93%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENK%3C/text%3E%3C/svg%3E)
Naveen Kumar
1
Reputation point
2 answers
How to convert type of any block blob to append blob inside adls gen2? Is there any process or activity which will convert the type of my entire blob without changing the content?
I have a json file inside a data lake gen2 storage and that json file blob type is a block blob. I want to convert that blob type to append blob through any services of azure. Need to know the process in detail.
asked 2024-05-31T06:30:48.3333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 36%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECN%3C/text%3E%3C/svg%3E)
Chakraborty, Nabarun (External)
0
Reputation points
answered 2024-06-03T12:09:39.6833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 64%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENR%3C/text%3E%3C/svg%3E)
Nehruji R
4,126
Reputation points Microsoft Vendor
1 answer
The public IP address range for the Azure Databricks control plane will be updated - Is there anything else I should do?
Because I received this message regarding the update: The public IP address range for the Azure Databricks control plane will be updated on 30 May 2024—you may need to take action You're receiving this email because you use Azure Databricks. I wonder if…
asked 2024-05-29T08:23:58.0766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(41.6, 44%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZA%3C/text%3E%3C/svg%3E)
Zakrzewska, Agata
0
Reputation points
commented 2024-06-03T06:52:15.9466667+00:00
PRADEEPCHEEKATLA-MSFT
84,051
Reputation points Microsoft Employee
1 answer
How to ignore the records by applying an auditing filed column condition using ADF Data Flows
Hi All I am building a data transamination using mapping data flows ,I have a time stamp field Like TimeStampUpdated in the target table. I want to lockup historical data with incremental data transamination and ignore the records coming in the…
asked 2024-05-30T09:28:46.6066667+00:00
venkat rao
65
Reputation points
commented 2024-06-03T05:46:23.9533333+00:00
Smaran Thoomu
12,090
Reputation points Microsoft Vendor
1 answer
Unable to create a cluster in Databricks getting quota exceeded error while everything is new
Azure Quota Exceeded Exception: Error code: QuotaExceeded, error message: Operation could not be completed as it results in exceeding approved standardDADSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location:…
asked 2024-05-30T21:08:45.6433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 7.000000000000001%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Sibbinath, Akhil
0
Reputation points
commented 2024-06-03T05:43:37.29+00:00
Smaran Thoomu
12,090
Reputation points Microsoft Vendor
2 answers
Azure to AWS
Hello We need to transfer files from ADLS to AWS (S3 bucket) for a SAS application hosted in third party in batches. We need to ensure data security and best practices. My understanding, we can use ADF to create a linked service for AWS S3 but IT DOES…
asked 2024-05-20T08:39:35.8366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 35%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sourav
80
Reputation points
edited an answer 2024-06-01T10:27:52.2833333+00:00
Sumarigo-MSFT
44,891
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Understand costs AZ Databricks job
I have an Azure Data Factory pipeline that executes multiple Databricks Notebooks using job clusters. I need to track the cost of these job clusters, including both the Databricks and the underlying VM costs, specifically for this set of jobs. Currently,…
asked 2024-05-31T07:58:49.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 64%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EW%3C/text%3E%3C/svg%3E)
WeirdMan
220
Reputation points
answered 2024-05-31T08:55:07.06+00:00
Vinodh247-1375
12,506
Reputation points