Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,212 questions
1 answer
getting error while creating mount point in data bricks
Hi, I am getting the following error when trying to create mount point. py4j.security.Py4JSecurityException: Method public com.databricks.backend.daemon.dbutils.DBUtilsCore$Result…
asked 2024-10-24T09:28:58.9066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 77%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Lakshmi Narayana Sarma Bhamidipati
110
Reputation points
commented 2024-11-05T12:48:25.7533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECB%3C/text%3E%3C/svg%3E)
Chandra Boorla
2,605
Reputation points Microsoft Vendor
1 answer
Error occurred when using ODBC driver with Workload Identity to connect to Azure Databricks Workspace from an AKS Pod deployment
Hello everybody. As per the title, I am trying to achieve the followings: Connect to an Azure Databricks Workspace, in order to execute an SQL Query, from an Azure Function running on an AKS Pod. Pod deployment is made using CI/CD pipelines with…
asked 2024-11-04T15:55:37.3633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 93%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDI%3C/text%3E%3C/svg%3E)
Dragos Ionita
0
Reputation points
edited a comment 2024-11-05T10:21:51.4866667+00:00
Dragos Ionita
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Databricks is not sending email when Databricks Alert condition is met
Azure Databricks is not sending an email to the destination email address when Databricks Alert condition is met, Sometimes this is working and sometimes not.
asked 2024-11-01T09:50:13.5066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 84%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
Mekala, Vamshi
20
Reputation points
commented 2024-11-05T08:58:24.0766667+00:00
Mekala, Vamshi
20
Reputation points
0 answers
Autoloader in Synapse
I want to load data incrementally using Azure Synapse Analytics Notebook(dedicated spark pool) there is an option on databricks to load data incrementally which is autoloader so my question was can we use this option in Azure Synapse Analytics…
asked 2024-01-24T11:33:32.82+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 69%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Aditya Singh
160
Reputation points
edited the question 2024-11-05T03:52:58.6933333+00:00
Sumarigo-MSFT
46,441
Reputation points Microsoft Employee
1 answer
Azure Databricks Workspace: Users are being added to the admins group on their own
We have an Azure Databricks workspace with Unity Catalog enabled. We are using SCIM provisioning to sync some Entra security groups to the workspace so that we can control access to the catalog items. We have noticed that some users are being added…
asked 2024-11-01T10:04:34.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 72%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECO%3C/text%3E%3C/svg%3E)
Chris O'Brien
0
Reputation points
commented 2024-11-05T01:21:03.5733333+00:00
Chandra Boorla
2,605
Reputation points Microsoft Vendor
2 answers
I don't have access to admin console of databricks even though I am admin user
I created azure account and launched databricks, I don't have access to admin console of databricks even though I am the only admin user
asked 2024-11-01T03:49:21.9833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 73%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
Ramya
0
Reputation points
commented 2024-11-05T01:19:46.23+00:00
Chandra Boorla
2,605
Reputation points Microsoft Vendor
2 answers
Error when creating Access Connector for Azure Databricks
So, just created storage, and would like to setup the access connector, and faces the below error. Anyone seen this before ? All insights are very much appreciated
asked 2024-10-31T10:38:45.89+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 73%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAT%3C/text%3E%3C/svg%3E)
Anders Thøgersen
0
Reputation points
commented 2024-11-05T01:13:43.6766667+00:00
Chandra Boorla
2,605
Reputation points Microsoft Vendor
1 answer
I dont see Account Console in Databricks. Please advise
I have personal account and when I log into databricks I dont see account console to admin stuff
asked 2024-10-31T13:46:23.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(259.20000000000005, 18%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGE%3C/text%3E%3C/svg%3E)
Gokhan Erkus
0
Reputation points
commented 2024-11-04T15:19:15.6933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
16,725
Reputation points Microsoft Vendor
1 answer
Azure Databricks Compute Instance Quota Issue
Unable to create a compute instance in Azure Databricks due to a quota issue. The error message received is: This account may not have enough CPU cores to start a cluster. Contact your administrator to increase the limits. Additionally, it is noted…
asked 2024-10-30T17:40:56.9433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 63%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDK%3C/text%3E%3C/svg%3E)
Dmitriy Khodyrev
0
Reputation points
commented 2024-11-04T15:17:27.67+00:00
Smaran Thoomu
16,725
Reputation points Microsoft Vendor
3 answers
Azure Databricks Compute error: This account may not have enough CPU cores to satisfy this request Estimated available: 0, requested: 4
I have created Azure Free Tier Subscription with a credit $200 a day ago and deployed Azure Databricks for 14 days free as well. In Quotas I have 4CPU free: But when I'm trying to create Compte in Azure Databricks it says This account may not have…
asked 2024-10-27T20:06:04.87+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 30%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESV%3C/text%3E%3C/svg%3E)
Sergei Varaksin
0
Reputation points
answered 2024-11-04T06:15:09.8533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGG%3C/text%3E%3C/svg%3E)
Ganesh Gurram
945
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Automating conversion of python scripts to databricks notebook
I'm looking to convert a Python .py file into a format compatible with Databricks, where the file can be split into multiple cells without using the Databricks GUI. Eventually, I’d like to automate this process. What’s the minimal code needed to create a…
asked 2024-10-29T14:45:29.6666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 32%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES5%3C/text%3E%3C/svg%3E)
Stephen-5002
40
Reputation points
edited an answer 2024-11-04T04:08:48.58+00:00
Smaran Thoomu
16,725
Reputation points Microsoft Vendor
1 answer
is there a spark connector for Azure Cosmos DB MongoDB API RU?
I'm using Azure Databricks to develop a proof of concept. Is there support for a Spark Connector for Azure Cosmos DB MongoDB API? If yes, what versions are supported. Is both batch and stream operations supported?
asked 2024-10-30T16:53:46.0733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 22%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELJ%3C/text%3E%3C/svg%3E)
Lum, Joey
0
Reputation points
commented 2024-11-04T01:27:45.2166667+00:00
Chandra Boorla
2,605
Reputation points Microsoft Vendor
2 answers
Not able to create compute in Databricks on Azure Free Trialaccount
Hi Team, I am not able to create compute in Databricks on Azure Free Trial account. I tried all options for creating Databricks workspace - 14 days trial, standard account, premium account - but getting same error in all of these. Error message- This…
asked 2024-10-28T17:12:40.6533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 25%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ET%3C/text%3E%3C/svg%3E)
techie1234
5
Reputation points
commented 2024-11-01T20:28:07.6433333+00:00
Ganesh Gurram
945
Reputation points Microsoft Vendor
2 answers
How to fix the below issue which I am facing while installing the maven library with coordinates as 'com.databricks:spark-xml_2.12:0.14.0'
How to fix the below issue which I am facing while installing the maven library with coordinates as 'com.databricks:spark-xml_2.12:0.14.0' in ADF pipeline in azure databricks notebook activity run failed with error message Library installation…
asked 2024-10-27T08:34:14.07+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 43%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
Dathathraiah
0
Reputation points
commented 2024-11-01T09:17:55.4666667+00:00
Vinodh247
22,951
Reputation points MVP
1 answer
Notebook : AccessDeniedException
Hello, I am trying to save a delta/parquet file into the ADLS Gen2 using a PySpark Notebook as follows : delta_output_path =…
asked 2024-10-29T15:28:46.6966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 90%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWA%3C/text%3E%3C/svg%3E)
Wajih Arfaoui
70
Reputation points
commented 2024-11-01T07:02:24.2333333+00:00
Smaran Thoomu
16,725
Reputation points Microsoft Vendor
1 answer
copy tables from sql server on-prem to .parquet format in delta lake
Currently our copy process for 10 tables is running in pipeline in ADF, our source is ON-PREM SQL Server and target is in BLOB Storage GEN2. it copies data(tables) in parquet file format in HOT storage Account --- StorageV2 (general purpose v2) but I…
asked 2024-10-29T19:03:53.5233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 10%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZH%3C/text%3E%3C/svg%3E)
Zaeemul Haque
25
Reputation points
commented 2024-11-01T06:55:45.1833333+00:00
Chandra Boorla
2,605
Reputation points Microsoft Vendor
1 answer
Unable to access account console in daatabrics
Hello guys I created a storage account through my pay as u go subscription and lauched a premium tier workspace for databricks but i was trying to connect my workspace to metastore in unity catalog but i dont see my manage account button to access…
asked 2024-10-30T11:19:15.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 42%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Mantri Sai Ashish
0
Reputation points
answered 2024-10-31T12:13:25.8766667+00:00
PRADEEPCHEEKATLA-MSFT
90,146
Reputation points Microsoft Employee
2 answers
I am not able to execute the "df.write.mode("overwrite").format('csv').option("path", "/mnt/sepadls/bronze/Employee").saveAsTable("Employee_Ext")
I am trying to create a folder called Employee in bronze layer and register as a table in hive meta store and below is the code. It's creating a folder called Employee in bronze layer but table is not getting registered and throwing an error as …
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 46%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
2 answers
Complete Azure Databricks Cluster Recommendation
Hi, We are working on a project where we need to create Databricks cluster configurations recommendation with cluster versions, modes etc etc. Can anyone help ?
asked 2024-10-28T07:18:47.36+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 3%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Prasant Kumar Das
20
Reputation points
commented 2024-10-30T13:18:58.2966667+00:00
Smaran Thoomu
16,725
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Connection Challenges with Azure Databricks and SQL Server On VM in Serverless compute.
We successfully configured VNet peering between the SQL VM's VNet and the Databricks VNet, allowing access to the SQL Server on VMs. Testing the connection from Databricks using a job cluster yielded successful results, confirming that everything…
asked 2024-10-28T05:19:04+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 87%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVD%3C/text%3E%3C/svg%3E)
Vetrivel Dhakshinamoorthy
20
Reputation points
accepted 2024-10-30T04:39:44.5666667+00:00
Vetrivel Dhakshinamoorthy
20
Reputation points