Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,162 questions
1 answer
Databricks-restrict access to users for data
Hi I worked on creating a new cluster (ClusterA) and added a few users (UserA, B) to my workspace. I am the admin and have created a database and a few tables under ClusterA. Now I need to add UserA to see the tables but not userB I enabled…
asked 2021-05-25T21:24:27.153+00:00
Sarah C Benjamin
1
Reputation point
commented 2021-06-14T19:56:05.76+00:00
Sarah C Benjamin
1
Reputation point
1 answer
notebook git revision issue
Hi Team, I am working on Azure Databricks, I am trying to use Azure devop serviceswith databricks. I going to add git link in in git preferences. I have added all required field but save button is disable mode.I am not able to save changes with in…
asked 2021-06-10T11:54:15.38+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 36%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
PoojaSunkara
1
Reputation point
commented 2021-06-14T09:46:12.14+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
1 answer
Azure databrciks pyspark code needs support based on the file extension received in datalake
In my azure data lake everyday I will receive any one of the below file (only the file extension will vary but the file name remains same). File Name: Receipts_currendate.xlsx(extension is in small letter) or Receipts_currentdate.XLSX(extension is in…
asked 2021-06-10T08:53:48.223+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 80%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
Chitra Marimuthu
1
Reputation point
commented 2021-06-14T09:45:34.383+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Databricks connectivity to Eventhubs error
Hi I am connecting from Azure Databricks to event hubs using python program. I used the program from microsoft documentation. Python program runs well and inserts events into eventhub when run locally from my pc. When I run the program from databricks…
asked 2021-06-10T20:41:22.423+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 93%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKK%3C/text%3E%3C/svg%3E)
kranthi k senapathi
106
Reputation points
commented 2021-06-14T04:07:51.03+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
0 answers
databricks update, insert to non delta table
I have a table in databricks which is NOT a delta table. I would like to update rows or insert new ones. Both sql INSERT and UPDATE did not work for non delta tables. Is there any workaround? My table is saved as parquet. I know there is a way of…
asked 2021-06-09T16:28:16.283+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 26%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
braxx
436
Reputation points
commented 2021-06-11T11:31:14.403+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Error from databricks jobs API : Failed to retrieve tenant ID for given token
I have a pipeline in ADF that failed today with a message that isn't familiar. It was connecting to the Databricks REST API. Error 403 Failed to retrieve tenant ID for given token HTTP ERROR 403 Problem accessing /api/2.0/jobs/runs/submit. …
asked 2021-05-13T16:04:08.267+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 78%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
David Beavon
976
Reputation points
accepted 2021-06-09T17:10:05.113+00:00
David Beavon
976
Reputation points
1 answer
One of the answers was accepted by the question author.
Databricks access to ADLS Gen 1 using certificate
I've referred to the databricks documentation https://docs.databricks.com/data/data-sources/azure/azure-datalake.html#language-scala which talks about how to mount ADLS gen 1 using client id and secret. I wanted to check if doing the same using…
asked 2021-05-31T14:02:29.34+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 11%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECA%3C/text%3E%3C/svg%3E)
Chand, Anupam SBOBNG-ITA/RX
461
Reputation points
commented 2021-06-03T08:51:58.793+00:00
Chand, Anupam SBOBNG-ITA/RX
461
Reputation points
1 answer
Clusters in DataBricks
I am new to clusters and Databricks in general. We have a few jobs running in Databricks using clusters that have libraries configured to them. In order for the job to run successfully because the notebooks rely on libraries, the libraries need to be…
asked 2021-05-18T20:25:31.24+00:00
Sarah C Benjamin
1
Reputation point
commented 2021-05-31T12:12:45.213+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Searching In a blob file
Hi, I have a 30Gb csv file in my blob container. I have a list of about 1000 records, and I want to search for these records inside the large file. what is the right/best way to do that? I tried using local python script on my pc but of course it…
asked 2021-05-27T07:39:05.653+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 77%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Arkady Mankovsky
21
Reputation points
accepted 2021-05-31T06:22:19.197+00:00
Arkady Mankovsky
21
Reputation points
1 answer
Extract Delta Changes on big CSV files
We are exporting data from Microsoft Dataverse (Dynamics 365) into Azure Data Lake. The files are saved in csv formats and partitioned in yearly files based on the modified on date. The file could grow quite large for some frequently used tables in a…
asked 2021-05-20T03:46:23.307+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 28.999999999999996%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELY%3C/text%3E%3C/svg%3E)
Lintao Yu
1
Reputation point
commented 2021-05-28T17:16:23.847+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,081
Reputation points
2 answers
Cannot delete Databricks Workspace
Hi, I have a databricks workspace which I tried deleting, the deletion failed part way through and since then the provisioning state says "Deleting", however, it is not getting deleted. No new deletion requests work either, I get an error…
asked 2021-05-25T21:10:58.817+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 4%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Rahul Gupta
6
Reputation points
commented 2021-05-28T12:25:05.187+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
2 answers
Difference connecting directly to databricks workspace vs SQL Analytics workspace using Power BI
Hi, Wanted some good document links on what will be the advantages/disadvantages of connecting directly to data bricks workspace using Power BI to access datalake sources against connecting the same using SQL analytics workspace. Please suggest. …
asked 2021-05-23T14:35:17.943+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 0%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Abhishek Gaikwad
191
Reputation points
commented 2021-05-28T12:22:04.183+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Databricks Cluster Logs Driver stdout - delayed in dbfs location
I have a databricks cluster with logging enabled in a dbfs location. My process requires to read the cluster logs, specifically the driver/stdout logs. This stdout is nothing but the console output which is also visible in UI : Clusters -> ClusterName…
asked 2021-05-25T06:50:00.98+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 10%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAJ%3C/text%3E%3C/svg%3E)
Ayushri Jain
176
Reputation points
accepted 2021-05-28T06:31:10.67+00:00
Ayushri Jain
176
Reputation points
4 answers
One of the answers was accepted by the question author.
Azure Databricks Synapse Connectivity
We are trying to use PolyBase in Azure Data Factory to copy the Delta lake table to Synapse. Using a simple Copy Activity in Azure Data Factory, our linked Services connections from Delta lake and Synapse show connection is successful, yet the copy step…
asked 2021-03-29T20:30:55.797+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 41%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
Sahar Mostafa
26
Reputation points
answered 2021-05-26T16:30:02.147+00:00
brajesh jaishwal
26
Reputation points
1 answer
One of the answers was accepted by the question author.
While creating azure databricks resource getting below error
Hi Team, I am not able to see azure databrick resource in my subscription, and while going to azure databricks portal and creating the resource getting attached error, please help. Regards, Himanshu
asked 2021-05-25T05:37:45.507+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 84%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHJ%3C/text%3E%3C/svg%3E)
Himanshu Jain
21
Reputation points
commented 2021-05-25T10:00:10.983+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Azure and Python / Pandas (R)
We have our own Azure (windows) server where we manage everything except OS etc. We have access to the Azure Portal where we can add other servers and many other cool things but I am not used to that. I am a Python/anaconda guy so I installed…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 44%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
2 answers
One of the answers was accepted by the question author.
How to convert `null` in Azure Data Factory Copy Activity with PolyBase
Hi, I'm trying to copy a Databricks Delta lake table to Synapse, using the Azure Data Factory Copy Activity with PolyBase. The Delta lake table has two columns which are all nulls. Here are the sample data in the Delta lake table: …
asked 2021-04-13T19:44:50.967+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 75%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFZ%3C/text%3E%3C/svg%3E)
Fangzhou Zhang
231
Reputation points
answered 2021-05-24T16:43:29.627+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 76%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENM%3C/text%3E%3C/svg%3E)
Nicholas Moulton
1
Reputation point
1 answer
How can I clear these message errors. I am supposed tofind a page of the first Customer instead this full page message?
C# Step by Step Chapter 27: Pg 718 When I click Customers > View > Display in Chrome, I get the following Webpage, which is expected. But this command to show the first Customer of the AdventureWorks database must render a clean…
asked 2021-05-22T15:39:03.9+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 42%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Amos Matthew
26
Reputation points
answered 2021-05-22T17:44:54.013+00:00
Amos Matthew
26
Reputation points
1 answer
One of the answers was accepted by the question author.
Billing message while creating database in Azure fundamental online course
I signed up for Azure fundamental online course recently. While running the exercise on SQL database creation using the Sandbox option, a message popped up on the screen. ‘You don’t have permission to create a database. You will be billed by end of the…
asked 2021-05-18T08:51:52.433+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 23%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJD%3C/text%3E%3C/svg%3E)
Joseph Dulibeako
21
Reputation points
commented 2021-05-20T09:43:03.05+00:00
Joseph Dulibeako
21
Reputation points
1 answer
Loading 22 Mil + records into neo4j database via databricks environment
I am using neo4j spark connector and via databricks environment - I have loaded 22 Mil records into neo4j database and While establishing relationships for this 22mil records with the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 56.00000000000001%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)