Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,047 questions
3 answers
One of the answers was accepted by the question author.
Power BI custom columns
Hi everyone, I need to add a custom column in Power BI Edit Query, where the custom column is calculated from another table. Please help. Can this be done? Looking forward to your kind feedback and assistance. Regards Thabo
asked 2020-07-07T07:57:50.727+00:00
Thabo Mofokeng
21
Reputation points
answered 2020-10-30T05:21:27.51+00:00
Rachit Malik
1
Reputation point
1 answer
One of the answers was accepted by the question author.
Azure Databricks queries for monitoring using log analytics
Hi, I have premium version of azure databricks and i have enabled monitoring as well in this .. so i have sent its all data to log analytics workspace .. now i want to query on databricks .. can u provide me with written queries so i can use them in my…
asked 2020-10-20T06:30:51.913+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 28.000000000000004%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERB%3C/text%3E%3C/svg%3E)
Rohit Boddu
461
Reputation points
accepted 2020-10-30T03:42:23.637+00:00
Rohit Boddu
461
Reputation points
1 answer
One of the answers was accepted by the question author.
Locking down a port on Azure Databricks (managed resoruce)
I am trying to lock down a high risk port on a managed azure databricks workspace. My problem is the exact opposite of the request on the question "Opening up port 443 on Azure Databricks (managed resource)". Instead of wanting to open up…
asked 2020-10-20T00:21:22.12+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 43%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDW%3C/text%3E%3C/svg%3E)
Daniel Widdis
21
Reputation points
commented 2020-10-28T03:31:52.943+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Read files from Azure File share using databricks
How can I read files stored in Azure file share using databricks. I am able to access container blob storage, but not the files in file share.
asked 2020-10-21T08:52:47.307+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(48, 91%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Jeeva
161
Reputation points
commented 2020-10-28T03:28:44.43+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
asked 2020-10-25T16:16:37.29+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 97%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBH%3C/text%3E%3C/svg%3E)
BATRAH HAMZA
1
Reputation point
answered 2020-10-26T14:53:48.253+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 45%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGM%3C/text%3E%3C/svg%3E)
GiftA-MSFT
11,161
Reputation points
0 answers
How to call websocket api and make a data set based on time?
I want to call websocket api from data anlytics or data factory which gives continous data from api. i want to read that data and based on time after every 5 minutes want to take out open high low and close values from that set. i can able to do it using…
asked 2020-10-11T07:36:44.293+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 13%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAF%3C/text%3E%3C/svg%3E)
ashish fugat
11
Reputation points
commented 2020-10-22T16:58:21.693+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure databricks workload type
I am calling a azure databricks cluster from a AI/ML pipeline. I want to understand it will get executed under which feature type: Data analytics or data engineering. I want to understand this to get my databricks usage cost calculation.
asked 2020-10-19T21:51:01.31+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 55.00000000000001%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Sandhya Prusty
21
Reputation points
commented 2020-10-22T07:10:38.28+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
1 answer
Why we dont have diagnostic settings feature in Monitoring Section of "Azure Databricks of standard version"
Hi, I want to monitor Azure Databricks using Azure monitor .. I first want to send all its diagnostics settings to log analytics and then i can analyze from there. but there is no diagnostics setting feature Available for "Azure databricks Standard…
asked 2020-10-16T09:32:25.407+00:00
Rohit Boddu
461
Reputation points
commented 2020-10-21T11:15:44.667+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure Databricks table
Hi, One of my Datascientist by mistakenly delete data file working on Azure Databricks. Now How to restore deleted table from Azure Data Lake?
asked 2020-10-16T19:55:36.973+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 46%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVK%3C/text%3E%3C/svg%3E)
Vijay Kumar
2,026
Reputation points
accepted 2020-10-20T16:05:50.507+00:00
Vijay Kumar
2,026
Reputation points
2 answers
How to maintain uniqueness in dataframe in databricks
When you load data into dataframes in databricks how can you make sure the rows in dataframes are not duplicated. In SQL you can handle by using unique constraint on the tables. how this can be handle in dataframes to ensure rows are not duplicated.
asked 2020-10-13T13:01:40.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 0%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Abhishek Gaikwad
191
Reputation points
commented 2020-10-15T18:31:35.803+00:00
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Getting SSL certificate error while connecting to Synapse SQL pool from Databricks in Azure using JDBC
--------------------------------------------------------------------------- Py4JJavaError Traceback (most recent call last) <command-3031308706598466> in <module> 6 .option("tempDir",…
asked 2020-10-06T17:17:39.827+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 91%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Ajay Verma
21
Reputation points
commented 2020-10-15T05:11:29.787+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
1 answer
notebook-scoped lib failing to find the dependent cluster-scoped lib
Our app hit an issue with ModuleNotFoundError recently with Databricks Runtime 6.4 and it looks this is a databricks issue, so wanted to ask about that. The problem can happen as follows: Two libraries A.B and A.C; A.C depends on A.B Install A.B…
asked 2020-10-07T21:45:54.727+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 41%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETT%3C/text%3E%3C/svg%3E)
Tri Tran
1
Reputation point Microsoft Employee
answered 2020-10-14T21:35:03.173+00:00
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
4 answers
Unable to read csv file from cosmos gen1 through spark. Pandas read works.
This works, df = pd.read_csv('/dbfs/mnt/ajviswan/forest_efficiency/2020-04-26_2020-05-26.csv') sdf = spark.createDataFrame(df) sdf.head() But spark read does not work. df =…
asked 2020-10-08T15:35:40.867+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 91%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAP%3C/text%3E%3C/svg%3E)
Ajay Prasadh Viswanathan
1
Reputation point Microsoft Employee
commented 2020-10-14T11:34:54.997+00:00
Ajay Prasadh Viswanathan
1
Reputation point Microsoft Employee
1 answer
Azure Databricks Access Controls in Programmatic way
Can someone please let me know is there any programmatic way to implement Access control on Azure Databricks on Workspace object, cluster, pool, job, and table etc.. Any powershell or API for the same ?
asked 2020-10-02T11:20:29.373+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Akash Verma
21
Reputation points
commented 2020-10-14T10:18:58.663+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
1 answer
Azure Databrick workspace linking to Azure Machine Learning workspace error unable to get Workspace.from_config()
We are in a POC of Azure Databricks and Azure Machine Learning integration so that we can advance our MLOps practice. We have been dealing the with workspace linking issue for the past few weeks and have hit a wall. We worked with Databricks and search…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 33%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETD%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 83%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Writing parquet file throws…An HTTP header that's mandatory for this request is not specified
I have two ADLSv2 storage accounts, both are hierarchical namespace enabled. In my Python Notebook, I'm reading a CSV file from one storage account and writing as parquet file in another storage, after some enrichment. I am getting below error when…
asked 2020-10-12T23:48:19.033+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 33%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPM%3C/text%3E%3C/svg%3E)
Porsche Me
131
Reputation points
accepted 2020-10-13T13:44:57.697+00:00
Porsche Me
131
Reputation points
3 answers
is there any limitation of writing billions of record from Azure Data bricks to Azure SQL using JDBC
Hi All, we are follow up this url- https://video2.skills-academy.com/en-us/azure/databricks/data/data-sources/sql-databases we want to know is there any limitation of writing billions of record from Azure Data bricks to Azure SQL using JDBC …
asked 2020-10-08T14:37:03.543+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 20%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
manish verma
441
Reputation points
commented 2020-10-12T21:06:38.3+00:00
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
1 answer
Where to store delta-files
Hey guys! I have been wondering about the below question for a while and I hope you can help me get a good night sleep again. Question: When you are working in Azure DataBricks to do transformations, you can save the results using a delta format.…
asked 2020-10-05T13:05:54.297+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 21%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECG%3C/text%3E%3C/svg%3E)
Christian Gert Hansen
1
Reputation point
commented 2020-10-08T20:15:56.767+00:00
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
1 answer
DatabricksL Error Code 3208
This issue was already reported in Git50030Hub. We have the same issue. Very little technical details here, can someone from Microsoft add more technical details? Microsoft's t-shooting doc here is very amusing. This issued showed up in our newly created…
asked 2020-10-06T19:08:16.867+00:00
Porsche Me
131
Reputation points
commented 2020-10-08T10:23:40.55+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
1 answer
'databricks configure --token' command waiting for the input
I run following task in Azure DevOps, it always waiting for input? Why wasn't my bash automatic supply working? Python 3.7 installed databricksUrl=https://... databricksToken=***** databricks configure --token…
asked 2020-10-01T04:37:38.963+00:00
Porsche Me
131
Reputation points
commented 2020-10-07T13:05:45.757+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee