Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,150 questions
0 answers
How to pass Wild Card File Name in ADF Databricks Activity
HI , I am working in for each Activity in the ADF , in For each we have copy activity and then we pass the table name to databricks as parameter . Copy activity supports wild card paths @Anonymous (item().TABLE_NAME,'*')---- its picking…
asked 2021-04-26T05:20:37.887+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 28.000000000000004%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Shubham Bartakke
1
Reputation point
commented 2021-05-03T19:15:23.323+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
Error when triggering Databricks notebook via datafactory while reading XML file
I am reading xml file in spark databricks and below is the command I am using to read the file. val readxml =…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(83.2, 0%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
0 answers
Databricks Cluster (GC issue)
I have the below cluster setting. Currently, I am getting data every 30 Milliseconds. So for 1 hr, I am getting around 1 GB of Data. I am running this job every 5 hrs but it is throwing me GC error, Please help me. Also attached my current…
asked 2021-04-14T09:25:38.62+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 55.00000000000001%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EIM%3C/text%3E%3C/svg%3E)
Imran Mondal
246
Reputation points
commented 2021-05-03T06:24:21.29+00:00
Imran Mondal
246
Reputation points
1 answer
One of the answers was accepted by the question author.
What python libraries have databricks pre-installed and do not need to be installed in the cluster?
What python libraries have databricks pre-installed and do not need to be installed in the cluster?
asked 2021-04-27T14:17:38.3+00:00
Daniel Ceballos
21
Reputation points
accepted 2021-04-30T01:10:07.677+00:00
Daniel Ceballos
21
Reputation points
2 answers
One of the answers was accepted by the question author.
databricks/simba jdbc driver - warnings/error messages
I've setup a sql ide (workbench/j) for my sparksql databricks querying - writing code in a web page textarea control is just not for me... https://video2.skills-academy.com/en-us/azure/databricks/integrations/bi/workbenchj i'm getting this…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 1%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJK%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
parsing json databricks / python
I am very beginner in databricks and python, so it maybe basics for you but for me it is still new. I am trying to parse the json to get all childitems form "valid" and "date_of_creation" sections as columns in table which I will…
asked 2021-04-27T14:28:02.383+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 26%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
braxx
436
Reputation points
commented 2021-04-29T14:26:22.617+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,796
Reputation points Microsoft Employee
1 answer
Azure databricks create Azure Key vault secret scope failed
I am the contributor of the key vault , and I am the admin of the databricks workspace , and I have the following error while creating the secret scope. I know after click create , it would try to add "AzureDatabricks" service…
asked 2021-04-22T13:15:41.92+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 40%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
sakuraime
2,326
Reputation points
commented 2021-04-27T10:45:38.147+00:00
PRADEEPCHEEKATLA-MSFT
88,381
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Log cluster name
When i run any notebook I want to log by using which cluster the notebook was executed. Is their any way to get the cluster name using python in databricks
asked 2021-04-24T09:04:03.153+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 0%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Abhishek Gaikwad
191
Reputation points
accepted 2021-04-27T10:17:04.947+00:00
Abhishek Gaikwad
191
Reputation points
1 answer
One of the answers was accepted by the question author.
User level auditing in Databricks Azure
We are looking into collecting audit logs for Azure Databricks. We have enabled diagnostic settings for the databricks workspace which sends log data to log analytics, it contains the monitoring audit logs, as in high level - who created which…
asked 2021-04-26T09:00:01.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 10%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAJ%3C/text%3E%3C/svg%3E)
Ayushri Jain
171
Reputation points
accepted 2021-04-27T06:33:12.43+00:00
Ayushri Jain
171
Reputation points
1 answer
Databricks Spark connection issue over Simba JDBC
I am trying to connect Spark Databricks from PERL code over Simba JDBC (Databricks recommended way) .For ref this is the JDBC driver:…
asked 2021-04-17T06:54:20.217+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 61%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDP%3C/text%3E%3C/svg%3E)
Debashis Paul
1
Reputation point
commented 2021-04-27T04:58:28.283+00:00
PRADEEPCHEEKATLA-MSFT
88,381
Reputation points Microsoft Employee
2 answers
I am not getting all the files using listFiles(basep: String, globp: String) in Azure Databricks
Hi, I have a question regarding this article https://video2.skills-academy.com/en-us/azure/databricks/kb/data/list-delete-files-faster I have a directory in my ADLS in which I have some folders and inside folders, I have some parquet files. I was using…
asked 2021-04-23T06:01:58.243+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 21%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Manoj Kumar
1
Reputation point
answered 2021-04-26T12:49:34.373+00:00
Manoj Kumar
1
Reputation point
1 answer
One of the answers was accepted by the question author.
pass variable in drop down from another notebook
Hi, I am trying to pass a variable from notebook A to Notebook B which has drop down widget list. However getting below error. error message : select sequence must include X Notebook A has below variable defined. parameters_global['Source'] =…
asked 2021-04-20T06:28:10.157+00:00
Abhishek Gaikwad
191
Reputation points
accepted 2021-04-23T08:56:30.397+00:00
Abhishek Gaikwad
191
Reputation points
1 answer
One of the answers was accepted by the question author.
Attribute error when mounting blob
It is my second day with databricks and Python. So forgive me if my question sounds silly. When mounting a blob storage container I got "Attribute error'. I previously created scope credentials and SAS token (for container), which is stored in…
asked 2021-04-20T14:57:02.473+00:00
braxx
436
Reputation points
commented 2021-04-23T04:17:20.457+00:00
PRADEEPCHEEKATLA-MSFT
88,381
Reputation points Microsoft Employee
1 answer
How to add extended infrastructure IP addresses in nsg for databricks ?
Hi Team, Could you please help me to add extended infrastructure IP addresses to databricks nsg ? MS tracking id:1TW4-LDG Thanks Sidhartha
asked 2021-04-14T02:40:31.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(131.20000000000002, 56.00000000000001%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
sidhartha mohanty
1
Reputation point
commented 2021-04-19T03:57:25.823+00:00
PRADEEPCHEEKATLA-MSFT
88,381
Reputation points Microsoft Employee
1 answer
Cannot mount Azure VM(windows server) file share on Azure Databricks instance.
Hi, I wanted to mount windows file share(from Windows server VM in Azure) on a databricks instance. To test it I followed these steps. I created a Windows VM and a Linux VM in azure. I created VNet-peering between these two VMs. After…
asked 2021-04-07T19:45:26.647+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(92.8, 65%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sai Suresh (Quadrant Resource)
1
Reputation point
commented 2021-04-19T03:55:51.58+00:00
PRADEEPCHEEKATLA-MSFT
88,381
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
azure databricks list and delete screte scopes
there is command line databricks secrets list-scopes to list scopes however , are there URL method to list and delete secret scopes like https://adb-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.azuredatabricks.net#secrets/createScope <<< to…
1 answer
One of the answers was accepted by the question author.
Pass different parameter to same notebook databricks
I have a notebook in python which provides profiling output results for the table . I want to schedule this notebook. For example the notebook name is Profiling notebook and i have table A,B,C,D.... When i schedule this notebook do i have to…
asked 2021-04-15T11:11:13.613+00:00
Abhishek Gaikwad
191
Reputation points
accepted 2021-04-18T05:31:56.213+00:00
Abhishek Gaikwad
191
Reputation points
1 answer
pyspark data frame for json
hello all , i am using MS-Azure databricks platform for python+spark , cluster = spark 3.1.1 kindly tell me how can i import json file in it syntax that i have used now: df =…
asked 2021-04-10T04:29:35.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 79%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Abdul Sattar
1
Reputation point
commented 2021-04-16T16:57:06.143+00:00
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
0 answers
How to add custom validation on excel/csv/sql ?
I want to add a custom validation like column which contain a date that follows a specific format like mm:yyyy and number with specific range in Azure data factory
asked 2021-04-06T15:35:20.113+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 47%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Shubham Dhumal
1
Reputation point
commented 2021-04-16T15:39:15.55+00:00
Saurabh Sharma
23,796
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
How to round and convert the nested values from dictionary to integer in Python?
enc = {'Category': {'a1': 1885.4403010060062, 'a2': 1072.0176453884112, 'a3': 1448.579447138836, 'a4': 1471.5656350474158, 'a5': 1232.3527616505146, 'a6': 891.9808067951503, 'a7': 1378.9613179304858}} I have tried this. for key, value in…
asked 2021-04-13T14:24:43.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 12%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Siddhesh Bhurke
46
Reputation points
commented 2021-04-16T13:46:45.637+00:00
Siddhesh Bhurke
46
Reputation points