Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,045 questions
2 answers
Databricks cluster does not work with free trial subscription
In Azure Databricks, tried to create cluster. Cluster creation requires to provide worker type and driver type, in which minimum 4 cores are required on each which means in total 8 cores. Free trail subscription gives only 4 cores what I read. So how…
asked 2020-06-11T22:24:30.773+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 42%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Seuli Das Majumder
101
Reputation points
answered 2020-10-01T14:36:15.337+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
1 answer
dbutils.widgets.dropdown() issue: com.databricks.dbutils_v1.TooManyDefaultChoices: Too many default choices (1116). Limit is 1024
HI, While running dbutils.widgets.dropdown(), I face the following issue: com.databricks.dbutils_v1.TooManyDefaultChoices: Too many default choices (1116). Limit is 1024. I searched but could not find an answer, how to change the default value. For sure,…
asked 2020-09-21T12:44:16.483+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 37%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECK%3C/text%3E%3C/svg%3E)
Chaitanya Kumar Dondapati (Volkswagen)
1
Reputation point
commented 2020-09-30T19:48:14.75+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,051
Reputation points
1 answer
Mounting entire ADLS on azure databricks
Hi, I want to mount entire ADLS storage account on databricks. I've checked in documents I can mount a single filesystem at a time but I want to mount entire ADLS on databricks. I've around 70 containers in my ADLS and I want to mount all of them…
asked 2020-09-23T11:35:43.993+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 38%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashima Gupta
1
Reputation point
commented 2020-09-29T19:04:09.19+00:00
MartinJaffer-MSFT
26,051
Reputation points
1 answer
One of the answers was accepted by the question author.
remainingCores is too small error. Spark and Kafka cluster on HDInsight problem
I tried to create a Kafka and Spark cluster in the HDInsight using either of templates in these links: …
asked 2020-09-27T06:57:32.923+00:00
Erfan Mahmoodnejad
41
Reputation points
commented 2020-09-29T06:31:58.623+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Can i access Gen1 and Gen 2 datalake sources from the same databricks cluster
If I have access to Gen 1 datalake sources on databricks cluster. Can I add Gen2 sources on the same databricks cluster and access them both Gen 1 and Gen 2 from the same databricks cluster Or Do i have to create separate cluster for connecting to gen…
asked 2020-09-27T05:54:11.377+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 0%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Abhishek Gaikwad
191
Reputation points
accepted 2020-09-29T03:56:29.637+00:00
Abhishek Gaikwad
191
Reputation points
1 answer
Calling Azure Databricks query from Azure Function
Hi Everyone, Is possible to call a SQL query in Azure Databricks(Cluster) inside a Azure Function ? Normally to conenct SQL server from Azure function, we use to create connection string in application seetings and use to retrive using environmnet…
asked 2020-09-16T13:24:48.947+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 84%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
Gowrikanth
1
Reputation point
commented 2020-09-25T08:50:42.997+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 79%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKG%3C/text%3E%3C/svg%3E)
Krish G
2,331
Reputation points
0 answers
Is it possible to add a Databricks Role/Group to appRoles for OAuth 2.0(Azure AD) tokens
I have configured a service principal for Azure Databricks in Azure AD as per this documentation: https://video2.skills-academy.com/en-us/azure/databricks/dev-tools/api/latest/aad/service-prin-aad-token I am able to access Azure Databricks using an OAuth…
asked 2020-09-09T11:48:28.817+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 4%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMR%3C/text%3E%3C/svg%3E)
Mukil Rajeev
1
Reputation point
commented 2020-09-16T22:00:52.237+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,437
Reputation points Microsoft Employee
1 answer
Alternative to using Azure SQL Managed Instance for Complex calculations
I have inherited an Architecture which carries out complex calculation in Azure SQL Managed Instance via Databricks. The Databricks is connected via Apache Spark connector. The Databricks does the calculations in SQL and the end results are in Tables…
asked 2020-09-02T16:31:09.39+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 90%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
Kman
41
Reputation points
commented 2020-09-16T19:45:52.69+00:00
MartinJaffer-MSFT
26,051
Reputation points
1 answer
Databricks certification in Azure
Hello team, I am planning to do certification in Databricks.So I just wanted to know is there any certificate on databricks on Azure if yes please advise the exam number otherwise which one i can get certified. Please advise Thanks in advance …
asked 2020-09-16T17:01:16.283+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 24%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kulkarni, Rohit
1
Reputation point
answered 2020-09-16T17:02:51.113+00:00
Leon Laude
85,716
Reputation points
1 answer
Not able to delete any resource
Please help me regarding this error. Not able to delete resource group.
asked 2020-09-10T09:57:36.12+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 89%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDC%3C/text%3E%3C/svg%3E)
demo case
1
Reputation point
commented 2020-09-14T07:51:40.453+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Apache Spark Event Hubs connector OAuth authentication - is it supported?
Does anyone has an example of reading streams from Azure EventHub using OAuth authentication in Databricks? I find only examples which use SAS keys and to be honest I am sure whether this is officially supported.
asked 2020-09-08T07:58:49.787+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 67%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGS%3C/text%3E%3C/svg%3E)
Georgi Slavov
1
Reputation point
answered 2020-09-10T09:29:59.22+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
1 answer
Security Recommendations for Azure Data and Analytics Services
I am working on Securing Data and Analytics Services on Azure. I want to know what security controls i can apply after creating of services and what i can apply only during the service creation. Below are the recommendation i have found as of now. Could…
asked 2020-09-02T20:20:43.98+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Akash Verma
21
Reputation points
commented 2020-09-09T11:11:32.767+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
1 answer
how add class library for BeautifulSoup in Azure Databricks
hi all, how add class library for BeautifulSoup in Azure Data-bricks i want to run below code in pyspark notebook from bs4 import BeautifulSoup import pandas as pd table = BeautifulSoup(open('C:/age0.html','r').read()).find('table') df =…
asked 2020-08-27T07:00:54.517+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 57.00000000000001%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVM%3C/text%3E%3C/svg%3E)
Verma, Manish Kumar
131
Reputation points
commented 2020-09-02T04:36:38.853+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
1 answer
Want to Read Microsoft Team Post in Azure Data Factory
I Have requirement to read Microsoft Team post in Azure Data Factory. currently i am using logic apps , but it generate output in html file and it is difficult to read what is the best way to get data from Microsoft team, so we can read data in…
asked 2020-08-27T08:52:24.843+00:00
Verma, Manish Kumar
131
Reputation points
commented 2020-08-31T06:13:56.377+00:00
MayankBargali-MSFT
69,946
Reputation points
2 answers
Is there any other free streaming sources like twitter for practicing spark and Kafka
Friends I am learning Kafka and Spark. I worked in Kafka and spark integration using Twitter api but I want to do more practice Is there any other free streaming sources like twitter for practicing spark and Kafka
asked 2020-08-25T18:56:42.463+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(9.6, 76%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Surendiran Balasubramanian
1
Reputation point
commented 2020-08-31T04:31:30.637+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Using pyspark dataframe input insert data into a table
Hello, I am working on inserting data into a SQL Server table dbo.Employee when I use the below pyspark code run into error: org.apache.spark.sql.AnalysisException: Table or view not found: dbo.Employee;. The table exists but not being able to insert…
1 answer
Write json document to azure table
Hi, I am using below code to write json document in a Azure Data Lake Gen2 container into a SQL Server table. Code: df =…
asked 2020-08-21T05:07:09.597+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 20%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERD%3C/text%3E%3C/svg%3E)
Raj D
586
Reputation points
commented 2020-08-26T10:55:25.62+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
1 answer
how to dynamically explode array type column in pyspark or scala
HI, i have a parquet file with complex column types with nested structs and arrays. I am using the scrpit from below link to flatten my parquet file. https://video2.skills-academy.com/en-us/azure/synapse-analytics/how-to-analyze-complex-schema …
asked 2020-08-19T18:34:33.73+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 37%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
reddy
41
Reputation points
commented 2020-08-26T10:54:33.947+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
1 answer
how to move compressed parquet file using adf or databricks
hi, i have a requirement to move parquet files from aws s3 into azure then convert to csv using adf. i tried to download that few files on to my local file system and tried to copy via copy activity within adf. The files are in this format …
asked 2020-08-11T14:40:54.08+00:00
reddy
41
Reputation points
commented 2020-08-24T06:01:31.98+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
1 answer
How to Transform files in subfolders with one script in databricks
i have a adls gen2 folder with sub folders with parquet files in each folder. My requirement is to transform all parquet files in sub folders and load into another folder in adls gen 2 with same folder structure with one script. is it possible to do or…
asked 2020-08-14T16:55:40.797+00:00
reddy
41
Reputation points
commented 2020-08-21T15:37:42.363+00:00
HarithaMaddi-MSFT
10,136
Reputation points