Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,070 questions
1 answer
One of the answers was accepted by the question author.
Best way to connect to a databricks datamart for further exploration in PowerBI
Situation: Databricks is used as enterprise data platform. In reality local teams are moving forward with a different speed than the global organization is. To support this but still keeping control on the "golden source" we're setting up an…
asked 2021-01-14T13:44:27.313+00:00
Peter Verrykt
21
Reputation points
accepted 2021-01-17T13:10:39.733+00:00
Peter Verrykt
21
Reputation points
1 answer
How do I orchestrate ML model retraining periodically?
I have to retrain every month or so a PyTorch Model trained on data obtained from processing tables sitting in Azure Data Lake Storage gen 1. So far, I have the following building blocks: A Databricks notebook that does the ETL job of…
asked 2021-01-08T23:32:08.983+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 62%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDF%3C/text%3E%3C/svg%3E)
Davide Fiocco
31
Reputation points
answered 2021-01-11T08:57:22.71+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 11%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERM%3C/text%3E%3C/svg%3E)
Ramr-msft
17,731
Reputation points
1 answer
One of the answers was accepted by the question author.
What is the use of oldest-time-to-consider param in Jobs API?
Hi, I haven't found any documentation around what this value "oldest-time-to-consider": "1457570074236" is used for in the Databricks Job API. Can someone please direct me to the documentation that talks about the significance of…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 21%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Can you write multiple streaming queries(same schema, different input sources) into same Azure storage without overwriting?
Hi, I have to this requirement to write multiple streaming queries(same schema, different input sources) into same Azure blob delta lake gen 3 storage without overwriting. I need the data to co-exist in the same write directory, say like in 'append'…
asked 2020-12-30T20:18:54.43+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 23%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Mayuri Kadam
81
Reputation points Microsoft Employee
accepted 2021-01-04T16:59:00.253+00:00
Mayuri Kadam
81
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Is there any way to do Custom dynamic mapping of different number of columns in dataflow or any other options to achieve this?
My source (CSV file in ADLS) has header record(3 columns) , detail records(5 columns) and trailer record(2 columns) . The header record has less number of columns than the detail records. When I try to convert this csv file to parquet, i m getting the…
asked 2020-12-30T07:35:06.643+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 85%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAC%3C/text%3E%3C/svg%3E)
Abarna C (DAAI - Cloud Data Platforms)
21
Reputation points
accepted 2021-01-04T07:01:50.97+00:00
Abarna C (DAAI - Cloud Data Platforms)
21
Reputation points
1 answer
MLOps using Azure Databricks & Azure ML - question on data prep for model inference and retraining.
I am using this blog (https://databricks.com/blog/2020/10/13/using-mlops-with-mlflow-and-azure.html) to set-up MLOps using Azure Databricks & Azure ML. As mentioned in the blog, we deploy MLflow model into an Azure ML environment using the built in…
asked 2020-12-14T15:09:41.913+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 66%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKP%3C/text%3E%3C/svg%3E)
Kiran Purushotham
11
Reputation points
commented 2021-01-02T12:20:39.327+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 55.00000000000001%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
GermanM
1
Reputation point
1 answer
How to use ARM template to restrict publicBlobAccess to managed Databricks storage account
In our organisation we are required to disable publicBlobAccess and enable TLS1_2 as minimum version on all storage accounts. Preferably we also use StorageV2 type instead of BlobStorage. When we create Databricks workspace, using ARM template, the…
asked 2020-12-04T07:52:20.833+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 22%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELV%3C/text%3E%3C/svg%3E)
Leerdam van, J (Jean-Marc)
16
Reputation points
commented 2020-12-24T08:18:12.133+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 45%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJK%3C/text%3E%3C/svg%3E)
Janne Kujanpää
216
Reputation points
1 answer
How to query 3rd party Azure DataLake Gen2 and only store the results
First, what I am trying to do is I want to query and aggregate raw JSON files stored in a 3rd party's Azure Data Lake (Gen2) and store those aggregates in my own data lake or relation db. I do not want to physically copy all of those raw JSON files…
asked 2020-12-15T16:14:02.687+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 56.00000000000001%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ5%3C/text%3E%3C/svg%3E)
JasonW-5564
161
Reputation points
commented 2020-12-22T12:18:09.123+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Linkedin connectivity with Azure
Hi, Is there a way to extract existing content from a Linkedin company page using Azure? Thanks
asked 2020-12-16T07:47:10.393+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 59%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDR%3C/text%3E%3C/svg%3E)
Darshika Rajendran
1
Reputation point
commented 2020-12-22T12:17:25.62+00:00
HarithaMaddi-MSFT
10,136
Reputation points
0 answers
Azure repo - Can't create or push tag
I host a project on Azure DevOps Repositories and I would like to create several git tag for project version release. I'm a GitKraken user so I added a new tag and push it to origin but this error occurs: So I checked my permission on Azure…
asked 2020-12-11T13:51:52.037+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 17%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMT%3C/text%3E%3C/svg%3E)
MATTONE THOMAS
1
Reputation point
commented 2020-12-19T11:42:47.243+00:00
Jaliya Udagedara
2,821
Reputation points MVP
1 answer
The file updated in databricks is not reflecting in Azure Portal
I have a databrick workspace, where I read a file from Azure blob storage, updated a file and uploaded it in another Azure blob storage space. Now when I access that file through any databricks workspace, I can see the file and access the content of it.…
asked 2020-12-11T03:08:59.943+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 2%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAP%3C/text%3E%3C/svg%3E)
Amey Pimpley
1
Reputation point
commented 2020-12-18T05:42:41.683+00:00
PRADEEPCHEEKATLA-MSFT
85,346
Reputation points Microsoft Employee
1 answer
I HAVE ERROR WHEN AZURE DATABRICKS WRITE COSMOS DB
I am trying to write a spark dataframe from azure databricks to a cosmos db database and I have this error Py4JJavaError: An error occurred while calling o753.save. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage…
asked 2020-12-05T13:57:36.433+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 26%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWG%3C/text%3E%3C/svg%3E)
Williams Gerard Gamboa Anchante
1
Reputation point
commented 2020-12-17T22:01:17.41+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
0 answers
Deletion of managed MLflow Artifacts
When using Workspace experiments in Azure Databricks with the default managed MLflow artifact location dbfs:/databricks/mlflow-tracking configured, we see the following message when deleting an MLflow experiment run: Deleted runs are restorable for…
asked 2020-11-20T08:50:37.143+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 86%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECS%3C/text%3E%3C/svg%3E)
Christoph Stumpf
1
Reputation point
commented 2020-12-17T19:35:10.29+00:00
HimanshuSinha-msft
19,381
Reputation points Microsoft Employee
1 answer
Unable to delete folder in databricks "DBFS://
Hi, When I run the command %fs ls '/' in the results I see a folder path as "dbfs://" and name as "/". and tried to run the command in the notebook %fs ls '//' I get the java error and even not able to delete the folder. Please…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 74%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
using streaming batch for multiple operations
I am new to spark and DataBricks and was trying to look for a solution where I can utilize a batch from a eventhub stream to accomplish multiple business logic but could not find any guidance. Stream I get from EventHub is a CDC stream from multiple…
asked 2020-12-15T04:00:40.087+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 2%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Rohit Sapru
41
Reputation points
commented 2020-12-16T05:15:45.41+00:00
PRADEEPCHEEKATLA-MSFT
85,346
Reputation points Microsoft Employee
1 answer
Databricks readstream writestream to Azure Synapse
I am having an issue on writing stream to Azure synapse with the following error . let's have a look and see if there is idea ?
asked 2020-12-10T06:08:39.147+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 40%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
sakuraime
2,321
Reputation points
commented 2020-12-15T06:19:23.62+00:00
PRADEEPCHEEKATLA-MSFT
85,346
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to set security permissions to Databases in databricks through Notebooks
We are stuck on the way to set security permissions to Databases by using Notebooks %sql. At first, let me explain our situations and settings. We run the following code on Notebooks: %sql CREATE DATABASE X ; GRANT USAGE ON DATABASE X TO…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 8%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Databricks Pyspark exception handling best practices
Hi, In the current development of pyspark notebooks on Databricks, I typically use the python specific exception blocks to handle different situations that may arise. I am wondering if there are any best practices/recommendations or patterns to handle…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 87%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
1 answer
Azure Databricks Cluster
Hi, Created new cluster in Databricks (QA environment) Afte that when i try to click on data tab inroder to create database getting below error checked that cluster is up and running.
asked 2020-11-25T07:27:39.867+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 46%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVK%3C/text%3E%3C/svg%3E)
Vijay Kumar
2,031
Reputation points
commented 2020-12-11T09:29:48.567+00:00
PRADEEPCHEEKATLA-MSFT
85,346
Reputation points Microsoft Employee
2 answers
Quota limit hit on tutorial notebook
I'm attempting to launch a default cluster (min 2, max 8) on the premium trial account in order to run 01-The-Databricks-Environment. I haven't been able to run any operations in the notebook. Receiving this error: Azure error code:…
asked 2020-12-07T20:09:56.14+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 50%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EZW%3C/text%3E%3C/svg%3E)
Zak Wear
1
Reputation point
commented 2020-12-11T04:53:19.223+00:00
PRADEEPCHEEKATLA-MSFT
85,346
Reputation points Microsoft Employee