Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,039 questions
1 answer
How to solve Invalid SessionHandle error with Azure Databricks ?
I am applying a SQLDatabaseChain Chatbot model by using LangChain SQLDatabaseChain and GPT4. I first created this model on Azure Databricks notebook like this : import json import os import langchain import mlflow from mlflow.models import…
asked 2024-06-17T14:38:14.4333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 46%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFY%3C/text%3E%3C/svg%3E)
Feng YIJUN
0
Reputation points
edited an answer 2024-07-01T07:42:48.6866667+00:00
Feng YIJUN
0
Reputation points
0 answers
While running SQL query in Azure Databricks workspace i.e. on SQL warehouse as well as on UC enabled shared cluster facing an SSL handshake error
Hello Team, We have UC enabled Azure databricks workspace, also the Public access and delta sharing is disabled on our workspace. So while running the below SQL query on SQL Warehouse as well as on UC enabled shared cluster, I am receiving an…
asked 2024-06-28T10:06:18.7566667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 24%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
110
Reputation points
commented 2024-07-01T05:39:55.8333333+00:00
PRADEEPCHEEKATLA-MSFT
83,891
Reputation points Microsoft Employee
1 answer
How do I share all of my databricks notebooks with all databricks users?
Hi all, I know that I've done this in the distant past, but we have a new instance of Databricks and I need to do a global setting to share all of my notebooks with all Databricks users (read only). That way I don't need to remember to share individual…
asked 2024-06-27T13:57:35.7666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 15%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBK%3C/text%3E%3C/svg%3E)
Barb K
0
Reputation points
edited a comment 2024-07-01T05:28:58.0633333+00:00
PRADEEPCHEEKATLA-MSFT
83,891
Reputation points Microsoft Employee
1 answer
while creating cluster in databricks i am getting following error Azure Quota Exceeded Exception: Error code: SkuNotAvailable, error message: The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_DS3_v2' is cur
while creating cluster in databricks i am getting following error Azure Quota Exceeded Exception: Error code: SkuNotAvailable, error message: The requested VM size for resource 'Following SKUs have failed for Capacity Restrictions: Standard_DS3_v2' is…
asked 2024-06-28T12:09:26.57+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 83%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
azurecloud
0
Reputation points
edited an answer 2024-07-01T04:37:52.52+00:00
PRADEEPCHEEKATLA-MSFT
83,891
Reputation points Microsoft Employee
1 answer
array in databricks
Hi , I got the 1st row in sorted order how can i get the view on row1,row2,row3 using array and how the reshuffling will happen 1,2,3,4,5 --- sorted order 5,1,2,3,4 - 1 4,5,1,2,3 - 2 3,4,5,1,2 - 3
asked 2024-06-25T08:20:36.5333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
265
Reputation points
commented 2024-06-30T15:08:50.3966667+00:00
PRADEEPCHEEKATLA-MSFT
83,891
Reputation points Microsoft Employee
0 answers
DataBricks Unity Catalog Lineage
Hi, I'm looking for support on the Databricks Unity Catalog (on the data lineage). So I'm trying to establish lineage between 2 schemas (with 50 odd tables within each schema). Data for the first schema is fetched from source files (via ADF pipeline),…
asked 2024-06-20T06:15:49.5666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 34%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
S, Santhosh M
0
Reputation points
commented 2024-06-30T15:03:31.34+00:00
PRADEEPCHEEKATLA-MSFT
83,891
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure Databricks fails
Hello, In the databricks notebook which is provided by Microsoft training classes, when I tried to import => read a data (csv or json) like path = source + "/wikipedia/pagecounts/staging_parquet_en_only_clean/" files =…
asked 2023-07-20T13:58:55.4033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 91%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
Catherine 刘
30
Reputation points
edited a comment 2024-06-29T15:40:00.8866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 73%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECS%3C/text%3E%3C/svg%3E)
Choi, Seung-Rak
0
Reputation points
2 answers
One of the answers was accepted by the question author.
databricks cluster sizing
Hey, how to calculate cluster core and workers node of 10gb data load every 2 hours ... what is the calculation behind this
0 answers
How can I remove the `sample` catalog from Azure Databricks Workspaces?
All Azure Databricks Workspaces come with a sample catalog owned by Databricks that I cannot seem to remove or hide. I have tried dropping it, and I have also tried revoking and denying permissions on the catalog, but I keep receiving an error that is…
asked 2023-10-31T19:35:30.61+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(281.6, 28.999999999999996%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETT%3C/text%3E%3C/svg%3E)
Tim Thein
5
Reputation points
commented 2024-06-28T04:06:26.55+00:00
PRADEEPCHEEKATLA-MSFT
83,891
Reputation points Microsoft Employee
1 answer
view in dataframe
hey, how we can create or replace view statement in spark sql in dataframe of databricks create or replace view as (select * from temp1)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Error while implementing File Notification Mode: com.microsoft.azure.storage.StorageException: This request is not authorized to perform this operation using this permission.
While implementing File Notification Mode in Autoloader, I get the following error. Has anyone faced the similar issue? Note: The Databricks Service Principal is having Contributor role to Storage account. com.microsoft.azure.storage.StorageException:…
asked 2024-06-21T06:16:38.9166667+00:00
Hiran Amarathunga
45
Reputation points
commented 2024-06-27T03:46:11.5833333+00:00
PRADEEPCHEEKATLA-MSFT
83,891
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Cross tenant AAD authentication for Purview & Databricks
Hi, I want to know if for Purview & Databricks, is it possible to authenticate with cross tenant AAD? That is to say, can users belonging to AAD in tenant1, be able to login to Purview & Databricks which are setup in tenant2? Thanks
asked 2024-06-20T03:44:04.3166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 18%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Amit Singh
60
Reputation points
accepted 2024-06-27T03:18:50.43+00:00
Amit Singh
60
Reputation points
0 answers
[INTERNAL_ERROR] The Spark SQL phase optimization failed with an internal error. You hit a bug in Spark or the Spark plugins you use.
"I am trying to extract data from Azure Cosmos DB using PySpark and I am getting the following error: Py4JJavaError: An error occurred while calling o700.save.: org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase optimization…
asked 2024-03-08T08:35:10.85+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 81%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENF%3C/text%3E%3C/svg%3E)
Nathalia Fernandez Rodrigues
0
Reputation points
commented 2024-06-26T16:39:36.0333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 69%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELP%3C/text%3E%3C/svg%3E)
Luce PHILIBERT
0
Reputation points
2 answers
Azure Data Bricks - User Doesn't have permission to perform this action while connecting to Azure Synapse Dedicate Pool
We are connecting Azure Synapse Analytics - Dedicated Pool using the PySpark Code that runs from Azure Data Bricks using SQL Authentication. While running, we are getting the below error when we use a user with db_datawriter and db_datareader…
asked 2024-06-07T11:26:00.4933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 97%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Praveen Sreeram
1
Reputation point
commented 2024-06-26T06:20:40.76+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 1%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
Praveen Kumar Sreeram
0
Reputation points
1 answer
vm cpu utilization
Hi, have 100 VMs in Azure portal's resource group for which i am running VM memory metrix(cpu usage) .. how can it will automatically recognized the new subscription of vm came so that it will show the cpu usage via loading adf or databricks pipeline
1 answer
Can we connect ADX to Databricks, without using App Registration.?
I am trying to Access ADX data into Databricks, but failing to configure connection between databricks and ADX. I don't have permissions to use App registration. Is there any way to do with managed identity or any other alternative.
asked 2024-06-25T12:25:47.8766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 10%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Ram Ahuja, Roma
0
Reputation points
answered 2024-06-25T13:27:06.6633333+00:00
PRADEEPCHEEKATLA-MSFT
83,891
Reputation points Microsoft Employee
1 answer
Serverless warehouse suddenly stops to start up.
Hey All. From today, suddenly we are getting below error while starting a serverless warehouse. Details for the latest failure: Error: Cluster launch timeout. Type: SERVICE_FAULT Code: K8S_DBR_CLUSTER_LAUNCH_TIMEOUT Warehouse details: Type:…
asked 2024-06-25T12:48:53.3833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 41%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBS%3C/text%3E%3C/svg%3E)
Bhargav Shir
0
Reputation points
answered 2024-06-25T13:14:17.5566667+00:00
PRADEEPCHEEKATLA-MSFT
83,891
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Connecting to databricks using .Net
Hello Team, Is it possible to connect to Databricks and perform CRUD operations on catalog schema tables and Delta tables using .NET? If so, what approach is needed to connect to Databricks using JDBC or ODBC? Are there any specific libraries in Visual…
.NET
.NET
Microsoft Technologies based on the .NET software framework.
3,561 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,039 questions
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 53%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENC%3C/text%3E%3C/svg%3E)
1 answer
How to know which type of service we need within azure databricks for our implementation
I need to know the different types of services available within databricks to implement my solution more cost efficiently. Is there any resource from azure we can reach out to whom we can explain our implementation and they can provide the list of…
asked 2024-06-21T09:22:35.21+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 85%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
Arun Kumar
0
Reputation points
edited a comment 2024-06-25T09:41:40.9366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
12,010
Reputation points Microsoft Vendor
1 answer
How can I migrate data bricks from one subscription to another subscription during cross subscription migration ?
I am unable to move data brick from the source subscription to the destination subscription during cross-subscription migration
asked 2024-06-19T19:20:06.0166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 66%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
Maaz Ahmed Nagori
0
Reputation points
commented 2024-06-25T08:26:08.2166667+00:00
PRADEEPCHEEKATLA-MSFT
83,891
Reputation points Microsoft Employee