Azure Databricks

1 answer

Azure Databricks fails

Hello, In the databricks notebook which is provided by Microsoft training classes, when I tried to import => read a data (csv or json) like path = source + "/wikipedia/pagecounts/staging_parquet_en_only_clean/" files =…

asked

Catherine 刘 35

edited a comment

Choi, Seung-Rak 0

2 answers

databricks cluster sizing

Hey, how to calculate cluster core and workers node of 10gb data load every 2 hours ... what is the calculation behind this

asked

Vineet S 750

accepted

Vineet S 750

0 answers

How can I remove the `sample` catalog from Azure Databricks Workspaces?

All Azure Databricks Workspaces come with a sample catalog owned by Databricks that I cannot seem to remove or hide. I have tried dropping it, and I have also tried revoking and denying permissions on the catalog, but I keep receiving an error that is…

asked

Tim Thein 5

commented

PRADEEPCHEEKATLA-MSFT 88,716 Microsoft Employee

1 answer

view in dataframe

hey, how we can create or replace view statement in spark sql in dataframe of databricks create or replace view as (select * from temp1)

asked

Vineet S 750

commented

Harishga 5,990 Microsoft Vendor

1 answer

Error while implementing File Notification Mode: com.microsoft.azure.storage.StorageException: This request is not authorized to perform this operation using this permission.

While implementing File Notification Mode in Autoloader, I get the following error. Has anyone faced the similar issue? Note: The Databricks Service Principal is having Contributor role to Storage account. com.microsoft.azure.storage.StorageException:…

asked

Hiran Amarathunga 95

commented

PRADEEPCHEEKATLA-MSFT 88,716 Microsoft Employee

1 answer

Cross tenant AAD authentication for Purview & Databricks

Hi, I want to know if for Purview & Databricks, is it possible to authenticate with cross tenant AAD? That is to say, can users belonging to AAD in tenant1, be able to login to Purview & Databricks which are setup in tenant2? Thanks

asked

Amit Singh 60

accepted

Amit Singh 60

0 answers

[INTERNAL_ERROR] The Spark SQL phase optimization failed with an internal error. You hit a bug in Spark or the Spark plugins you use.

"I am trying to extract data from Azure Cosmos DB using PySpark and I am getting the following error: Py4JJavaError: An error occurred while calling o700.save.: org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase optimization…

asked

Nathalia Fernandez Rodrigues 0

commented

Luce PHILIBERT 0

1 answer

vm cpu utilization

Hi, have 100 VMs in Azure portal's resource group for which i am running VM memory metrix(cpu usage) .. how can it will automatically recognized the new subscription of vm came so that it will show the cpu usage via loading adf or databricks pipeline

asked

Vineet S 750

commented

Vineet S 750

1 answer

Connecting to databricks using .Net

Hello Team, Is it possible to connect to Databricks and perform CRUD operations on catalog schema tables and Delta tables using .NET? If so, what approach is needed to connect to Databricks using JDBC or ODBC? Are there any specific libraries in Visual…

asked

Nagesh CL 646

commented

Nagesh CL 646

1 answer

How to know which type of service we need within azure databricks for our implementation

I need to know the different types of services available within databricks to implement my solution more cost efficiently. Is there any resource from azure we can reach out to whom we can explain our implementation and they can provide the list of…

asked

Arun Kumar S 0

edited a comment

Smaran Thoomu 14,875 Microsoft Vendor

1 answer

How can I migrate data bricks from one subscription to another subscription during cross subscription migration ?

I am unable to move data brick from the source subscription to the destination subscription during cross-subscription migration

asked

Maaz Ahmed Nagori 0

commented

PRADEEPCHEEKATLA-MSFT 88,716 Microsoft Employee

0 answers

I am unable to move data brick from the source subscription to the destination subscription during cross-subscription migration

asked

Maaz Ahmed Nagori 0

commented

Bhargava-MSFT 30,816 Microsoft Employee

2 answers

Databricks not sending audit logs to event hub

Hi, I'm trying to push all the logs from databricks using the diagnostic tool to event hub but is not working, it didn't push anything. I'm using the root access policy and also already created the eventhub name, what else I'm probably missing? Thanks in…

asked

Danny AMAYA 0

commented

PRADEEPCHEEKATLA-MSFT 88,716 Microsoft Employee

1 answer

logic app to save data in adls gen2

Hi all, I would like to utilize logic app to write into adls gen2. After the file is saved in adls gen2, it should be able to read by databrick as delta table. May I know is there a specific file format to do it or how it can be achived? thanks

asked

Yang Chow Mun 121

accepted

Yang Chow Mun 121

1 answer

Restricting files/folders to upload into External volumes in Azure databricks UC workspace

Hello Team, Is there a way to restrict the files or folders to upload/download from external volumes same like DBFS? Is there any option to disable the uploading files/folders feature in external volumes of azure databricks workspace with Unity Catalog.…

asked

Ashwini Gaikwad 130

answered

Bhargava-MSFT 30,816 Microsoft Employee

1 answer

How to restore myspace and files created with Student subscription?

Hello, I had a Student Azure subscription in 2022-2023 and created some resources and lots of files in Databricks. then my subscription expired and I could not renew it though I am still a student. I created a paid subscription today in hope to see my…

asked

Maria 71

accepted

Maria 71

2 answers

How to setup modern Arcitechure for Small/Medium Business?

Currently we're using the following setup which is slow to process the data and is slow on the power bi side: Azure VM for third parties to upload via sftp C# script to ETL data to azure sql server and move files to ADLS Gen2 Power BI report pulling…

asked

Jordan 25

accepted

Jordan 25

0 answers

How can I clear the cache in my Databricks Cluster

Ive tried many articles and links without success. Has anyone managed to successfully clear the cache on their Databricks cluster? This causes many issues as my memory is near its peak, meaning my notebooks often crashes. Any help or advised would be…

asked

richardthomas66@hotmail.com 0

commented

Bhargava-MSFT 30,816 Microsoft Employee

1 answer

While running databricks migrate pipeline facing an issue with invalid configuration for storage account key

Hello Team, I am trying to run the yaml pipeline for azure databricks migration from Non UC workspace to UC workspace for reference this is the rep https://github.com/databrickslabs/migrate so while exporting the hive metastore, I am running into error…

asked

Ashwini Gaikwad 130

answered

Bhargava-MSFT 30,816 Microsoft Employee

1 answer

How to get the public IP range of my Azure Databricks cluster? I Need to whitelist this IP range with a host that I want to connect and get data from.

How to get the public IP range of my Azure Databricks cluster? I Need to whitelist this IP range with a host that I want to connect to and get data from.

asked

Makarand Nidhalkar 20

accepted

Makarand Nidhalkar 20

Filter

Content

2,162 questions with Azure Databricks tags