Downsizing subnets that are associated with a databricks workspace
Hi, Is it possible to downsize the private and public subnets that currently have nothing attached to it but are use by a databricks workspace? Would this require the databricks cluster to be redeployed? Are there any extra steps needed so the…
Can we connect ADX to Databricks, without using App Registration.?
I am trying to Access ADX data into Databricks, but failing to configure connection between databricks and ADX. I don't have permissions to use App registration. Is there any way to do with managed identity or any other alternative.
Set cloudFiles.maxFileAge and cloudFiles.backfillInterval values in Autoloader
I'm using following in the autoloader options. .option("cloudFiles.maxFileAge", "90 days")\ .option("cloudFiles.backfillInterval", "1 day")\ Our data retention policy is 7 years. Shall I use maxFileAge as 7 years…
How to specify a custom catalog name for Azure Databricks Delta Lake Dataset in ADF
Hello, I am creating an Azure Databricks Delta Lake Dataset in ADF and I am only able to choose the database name that links to Databricks's hive_metastore. How can I specify a custom catalog name that I created in Databricks instead of…
Serverless warehouse suddenly stops to start up.
Hey All. From today, suddenly we are getting below error while starting a serverless warehouse. Details for the latest failure: Error: Cluster launch timeout. Type: SERVICE_FAULT Code: K8S_DBR_CLUSTER_LAUNCH_TIMEOUT Warehouse details: Type:…
How to solve Invalid SessionHandle error with Azure Databricks ?
I am applying a SQLDatabaseChain Chatbot model by using LangChain SQLDatabaseChain and GPT4. I first created this model on Azure Databricks notebook like this : import json import os import langchain import mlflow from mlflow.models import…
array in databricks
Hi , I got the 1st row in sorted order how can i get the view on row1,row2,row3 using array and how the reshuffling will happen 1,2,3,4,5 --- sorted order 5,1,2,3,4 - 1 4,5,1,2,3 - 2 3,4,5,1,2 - 3
Azure Databricks fails
Hello, In the databricks notebook which is provided by Microsoft training classes, when I tried to import => read a data (csv or json) like path = source + "/wikipedia/pagecounts/staging_parquet_en_only_clean/" files =…
databricks cluster sizing
Hey, how to calculate cluster core and workers node of 10gb data load every 2 hours ... what is the calculation behind this
How can I remove the `sample` catalog from Azure Databricks Workspaces?
All Azure Databricks Workspaces come with a sample catalog owned by Databricks that I cannot seem to remove or hide. I have tried dropping it, and I have also tried revoking and denying permissions on the catalog, but I keep receiving an error that is…
view in dataframe
hey, how we can create or replace view statement in spark sql in dataframe of databricks create or replace view as (select * from temp1)
Error while implementing File Notification Mode: com.microsoft.azure.storage.StorageException: This request is not authorized to perform this operation using this permission.
While implementing File Notification Mode in Autoloader, I get the following error. Has anyone faced the similar issue? Note: The Databricks Service Principal is having Contributor role to Storage account. com.microsoft.azure.storage.StorageException:…
Cross tenant AAD authentication for Purview & Databricks
Hi, I want to know if for Purview & Databricks, is it possible to authenticate with cross tenant AAD? That is to say, can users belonging to AAD in tenant1, be able to login to Purview & Databricks which are setup in tenant2? Thanks
[INTERNAL_ERROR] The Spark SQL phase optimization failed with an internal error. You hit a bug in Spark or the Spark plugins you use.
"I am trying to extract data from Azure Cosmos DB using PySpark and I am getting the following error: Py4JJavaError: An error occurred while calling o700.save.: org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase optimization…
vm cpu utilization
Hi, have 100 VMs in Azure portal's resource group for which i am running VM memory metrix(cpu usage) .. how can it will automatically recognized the new subscription of vm came so that it will show the cpu usage via loading adf or databricks pipeline
Connecting to databricks using .Net
Hello Team, Is it possible to connect to Databricks and perform CRUD operations on catalog schema tables and Delta tables using .NET? If so, what approach is needed to connect to Databricks using JDBC or ODBC? Are there any specific libraries in Visual…
How to know which type of service we need within azure databricks for our implementation
I need to know the different types of services available within databricks to implement my solution more cost efficiently. Is there any resource from azure we can reach out to whom we can explain our implementation and they can provide the list of…
How can I migrate data bricks from one subscription to another subscription during cross subscription migration ?
I am unable to move data brick from the source subscription to the destination subscription during cross-subscription migration
I am unable to move data brick from the source subscription to the destination subscription during cross-subscription migration
I am unable to move data brick from the source subscription to the destination subscription during cross-subscription migration
Databricks not sending audit logs to event hub
Hi, I'm trying to push all the logs from databricks using the diagnostic tool to event hub but is not working, it didn't push anything. I'm using the root access policy and also already created the eventhub name, what else I'm probably missing? Thanks in…