How to pass Wild Card File Name in ADF Databricks Activity
HI , I am working in for each Activity in the ADF , in For each we have copy activity and then we pass the table name to databricks as parameter . Copy activity supports wild card paths @Anonymous (item().TABLE_NAME,'*')---- its picking…
Error when triggering Databricks notebook via datafactory while reading XML file
I am reading xml file in spark databricks and below is the command I am using to read the file. val readxml =…
Databricks Cluster (GC issue)
I have the below cluster setting. Currently, I am getting data every 30 Milliseconds. So for 1 hr, I am getting around 1 GB of Data. I am running this job every 5 hrs but it is throwing me GC error, Please help me. Also attached my current…
What python libraries have databricks pre-installed and do not need to be installed in the cluster?
What python libraries have databricks pre-installed and do not need to be installed in the cluster?
![](https://techprofile.blob.core.windows.net/images/5h6uyi6AYkmj7Mzq7BbG4w.png?8D8569)
![](https://techprofile.blob.core.windows.net/images/5h6uyi6AYkmj7Mzq7BbG4w.png?8D8569)
databricks/simba jdbc driver - warnings/error messages
I've setup a sql ide (workbench/j) for my sparksql databricks querying - writing code in a web page textarea control is just not for me... https://video2.skills-academy.com/en-us/azure/databricks/integrations/bi/workbenchj i'm getting this…
parsing json databricks / python
I am very beginner in databricks and python, so it maybe basics for you but for me it is still new. I am trying to parse the json to get all childitems form "valid" and "date_of_creation" sections as columns in table which I will…
Azure databricks create Azure Key vault secret scope failed
I am the contributor of the key vault , and I am the admin of the databricks workspace , and I have the following error while creating the secret scope. I know after click create , it would try to add "AzureDatabricks" service…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Log cluster name
When i run any notebook I want to log by using which cluster the notebook was executed. Is their any way to get the cluster name using python in databricks
User level auditing in Databricks Azure
We are looking into collecting audit logs for Azure Databricks. We have enabled diagnostic settings for the databricks workspace which sends log data to log analytics, it contains the monitoring audit logs, as in high level - who created which…
Databricks Spark connection issue over Simba JDBC
I am trying to connect Spark Databricks from PERL code over Simba JDBC (Databricks recommended way) .For ref this is the JDBC driver:…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
I am not getting all the files using listFiles(basep: String, globp: String) in Azure Databricks
Hi, I have a question regarding this article https://video2.skills-academy.com/en-us/azure/databricks/kb/data/list-delete-files-faster I have a directory in my ADLS in which I have some folders and inside folders, I have some parquet files. I was using…
pass variable in drop down from another notebook
Hi, I am trying to pass a variable from notebook A to Notebook B which has drop down widget list. However getting below error. error message : select sequence must include X Notebook A has below variable defined. parameters_global['Source'] =…
Attribute error when mounting blob
It is my second day with databricks and Python. So forgive me if my question sounds silly. When mounting a blob storage container I got "Attribute error'. I previously created scope credentials and SAS token (for container), which is stored in…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
How to add extended infrastructure IP addresses in nsg for databricks ?
Hi Team, Could you please help me to add extended infrastructure IP addresses to databricks nsg ? MS tracking id:1TW4-LDG Thanks Sidhartha
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Cannot mount Azure VM(windows server) file share on Azure Databricks instance.
Hi, I wanted to mount windows file share(from Windows server VM in Azure) on a databricks instance. To test it I followed these steps. I created a Windows VM and a Linux VM in azure. I created VNet-peering between these two VMs. After…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
azure databricks list and delete screte scopes
there is command line databricks secrets list-scopes to list scopes however , are there URL method to list and delete secret scopes like https://adb-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.azuredatabricks.net#secrets/createScope <<< to…
Pass different parameter to same notebook databricks
I have a notebook in python which provides profiling output results for the table . I want to schedule this notebook. For example the notebook name is Profiling notebook and i have table A,B,C,D.... When i schedule this notebook do i have to…
pyspark data frame for json
hello all , i am using MS-Azure databricks platform for python+spark , cluster = spark 3.1.1 kindly tell me how can i import json file in it syntax that i have used now: df =…
How to add custom validation on excel/csv/sql ?
I want to add a custom validation like column which contain a date that follows a specific format like mm:yyyy and number with specific range in Azure data factory
How to round and convert the nested values from dictionary to integer in Python?
enc = {'Category': {'a1': 1885.4403010060062, 'a2': 1072.0176453884112, 'a3': 1448.579447138836, 'a4': 1471.5656350474158, 'a5': 1232.3527616505146, 'a6': 891.9808067951503, 'a7': 1378.9613179304858}} I have tried this. for key, value in…