Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,409 questions
1 answer
One of the answers was accepted by the question author.
Have previously been able to map a received dataset to data store. But have been unable to do so today - OK button does not become enabled after selecting a data store folder. Has something changed?
I received some (Azure Data Lake Storage Gen2 Folder) datasets from an outside (partner) organization through a data share invite. Previously was able to map the dataset to a folder in one of my (Azure Blob Storage) data stores, but today the OK button…
asked 2024-05-23T23:29:39.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 79%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERK%3C/text%3E%3C/svg%3E)
Richard Krueger
25
Reputation points
accepted 2024-06-19T13:29:17.7866667+00:00
Richard Krueger
25
Reputation points
2 answers
One of the answers was accepted by the question author.
How to setup modern Arcitechure for Small/Medium Business?
Currently we're using the following setup which is slow to process the data and is slow on the power bi side: Azure VM for third parties to upload via sftp C# script to ETL data to azure sql server and move files to ADLS Gen2 Power BI report pulling…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 42%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
1 answer
While running databricks migrate pipeline facing an issue with invalid configuration for storage account key
Hello Team, I am trying to run the yaml pipeline for azure databricks migration from Non UC workspace to UC workspace for reference this is the rep https://github.com/databrickslabs/migrate so while exporting the hive metastore, I am running into error…
asked 2024-06-17T15:46:25.1033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 24%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashwini Gaikwad
110
Reputation points
answered 2024-06-18T14:55:54.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
28,526
Reputation points Microsoft Employee
0 answers
Authentication failed when accessing dataset in compute instance using its MSI via AzureML SDK v2
The dataset can be previewed in AML workspace when the associated datastore is created using identity-based auth with workspace managed identity. I have the same user managed identity assigned to workspace and compute instance. But it only failed in…
asked 2024-06-14T22:27:10.02+00:00
Shiyao Li
0
Reputation points Microsoft Employee
commented 2024-06-18T13:56:50.44+00:00
Amira Bedhiafi
18,341
Reputation points
1 answer
One of the answers was accepted by the question author.
Data Lake as a storage/database for Express Angular Application?
So currently I'm using SQL server for our structured data, client uploads a file which has a minimum of a million of records, which gets uploaded to blob storage and then those million of records gets inserted into different tables in SQL. What I want to…
asked 2024-06-12T18:28:04.58+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 46%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJN%3C/text%3E%3C/svg%3E)
Jayesh Nathani
40
Reputation points
accepted 2024-06-17T07:09:35.5+00:00
Jayesh Nathani
40
Reputation points
0 answers
Secured kusto connection during data ingestion from fabric notebook to lakehouse.
Hi team Im looking for SN+I Auth or secured auth for kusto in fabric notebook.
asked 2024-06-12T18:10:15.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 96%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EUD%3C/text%3E%3C/svg%3E)
Uday Dinakar Sutar
0
Reputation points
commented 2024-06-17T04:45:12.3033333+00:00
Sumarigo-MSFT
44,891
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Self-hosted machine unable to access Data Lake Storage account when running a pipeline using Synapse
Hello - I need your help again: Here's the story: I have azure synapse workspace I have created managed private end point created for ADLS - working fine I have create private endpoint created for ADLS - working fine ADLS have set the public access…
asked 2024-06-07T08:58:57.6633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 88%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EXM%3C/text%3E%3C/svg%3E)
Xhevahir Mehalla
140
Reputation points
edited the question 2024-06-14T07:57:57.3666667+00:00
PRADEEPCHEEKATLA-MSFT
84,051
Reputation points Microsoft Employee
7 answers
Error trying to validate storage account name while creating a new synapse workspace
I am unable to create a new data lake storage (gen2) account name in the basic tab for create Azure Synapse Workspace. The error I get is - " There was an error trying to validate storage account name. please try again ". The error message is…
asked 2022-08-03T07:01:54.587+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(252.8, 2%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
Jozure
491
Reputation points
commented 2024-06-14T07:53:15.7966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(214.4, 86%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJK%3C/text%3E%3C/svg%3E)
Joy Kimaiyo
0
Reputation points
2 answers
Copy LOB based data from DB2 to Azure Data Lake Gen2 storage using ADF
Hi team, I was trying to work on a scenario to copy data from a table in db2 database having a column with datatype as LOB(CLOB) to Azure Data Lake Gen2 storage container using Azure Data Factory. However, its taking a considerable amount of time( a day)…
asked 2024-04-21T03:40:26.3+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 48%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDM%3C/text%3E%3C/svg%3E)
Debajit Mitra
0
Reputation points
commented 2024-06-14T03:39:35.9133333+00:00
Debajit Mitra
0
Reputation points
1 answer
One of the answers was accepted by the question author.
org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename temp file
[Repeat Question due to old thread] We have built a streaming pipeline with spark autoloader. Source Folder is a azure blob container. We've encountered a rare issue (could not replicate it). Below is the exception…
asked 2024-06-07T01:40:56.47+00:00
Hiran Amarathunga
45
Reputation points
accepted 2024-06-14T02:26:30.36+00:00
Hiran Amarathunga
45
Reputation points
1 answer
Run Databricks notebook from ADF - error to find azure module to save the data in blob storage
Hi Guys, The requirement is - Call Rest API, read the records in jsonlines format and load into table in Azure SQL server. I used Databricks to read the jsonlines from Open API using Python script. It can read and keep the data into a file in Azure blob…
asked 2024-06-12T12:39:10.24+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 35%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sarmistha Sarkar
0
Reputation points
answered 2024-06-13T05:37:40.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
7,835
Reputation points Microsoft Vendor
1 answer
unable to connect adls gen storage from Purview
unable to connect adls gen storage from Purview gettting following error ADLS Gen2 operation failed for: Storage operation '' on container 'bloblblob' get failed with 'Operation returned an invalid status code 'Forbidden''. Possible root causes: (1).…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 64%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENR%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
How to assign a Static Public IP to Azure Storage Account
I have a storage account in Azure and I need to assign a static public IP because we use a dedicated internet link that requires the endpoint to have a public IP address. The dynamic IP addresses from Azure change over time, so I need a fixed IP. Is…
asked 2024-06-05T18:33:50.5833333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 80%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Steven Axel Williams
20
Reputation points
accepted 2024-06-10T12:53:53.5933333+00:00
Steven Axel Williams
20
Reputation points
1 answer
One of the answers was accepted by the question author.
My Dev, test, prod environments are in different resource groups of same subscription. How do I create a devops pipeline in this case?a DevOps pipeline to deploy a
Hi, My dev, test and prod environments are in different resource groups of the same subscription. I am involved in a data engineering project where I will be using primarily below resources - ADLS - data storage ADF - Orchestration Azure Databricks - QC…
asked 2024-05-14T19:23:38.4366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 90%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Shashwat Tiwary
40
Reputation points
accepted 2024-06-10T09:53:29.01+00:00
Shashwat Tiwary
40
Reputation points
1 answer
Can i copy data from ADL to Neo4j graph database using ADF
Hello, I want to load data from ADL to neo4j database using ADF. Can I do that using ADF?? If yes, what are the different options I have? My first priority is I want to use ADF activities if that's not possible i can use python coding.
asked 2024-06-06T14:27:54.4+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 73%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Siddartha Reddy Jammula
20
Reputation points
commented 2024-06-10T07:33:17.0466667+00:00
phemanth
7,835
Reputation points Microsoft Vendor
0 answers
Calling Power BI reports from ADF
Hello, I have one report with multiple client folders this is the ADLS gen 2 folder location for one client and data source connection: AzureStorageDataLake for different files: …
asked 2024-06-06T11:57:54.79+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 1%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBA%3C/text%3E%3C/svg%3E)
Bodke, Anil
0
Reputation points
commented 2024-06-10T03:40:58.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Harishga
5,425
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
How to Create Delta Table in Azure Synapse Analytics with Id Auto Increment Identity Column ?
I have created the Delta Lake Delta tables In ADLS using Synapse Notebook and in that table, I want to add an identity column (Auto increment 1,1) but I am not able to create the same, Below is my Create table script and error which i am facing. Table…
asked 2024-06-04T06:11:14.4766667+00:00
Vedant Desai
651
Reputation points
accepted 2024-06-07T09:52:14.62+00:00
Vedant Desai
651
Reputation points
1 answer
Building a chatbot via Azure
Hi Azure community! I want to build a chatbot for a company. It will be able to answer questions using the company data sources, including csv and pdf files. For example, if I ask the chatbot to summarize some csv financial reports, it will return an…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 92%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ET%3C/text%3E%3C/svg%3E)
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 61%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
2 answers
I am unable to mount containers using databricks and storage gen 2 ?
what is the issue?
asked 2024-06-05T15:57:33.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 14.000000000000002%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
smriti das
0
Reputation points
answered 2024-06-06T12:19:54.16+00:00
Luis Arias
5,901
Reputation points
2 answers
The synapse link is not able to write the data in Azure data lake gen 2 storage account
Hi Team, The synapse link is not able to write the data in Azure data lake gen 2 storage account. I am getting the following error message. {"code":"AuthorizationFailed","message":"The client…
asked 2024-05-24T08:27:24.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 60%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Abhishek Sajjan
0
Reputation points
commented 2024-06-06T02:01:47.0166667+00:00
AnnuKumari-MSFT
31,806
Reputation points Microsoft Employee