How to get list of child files/directories having parent DataLakeDirectoryClient class instance
Hello All! I have to scan whole data lake file system. Having code like: PagedIterable<PathItem> pItems = ((DataLakeFileSystemClient)prmParent).listPaths(); for( PathItem pItem : pItems ){ if pItem.isDirectory() ){ …
Unable to add service principle, groups to the $logs container in adls2
Recently enabled storage analytics on ADLS Gen2 storage account.I can see the $logs container and the logs are writing to this on an hourly basis. But when I'm trying to add service principal to this container getting permission denied. I'm able to…
Data Masking in Azure data Factory
We are using Azure Data factory to move data from Source like Azure SQL and Azure Postgres to destination as Azure data lake.There is some sensitive data which needs to be masked. Is it possible to have data masking in Azure Data factory during…
Getting Data from azure datalake through api
Hi Team, Looking for any api available to get data from azure data lake directly ,with out using any data transformation tools like pipelines/dataflow. Csv or Json files sitting in datalake storage folders, so the same data or files can be fetched…
My partitions are returning inconsistent results (Mapped JSON file data)
I have a set of tables based on a number of JSON files taken from a data lake v2 container. I'm building an external table using derived data from JSON mapping, then using derived columns based on those columns to generate an identifier. When I go to…
Azure Data Lake Gen2 PUT authorization
I'm trying to create a Shared Access Signature client side in my Node app. The reason being that I do not want to stream files through my app. I want the user to be able to upload a file to my Azure Data Lake Gen2 Blob Storage container directly. I have…
Writing parquet file throws…An HTTP header that's mandatory for this request is not specified
I have two ADLSv2 storage accounts, both are hierarchical namespace enabled. In my Python Notebook, I'm reading a CSV file from one storage account and writing as parquet file in another storage, after some enrichment. I am getting below error when…
Where to store delta-files
Hey guys! I have been wondering about the below question for a while and I hope you can help me get a good night sleep again. Question: When you are working in Azure DataBricks to do transformations, you can save the results using a delta format.…
Error Setting Access Rights on ADL (Gen1) User Folders During HDInsight Cluster Creation
Hello All The issue occurs when the cluster is created. One of the last operations that Ambari performs are 'Post user creation hook for 1 users'. This fail with the attached errors. Here is some more information: It's an ESP cluster -- we…
![](https://techprofile.blob.core.windows.net/images/-kBKepa7U0a5d4kNlSs0Pg.png?8D81DE)
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Can't connect to our ADLS gen2 in Powerbi Online (scheduled refresh)
Hi all, We have recently set up a ADLS gen 2 and we want to update our reports automatically. In the powerBI desktop version everything works, but online I get the following error: Failed to update data source credentials: The credentials provided…
Mounting entire ADLS on azure databricks
Hi, I want to mount entire ADLS storage account on databricks. I've checked in documents I can mount a single filesystem at a time but I want to mount entire ADLS on databricks. I've around 70 containers in my ADLS and I want to mount all of them…
Adls Availability Alerts
Hi Team, I have set alerts on my adls gen2 such as if availability is less than 80 percent it should send alert notification. My question is this availability is Microsoft responsibility, right ? how to find out root cause of why this availability…
Copy Different type of file from Gen1 Azure lake to Azure Gen2 lake with attribute( like last updated)
I need to migrate all my data from Azur data lake Gen1 to Lake Gen2. In my lake we have different types of file mixed (.txt, .zip,.json and many other). We want to move them as-it-is to GEN2 lake. Along with that we also want to maintain last updated…
How to delete specific data from files stored in Azure Data Lake
I'm migrating data sets to be stored in ADLS. After certain period I want to delete data within this data set that is more than 3yrs old. How can I do that? I do not want to delete the file, but specific data within the files.
Data Factory V2 Storage Account not Refreshing - Because it thinks its blob Storage
I have created a Storage Account set to be Azure Data Lake V2 by setting the hierarchical namepace. And I used Azure Storage Explorer to add some files into the blob Container (Which is an Azure datalake gen2) I'm now creating a Data Lake V2 Linked…
I'm trying to source data in a comma-delimited file from an Azure Gen2 Storage account using Flexible File Source task. The connection succeeds but Preview of data is blank.
I'm trying to pull data from ADLS Gen2 using the SSIS Flexible File Source task. I'm getting blank results when trying to view the data in the Columns tab of the dialog.
![](https://techprofile.blob.core.windows.net/images/iBVLlv5_AwAAAAAAAAAAAA.png?8D8491)
Error occurs at source transformation when trying to source csv file from folder in data lake storage gen2 while debugging the pipeline
While using mapping data flow activity, In source transformation I am trying to read a csv file from a folder in Azure data lake gen2. I am able to preview the data in "Data Preview" in source transformation but when trying to debug the…
Write Access for Azure Data Lake Gen 2 Container
Hi, I have an Azure Data Lake Gen 2 Storage Account. I have created multiple containers inside it say A , B & C. Now , I want to control access for files & folders for respective containers. I have tried giving the following access - …
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Can't access txt file with json format from Data Lake Gen2 inside a Data Flow
I have stored a txt file with json format (coming from a MongoDB) in my Data Lake Storage Gen2 and inside my Data Flow I would like to access that file, convert it into a column structure, transform the data and shift it to my Azure SQL DB. Currently I…
Common data service is not able to connect to Azure data lake store Gen 2 when "Selected Networks" is configured in firewall.
Issue: Common data service is not able to connect to Azure data lake store Gen 2 when "Selected Networks" is configured in firewall. Description: Both environment and storage account are in west europe region. However, the reason connection…