Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,410 questions
0 answers
File length exception while uploding file to Gen2 lake
I am using java sdk to copy file from S3 to Azur data lake Gen2. S3Object fullObject = s3Client.getObject("bucket", "folder/file.txt"); S3ObjectInputStream strm = fullObject.getObjectContent(); …
asked 2020-10-17T10:48:27.717+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 62%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBK%3C/text%3E%3C/svg%3E)
Bhuvnesh Kumar
26
Reputation points
commented 2020-12-08T11:15:25.703+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
0 answers
Getting error when connecting to Azure Data Lake Storage Gen 2 ABFSS using spark from local
Error is Exception in thread "main" java.io.IOException: No FileSystem for scheme: abfss Log is below 20/11/10 22:58:18 INFO SharedState: Warehouse path is 'file:/C:/sparkpoc/spark-warehouse'. 20/11/10 22:58:18 INFO…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 96%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKJ%3C/text%3E%3C/svg%3E)
1 answer
Issues to write csv files into my storage account using ADF
Hello everyone, Im trying to load csv files in my Azure Storage Account but Im getting some issues that Im not sure what they are exactly and Id like to ask some clarifications. When I build a pipeline and a few blocks to lookup a table list…
asked 2020-11-20T19:30:33.3+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 41%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFR%3C/text%3E%3C/svg%3E)
Felipe Regis E Silva
46
Reputation points
commented 2020-12-02T20:02:56.673+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
2 answers
ADHOC Pipeline execution based on date range(startdate and endate)
Hi Team, I want to execute the pipeline based on start date and enddate and it should load the data into ADLS in the incremental way. for example start date=2020-11-14 and enddate=2020-11-17 then it should load the file into data lake like…
asked 2020-11-18T15:05:47.357+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 98%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPC%3C/text%3E%3C/svg%3E)
pankaj chaturvedi
86
Reputation points
commented 2020-12-01T12:24:44.4+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,781
Reputation points Microsoft Employee
2 answers
Azure Functions “The operation has timed out.” for timer trigger blob archival
I have a Python Azure Functions timer trigger that is run once a day and archives files from a general purpose v2 hot storage container to a general purpose v2 cold storage container. I'm using the Linux Consumption plan. The code looks like this: …
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 26%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
2 answers
One of the answers was accepted by the question author.
compression FILESTREAM SQL Server Express 2014
Supprimer l'alerte|Modifier|Type de modification Question Vous ne pouvez pas voter pour votre propre billet. 0 Hello I'm Using FILESTREAM to Store BLOBs in the NTFS File System in SQL Server Express 2014, the problem is when I store the image…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(124.80000000000001, 56.00000000000001%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
1 answer
Built in Policy is Not working as expected. Policy Name is 'Require encryption on Data Lake Store accounts' with Deny effect
Policy is not working as expected because if we already creates a data lake storage accounts but the resource showing in compliant state even the encryption and firewall properties are not enabled. even i created and tried with Audit and AINE effect but…
asked 2020-11-12T06:24:40.873+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 26%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Manikanta
1
Reputation point
answered 2020-11-26T13:00:36.003+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 39%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
SwathiDhanwada-MSFT
18,466
Reputation points
1 answer
Convert from Datalake Json Format to TSI
I have a lot of data (50GB) in DataLakeGen2 as JSON files and would like to use them in TSI. How do I best convert the data so that I have them available in Parquet format for tsi?
asked 2020-11-16T17:07:55.237+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(313.6, 19%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
Cl.Mo
1
Reputation point
commented 2020-11-25T11:12:05.18+00:00
António Sérgio Azevedo
7,666
Reputation points Microsoft Employee
0 answers
move xml file from shared drive and injest to data lake
Dear Fellows, I need some help with Microsoft azure, I have a file keep coming in shared folder, as soon as the file lands in shared folder it needs to be moved to data lake, kindly advise how can I do it or if there is any tutorial. Regards
asked 2020-11-16T21:24:53.77+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 32%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
HuuM
46
Reputation points
commented 2020-11-20T19:01:16.413+00:00
Saurabh Sharma
23,781
Reputation points Microsoft Employee
1 answer
Revoke existing SAS token for ADLSgen2
How can i revoke the existing SAS token generated for Azure datalake storage . As of now , I only see option of changing the Account key.
asked 2020-11-10T05:54:06.69+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 45%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
Arvind Kumar
1
Reputation point
commented 2020-11-17T18:07:22.13+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,051
Reputation points
1 answer
copying data from azure file share to synapse
Hi , I have one requirement like - 1)my users will load some data to azure file share that is mounted on there machine 2) now i want to copy that data from azure file share to container in storage account 3)from that container i want to serve data to…
asked 2020-11-04T07:49:56.757+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 28.000000000000004%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERB%3C/text%3E%3C/svg%3E)
Rohit Boddu
461
Reputation points
commented 2020-11-16T22:08:03.217+00:00
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Getting TlsVersionNotPermitted error when SSIS package executing from ADF
Hi, I have created one SSIS package which is reading data from Storage account and putting that into Azure DB. This is working perfectly well when executing from Visual Studio. But when I am triggering it after deployment in ADF, getting below error. …
asked 2020-11-12T18:01:00.723+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 30%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
Vikas
21
Reputation points
commented 2020-11-16T18:10:09.6+00:00
MartinJaffer-MSFT
26,051
Reputation points
1 answer
One of the answers was accepted by the question author.
Auto refresh regular Synapse dedicated SQL pool tables
We're using Synapse Analytics with dedicated SQL pool. We try to create some regular tables from some parquet files from Azure Data Lake Storage Gen2 to use materialzied view. Is there anyway we can have it auto refreshed when new data coming into the…
asked 2020-11-09T11:18:09.713+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 49%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
pmnguye2
21
Reputation points
commented 2020-11-11T21:00:41.603+00:00
MartinJaffer-MSFT
26,051
Reputation points
0 answers
HD Insights - converting the Aggregated Logs output into ADLSGen2 into Text format
Can someone share a reference to systematically output logs from the HDinsights application runs into a readable format -- for the stdout. Today when running a Spark application through HdInsights (triggered by Livy API), after the application finishes…
asked 2020-10-22T17:07:06.273+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 46%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBC%3C/text%3E%3C/svg%3E)
Brendan Cassidy
26
Reputation points Microsoft Employee
commented 2020-11-11T18:23:27.957+00:00
Brendan Cassidy
26
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
テナントを跨いだPowerBIの活用について
弊社にはazureのテナントが二つあります。 １のテナントにはPowerBIProライセンスがあり、2つ目のテナントのDateLake内にはPowerBIで可視化させたいデータがあります。 2つ目のデータを可視化させるためにオンプレミスデータゲートウェイを活用すれば1つ目のテナントにあるPowerBIを使用できますでしょうか。 宜しくお願い致します。 Convert Japanese to English: Title: Utilization of Power BI across…
asked 2020-11-09T06:19:11.863+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 65%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWS%3C/text%3E%3C/svg%3E)
Watanabe Shoya (渡邉 頌也)
21
Reputation points
accepted 2020-11-09T07:55:08.653+00:00
Watanabe Shoya (渡邉 頌也)
21
Reputation points
1 answer
One of the answers was accepted by the question author.
Use ADF to load a file in TEAMS into ADL
Is there a way to load an xlsx file in teams into Azure Data Lake using Azure Data Factory? Thanks geert
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,410 questions
Microsoft Teams
Microsoft Teams
A Microsoft customizable chat-based workspace.
9,503 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,024 questions
asked 2020-11-05T17:56:04.897+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 37%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGV%3C/text%3E%3C/svg%3E)
geert vanhove
116
Reputation points
accepted 2020-11-06T09:14:21.173+00:00
geert vanhove
116
Reputation points
0 answers
Azure Data Lake Gen2 - Use Case Advice
I am collecting weather data (history and forecast) from a third part web service. Since there will be a lot of data, and it will not have high use, I was planning to use Azure Data Lake Gen2 with blob storage, and storing the data in JSON files. My…
asked 2020-10-30T14:58:46.347+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 27%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEM%3C/text%3E%3C/svg%3E)
Eric Maibach
1
Reputation point
commented 2020-11-05T18:02:15.79+00:00
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
2 answers
Error while loading data from Azure Data Lake Gen2 to Azure Synapse Analytics using Azure Data Factory
Hi All, I am getting an error while loading data from Azure Data Lake Gen2 to Azure Synapse Analytics using Azure Data Factory. I am unable what exactly the error is? What permission has to be provided? How?. This error occurs when I use Polybase…
asked 2020-08-16T09:00:27.563+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 55.00000000000001%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EI%3C/text%3E%3C/svg%3E)
ImranMaqsood
6
Reputation points
answered 2020-11-04T04:57:11.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 77%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERH%3C/text%3E%3C/svg%3E)
Rudolph Harms
26
Reputation points
1 answer
Data Factory Copy activity failing with Error code 2200 - Operation failed as split count exceeding upper bound of 1000000
Hi , I am copying large data from ADLS gen 1 to ASDW(Azure synapse Analytics) using Data Factory (polybase set to yes and mentioned blob storage settings). The Source data is *.parquet format and is partitioned. Copy activity fails. In first attempt…
asked 2020-10-25T13:53:15.673+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 93%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKK%3C/text%3E%3C/svg%3E)
kranthi k senapathi
106
Reputation points
commented 2020-10-29T17:45:16.183+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
HDInsight azure adls gen2 'InternalServerError' ARM Template deployment
Creating Azure HDinsight Spark cluster with ADLS Gen 2,Userassigned managed idnetity with StorageBlobdataOwner role. Successfully assigned msi role to storage but getting error with HDInsight deployment(Internal server error) Theres some issue with…
asked 2020-10-08T04:56:53.52+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 8%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anon4914
1
Reputation point
commented 2020-10-28T07:51:37.547+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee