Azure HDInsight
An Azure managed cluster service for open-source analytics.
204 questions
1 answer
Azure HDinsight Ambari RBAC
Hello, Can someone please tell me is there any programmatic way to use Ambari in Azure HDinsight to managed users and Role based access control. To clarify, I am asking about Ambari RBAC not Azure RBAC And can we do ambari admin password reset using…
asked 2020-10-02T11:14:45.01+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 61%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Akash Verma
21
Reputation points
commented 2020-11-13T23:54:28.65+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,051
Reputation points
0 answers
HD Insights - converting the Aggregated Logs output into ADLSGen2 into Text format
Can someone share a reference to systematically output logs from the HDinsights application runs into a readable format -- for the stdout. Today when running a Spark application through HdInsights (triggered by Livy API), after the application finishes…
asked 2020-10-22T17:07:06.273+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(67.2, 46%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBC%3C/text%3E%3C/svg%3E)
Brendan Cassidy
26
Reputation points Microsoft Employee
commented 2020-11-11T18:23:27.957+00:00
Brendan Cassidy
26
Reputation points Microsoft Employee
0 answers
Need to know the actual size of traffic received in KB's/MB's/GB's in a day for per kafka topic in HDInsight Cluster
Hello All, I'm using HDI-3.6 Kafka cluster(0.1) in west Europe region with 26 Kafka brokers running in it on production environment. I'm trying to find out the actual size of traffic received in KB's/MB's/GB's in a day/hour for per Kafka topic in…
asked 2020-10-21T09:34:05.653+00:00
Sugumar Srinivasan
1
Reputation point
commented 2020-10-28T21:21:20.893+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,376
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
HD Insights - Script Action stored in ADLSGen2
When trying to run Bash scripts as Script Actions on Cluster creation is there a way to use a ADLSGen2 location? From the current documentation it shows ADLGen1, and public HTTPS requests but does not discuss this possibility with Gen2. In scripts…
asked 2020-10-26T20:12:52.64+00:00
Brendan Cassidy
26
Reputation points Microsoft Employee
accepted 2020-10-28T16:49:12.527+00:00
Brendan Cassidy
26
Reputation points Microsoft Employee
1 answer
HDInsight azure adls gen2 'InternalServerError' ARM Template deployment
Creating Azure HDinsight Spark cluster with ADLS Gen 2,Userassigned managed idnetity with StorageBlobdataOwner role. Successfully assigned msi role to storage but getting error with HDInsight deployment(Internal server error) Theres some issue with…
asked 2020-10-08T04:56:53.52+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 8%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anon4914
1
Reputation point
commented 2020-10-28T07:51:37.547+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
1 answer
ARM Template for Azure HDinsights
Can someone share any reference or ARM Template for Azure HDInsight which covers below things: Use Public-Private key pair together with a passphrase for SSH login Enterprise security package must be enabled TLS Version Encryption at rest…
asked 2020-10-02T11:08:56.783+00:00
Akash Verma
21
Reputation points
commented 2020-10-20T17:08:47.017+00:00
MartinJaffer-MSFT
26,051
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure HD Insight Kafka Broker Setting for max.incremental.fetch.session.cache.slots
Hi, I need to change the setting for Broker Config for max.incremental.fetch.session.cache.slots . But I am not able to see this as a property on Ambari Broker Config. How can I change it ?
asked 2020-10-12T21:53:44.247+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 4%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHA%3C/text%3E%3C/svg%3E)
Harjani, Ashish
201
Reputation points
accepted 2020-10-18T23:30:33.48+00:00
Harjani, Ashish
201
Reputation points
1 answer
Programmatic way to perform tasks on Azure HDinsight
Can someone please give reference to perform below tasks on Azure HDinsight via programmatic way: Enable diagnostics setting on HDinsights Cluster Enable Azure Monitor for the HDInsight cluster Enable NSG Flow Logs NSG Modifications (Rule Add,…
asked 2020-10-07T09:20:05.27+00:00
Akash Verma
21
Reputation points
commented 2020-10-09T11:08:02.917+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
1 answer
How we link livy server is end point correct in HDinsight
Hi All, i want to submit my spark job from intellij to use adf spark activity. instead of head node host which IP address i have to give. here i give password of my admin account username/password but it is error out i am new…
asked 2020-09-23T14:56:17.643+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 20%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
manish verma
441
Reputation points
commented 2020-10-07T13:06:32.897+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
1 answer
Error Setting Access Rights on ADL (Gen1) User Folders During HDInsight Cluster Creation
Hello All The issue occurs when the cluster is created. One of the last operations that Ambari performs are 'Post user creation hook for 1 users'. This fail with the attached errors. Here is some more information: It's an ESP cluster -- we…
asked 2020-09-22T07:21:55.527+00:00
Christoph Kiefer
141
Reputation points
commented 2020-10-07T13:04:47.837+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Spark unable to write file onto Blob storage
We use HDInsight with Spark, v3.6. So far, our code has been working as expected. As of last night, our job started failing. The error states that "output directory already exists". When looking at the blob storage, directories appear to be…
asked 2020-10-02T16:25:54.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 65%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Sachin Shah
101
Reputation points
accepted 2020-10-07T12:05:08.527+00:00
Sachin Shah
101
Reputation points
0 answers
Is it possible to send data from WPF app to Azure analytics?
Is it possible to send data from WPF app to Azure analytics? If yes, which is the minimum Visualstudio version & .net framework version required ? which are the best analytic services offered by Azure which can used with WPF app?
asked 2020-10-01T09:42:53.153+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 71%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
Hi
1
Reputation point
commented 2020-10-07T00:46:17.98+00:00
MartinJaffer-MSFT
26,051
Reputation points
2 answers
One of the answers was accepted by the question author.
How i call a Hdinsight Zeppelin notebook in Azure Data Factory
How i call a Hdinsight Zeppelin notebook in Azure Data Factory. i have a Zeppelin notebook in Hdinsight cluster , i need to call in azure data factory. what is best way to write spark code , and call in data factory.
asked 2020-09-21T11:31:03.803+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 57.00000000000001%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVM%3C/text%3E%3C/svg%3E)
Verma, Manish Kumar
131
Reputation points
accepted 2020-10-02T08:22:44.707+00:00
Verma, Manish Kumar
131
Reputation points
1 answer
How to add ambari infra to a hbase cluster?
I tried to add it by add service link next to it in the admin section but add service link is not responding. Is there any other way to add this service?
asked 2020-09-17T05:52:27.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(236.8, 50%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Ayushi Bhandari
1
Reputation point
commented 2020-09-30T12:16:45.453+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
1 answer
Apache Atlas on azure hdinsigt
How to install apache atlas on azure HDinsight?
asked 2020-09-16T08:24:28.39+00:00
Ayushi Bhandari
1
Reputation point
commented 2020-09-30T12:16:25.453+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Data Analytics Studio (DAS) on HDInsight Cluster
Dear All I am following the recommendations here to install the Data Analytics Studio (from Cloudera): https://video2.skills-academy.com/en-us/azure/hdinsight/interactive-query/apache-hive-migrate-workloads The custom script action is not working…
asked 2020-09-22T11:56:27.23+00:00
Christoph Kiefer
141
Reputation points
accepted 2020-09-30T07:11:23.807+00:00
Christoph Kiefer
141
Reputation points
1 answer
One of the answers was accepted by the question author.
remainingCores is too small error. Spark and Kafka cluster on HDInsight problem
I tried to create a Kafka and Spark cluster in the HDInsight using either of templates in these links: …
asked 2020-09-27T06:57:32.923+00:00
Erfan Mahmoodnejad
41
Reputation points
commented 2020-09-29T06:31:58.623+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Create On Demand HDInsight from Data Factory in an existing VNet
The Azure Data Factory can create an on-demand HDInsight (linked service), which is automatically deleted after the job running on it finishes. Is it possible to have this on-demand HDInsight created in an existing VNet?
asked 2020-09-23T07:07:55.617+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 4%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
Beniamin Murza
26
Reputation points
commented 2020-09-28T12:11:44.993+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Apache Kafka as a Service with Confluent Cloud Azure
We are trying to connect to on premise server from our Apache Kafka as a Service with Confluent Cloud; with on-premise SQL Server source, but receive below error Do we require any specials setting from azure cloud to on-premise?
asked 2020-09-26T04:42:30.917+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 48%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDN%3C/text%3E%3C/svg%3E)
Dondapati, Navin
281
Reputation points
answered 2020-09-28T09:59:03.723+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
0 answers
creating conda environment on azure hdinsight spark cluster taking hours
trying to follow the code in this tutorial , https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/training/train-in-spark/train-in-spark.ipynb when i send the run to run to azure ml, the run essentially stalls on the step…
asked 2020-09-10T14:14:09.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 1%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKK%3C/text%3E%3C/svg%3E)
kunal kapoor
1
Reputation point
commented 2020-09-15T22:43:58.83+00:00
MartinJaffer-MSFT
26,051
Reputation points