Azure HDInsight
An Azure managed cluster service for open-source analytics.
204 questions
1 answer
How will Azure Application Insight help to optimize my application
Hello ! I would like to know how will enabling Azure application insight help me to analyze the performance issues in my application . WIll enabling this slow down my application / is there anything we need to check before this is enabled. Thanks
asked 2023-10-12T09:36:42.2133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 67%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKB%3C/text%3E%3C/svg%3E)
kavipriya balaji
0
Reputation points
answered 2023-10-12T18:56:46.1966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(54.400000000000006, 38%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EQ%3C/text%3E%3C/svg%3E)
QuantumCache
20,186
Reputation points
1 answer
How to read and write data to hbase hdinsights with databricks
I'm doing a proof of concept to compare which tool best suits our company, Cosmos DB or A Hbase cluster on HDInsights: I'm trying to read and write data using Databricks on HDinsights HBase. I tried to use the SHC lib but it is only available for version…
asked 2023-09-21T14:38:31.71+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 92%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENI%3C/text%3E%3C/svg%3E)
Nobuyoshi Ishizuka
0
Reputation points
commented 2023-09-25T19:28:50.6966667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
28,606
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure HD Insights Spot instances.
How to use the Azure HDInsights Spot instances. What is the max limit here?
asked 2023-09-11T17:50:23.2666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 88%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH7%3C/text%3E%3C/svg%3E)
HariKrishna-7067
120
Reputation points
accepted 2023-09-12T17:01:28.91+00:00
HariKrishna-7067
120
Reputation points
2 answers
One of the answers was accepted by the question author.
Suggestions for Optimizing Data Processing Speed and Migrating to Azure
Problem Description: We are currently facing a data processing challenge: our data processing workflow runs Java programs on a single machine, and data is stored in a DB server with a total dataset of over a hundred million records. Each run of our data…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(284.8, 87%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
Remotly connect to AzureHDInsight 5.0 spark cluster
Hi, I have a local jupyter notebook and Im trying to connect to an AzureHDInsight 5.0 spark cluster. I found tutorials on how to create a jupyter notebook directly inside the Azure Cloud Interface of the cluster, but what if I want to connect remotly? …
asked 2023-08-29T12:42:49.91+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 24%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
Dario Bertolino
0
Reputation points
commented 2023-09-01T21:25:39.9933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,437
Reputation points Microsoft Employee
1 answer
HDinsight 4.0 to 5.0
Differences between HDinsight Componant functionality of 4.0 and 5.0 major change , minor change Does the Ambari page affect usability?
asked 2023-08-18T10:20:04.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 79%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPK%3C/text%3E%3C/svg%3E)
PANUWAT KLAHAN
0
Reputation points
commented 2023-08-30T10:53:36.19+00:00
ShaikMaheer-MSFT
38,321
Reputation points Microsoft Employee
1 answer
How is headnode failure handled in hdinsight ?
Hi, As HDInsight have fixed number of head nodes i.e 2. I'm curious how head node failures are handled in HDinsight ? Does failed headnode gets removed and a new headnode gets added in cluster ? failed headnode remains in cluster ? Thanks, Akshit Mehta
asked 2023-08-11T06:20:42.81+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(105.60000000000001, 21%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3E3%3C/text%3E%3C/svg%3E)
31610895
20
Reputation points
commented 2023-08-29T05:47:47.12+00:00
QuantumCache
20,186
Reputation points
1 answer
One of the answers was accepted by the question author.
Does HDInsight manage zookeeper node ? How are failures handled in zookeeper nodes ?
Hi, First question - As HDInsight have fixed number of zookeeper nodes i.e 3. I'm curious how zookeeper node failures are handled in HDinsight ? Does failed zookeeper node gets removed and a new zookeeper node gets added in cluster ? failed…
asked 2023-08-16T10:08:51.13+00:00
31610895
20
Reputation points
commented 2023-08-22T06:52:45.5433333+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
2 answers
How to use spot instances with HDInsight
Hi, Is there any way to use Spot VMs in HDInsight ? How to determine which VMs are spot VMs in this list - https://azure.microsoft.com/en-in/pricing/details/virtual-machines/linux/ ? Is there any max limit on number of Spot VMs that can be…
asked 2023-08-16T10:26:35.41+00:00
31610895
20
Reputation points
commented 2023-08-17T06:34:57.5766667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 45%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESZ%3C/text%3E%3C/svg%3E)
shlim@zenithn.com
1
Reputation point
2 answers
Upgrading Storage account to GPV2
Hi , I am having storage account in which I am getting an recommendation from upgrading it to GPV2 ie General Purpose V2 . But there is a dependency like it is connected with HD insight Cluster. So is there any issue like where if I upgrade the storage…
asked 2023-07-13T12:33:11.81+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 92%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Sumedh Patil
20
Reputation points
commented 2023-08-02T14:42:12.8833333+00:00
Sumarigo-MSFT
44,906
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
HDInsight data governance/lineage tool or framework
Hi folks, We are implementing a bigdata solution using HDInsight (Hive Interactive Query and Spark with an azure SQL db for hive metastore), is a requirement from the client to provide a data governance, linage, data masking solution. Based on what I've…
asked 2023-07-18T20:41:31.2933333+00:00
Federico Sardo
91
Reputation points
accepted 2023-07-25T12:50:42.55+00:00
Federico Sardo
91
Reputation points
1 answer
One of the answers was accepted by the question author.
unable to access index of repository mran.microsoft.com/
getting below error while installing r packages using mran.microsoft.com repo. this was working fine on June 2nd 2023. please help. #27 130.6 > install.packages("devtools", dependencies = TRUE) #27 130.6 Installing package into…
asked 2023-07-11T10:31:58.0233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 15%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAP%3C/text%3E%3C/svg%3E)
Addepalli, Pavan
20
Reputation points
edited the question 2023-07-18T05:28:27.6933333+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
2 answers
trino vm access to hdi hive metastore - nsg's wide open, destination host unreachable
Within the same resource group, created trino instance, and HDI cluster. Different Vnets of course. Set wide open nsg rules (inbound/outbound, all ports, all protocols). Trino node cannot connect to hdi presto metacatalog service (connection timeout in…
asked 2023-05-17T17:48:49.35+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 30%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
DR
0
Reputation points
commented 2023-05-28T14:28:08+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
1 answer
How to set static ip for head nodes in HDInsight Cluster?
Hi, I am working with HDInsight cluster, the type of the cluster is Hadoop. I am using ARM templates to create and destroy the cluster every day (8am to 6pm). Regarding networking, I am using an azure virtual network with a subnet for hdinsight. We are…
asked 2023-05-08T20:29:37.1033333+00:00
Federico Sardo
91
Reputation points
commented 2023-05-23T12:10:32.36+00:00
ShaikMaheer-MSFT
38,321
Reputation points Microsoft Employee
2 answers
Azure HDInsights SQL integration - lack of Managed Identity authentication
Hello, while creating HDInsight cluster with Spark, I found out that SQL database for metadata cannot be connected using managed identity - can You improve that? Note. There is a need to use managed identity for Storage Account Access (or Lake).
asked 2023-05-16T09:48:07.7133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 44%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKW%3C/text%3E%3C/svg%3E)
Krzysztof Świdrak
166
Reputation points
answered 2023-05-16T09:55:42.8266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 64%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESY%3C/text%3E%3C/svg%3E)
Sairam Yeturi
75
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Can we upgrade commons-io in HDInsight
Hello Team, We're currently running HDInsight 5.0 for the spark runtime. We're facing the issue wrt to the common-io lib which is available as the part of HDInsight 5.0 since it has lower version of it which 2.5v. In our project, we're dependent on a lib…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 44%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
Where is hue service and url in Hdinsight?
Hi, I created a hdinsight (hadoop) cluster and I used the script action to install hue. The installation was Succeeded: But I was unable to find hue service or url in ambari Could you please help me? Regards Fede
asked 2023-03-13T20:14:16.37+00:00
Federico Sardo
91
Reputation points
edited a comment 2023-04-25T04:51:32.5566667+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
2 answers
How to setup custom database for ambari in HDInsight?
Hi, I am creating a hadoop cluster in HDInsight. For ambari I created an Azure SQL Database and during the creation I selected not to use existing data. But when I want to create the cluster and try to select the database, I am facing this warning: I…
asked 2023-03-27T19:50:48.9866667+00:00
Federico Sardo
91
Reputation points
commented 2023-04-10T06:44:25.6333333+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
1 answer
Big data access
Hi, i've created a student account because I'm studying for a master, but that type of account doesn't work for big data. What do I need to access Hadoop and apache spark on Azure, and what are the costs? thanks joao
asked 2023-03-26T10:39:06.7133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 51%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJR%3C/text%3E%3C/svg%3E)
Joao Ribeiro
20
Reputation points
commented 2023-03-29T07:57:04.3266667+00:00
KranthiPakala-MSFT
46,437
Reputation points Microsoft Employee
3 answers
One of the answers was accepted by the question author.
how to access hadoop and Apache spark in azure
Hi, I've created my azure student account, but can't access HDinsight to use hadoop and Apachec spark for my master. How to i do? cheers joao acount (sba22203@student.cct.ie)
asked 2023-03-21T09:56:05.49+00:00
Joao Ribeiro
20
Reputation points
answered 2023-03-22T08:54:53.7633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 64%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
彬 陈
0
Reputation points