HDInsight HBase vs Databricks
Hi, this is probably answered or perhaps a tall question. What would be the difference/benefits between using HDInsight HBase vs Databricks. Azure storage is definitely one. If the aim is to have the convenience of traditional table & sql with…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
To use Azure Data Lake Storage Gen2 with Azure HDInsight clusters, do I have to attach the storage to clusters as linked additional storage?
To use Azure Data Lake Storage Gen2 with Azure HDInsight clusters, do I have to attach the storage to clusters as linked additional storage? Or as long as permission is granted to managed identity, spark scala application could access the storage using…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Creating a HDInsight Spark 4.0 cluster with managed identity and a Data Lake Store gen 2 storage account
Hi! I am trying to create a HDI cluster with an ADLS Gen2 storage account as primary storage account. I have created multiple containers inside my storage account, and I want to limit the access of the managed identity to the other containers. …
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
HDInsights cluster is in the Error status, even when the user assigned Managed Identity is assigned a role as Storage Blob Data Owner.
HDInsights cluster is in the Error status, even when the user assigned Managed Identity is assigned a role as Storage Blob Data Owner.
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
HD insight Cluster, Worker node, E32_V3 (256 GB), memory issue
HD insight Cluster - Worker node - E32_V3 (256 GB). It is showing 911 GB memory on ambari portal. MS document for E32_v3 of HD insight worker node showing 1600 GB space. Why there is a discrepancy.
Unable to create HDInsight cluster through free azure subscription
There are not enough cores available to support the selected number of nodes. Please adjust the number of nodes selected, pick a different region, or open a support case to request additional HDInsight cores. You have reached your subscription's…
azure hdinsight There are not enough cores available
I want to create HDInsight in my pay as you go subscription, but I get error: There are not enough cores available to support the selected number of nodes. I checked in my subscription usage and quotas for computing and usage is for every processor…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Files not getting saved in Azure blob using Spark in HDInsights cluster
We've setup HDInsights cluster on Azure with Blob as the storage for Hadoop. We tried uploading files to the Hadoop using hadoop CLI and the files were getting uploaded to the Azure Blob. Command used to upload: Hadoop fs -put somefile…
Connect Synapse Spark Pool with Kafka on HDInsight
I have created a Kafka on HDinsight cluster . I have also created Azure Synapse Analytics - Spark Pool on same region as HDinsight. I need guidance on how to consume topics from Kafka into Spark Structured Streaming. Any documentation or steps will be of…
HDInsight - Kafka - Version 3.2
Hi all Is there a roadmap to release a cluster with a higher kafka version than 2.4.1 in the near future? Thanks for the info in advance. Best reagrads, Michael
Can Azure Streaming Analytics read from Kafka on HDInsight and write to Deltalake table on Synapse lake.
Hello I am looking for guidance on building a new event driven platform. The options we are exploring for processing are - Azure Stream Analytics Apache Spark Structured Streaming in Synapse Source is like going to be Kafka on HDInsight …
what is the best way to copy data from my hadoop on prem cluster to the azure hdinsight cluster?
hi experts, what is the best way to copy data from my hadoop on prem cluster to the azure hdinsight cluster? So we recently deployed a new hdinsight cluster and now I would like to copy some data from my onprem cluster to hdinsight. Thanks,
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
How to run hdfs commands from my on prem cluster to azure?
Hi experts, How to run hdfs commands from my on prem cluster to azure? So I have an on prem cluster that I would like to run hdfs commands to read files that are from my Azure HDinsight cluster. How can I do this? Thanks,
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
HDinsight monitering
Hi Friends i am new to HD insight any ida about hd insight clusters monitor What are the major things we need to observe and moniter we are using apache Ambari
Looking for HDInsight support team alias
I am running into an error while running a spark job in Azure Data Factory's pipeline and would like to connect to the HDInsight support team for further assistance. If you can please provide the alias
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Can't create HDInsight Cluster(Hadoop)
Dear All, I am struggling against creating HDInsight. After reviewing documents and other posts, I upgraded free-trial to paid subscription and created paid subscription as well. However, regardless of subscription types(both subscription paid type) I…
![](https://techprofile.blob.core.windows.net/images/lXQCawBQcEmHu8zEgEBRpg.png?8DA5F6)
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
How to leverage Azure key vault secrets from HD Insight Jupyter notebook?
Hi, I am trying to store the user id and password in Secrets and retrieve them in HD Insight Jupyter notebook? Any guidance.
How to use GnuPG in HDInsight for encryption and decryption?
Hi, I am working with the HDInsight Spark cluster on Azure. Trying to encrypt files with pgp encryption using our private key. Is there a way that this can achieve rather than using the inbuilt encryption mechanism? How to set the home for GnuPG…
Not enough cores error while deploying resource group on Azure for Students
Hello! I am very new to Azure. I have an Azure for Students subscription and I'm trying to create an Apache Kafka cluster using Azure HDInsight. I selected West Europe as my region. I'm using this resource as a guide:…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Zeppelin notebook - sc.textFile does not work for HDI with ESP
We have HDI cluster with ESP enabled. From our zeppelin notebook, when I read data to a dataset (spark.read.text) it works but when I try to read it to an RDD (sc.textFile), I get an authentication exception: Note that, while sc.textFile…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)