Does HDInsight manage zookeeper node ? How are failures handled in zookeeper nodes ?

31610895 20 Reputation points
2023-08-16T10:08:51.13+00:00

Hi,

First question -
As HDInsight have fixed number of zookeeper nodes i.e 3. I'm curious how zookeeper node failures are handled in HDinsight ? Does failed zookeeper node gets removed and a new zookeeper node gets added in cluster ? failed zookeeper node remains in cluster ?

Second question -
Do clients need to write some code / manage zookeeper nodes in HDInsight ? or HDInsight takes care of this ?

Thanks,

Akshit Mehta

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
204 questions
0 comments
Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 84,531 Reputation points Microsoft Employee
    2023-08-16T10:22:23.2366667+00:00

    @31610895 - Thanks for the question and using MS Q&A platform.

    Yes, Azure HDInsight manages the ZooKeeper nodes. In case of a ZooKeeper node failure, HDInsight automatically handles the failover process. HDInsight has three ZooKeeper quorums running in parallel with each other. One quorum decides the active headnode in a cluster on which HDInsight HA services should run. Another quorum is used to coordinate HA services provided by Apache. If a ZooKeeper node fails, the remaining nodes in the quorum will elect a new leader and continue to function normally.

    Regarding your second question, clients do not need to write any code or manage ZooKeeper nodes in HDInsight. Azure HDInsight takes care of the ZooKeeper nodes and provides a fully managed service. Clients can simply use the ZooKeeper service provided by HDInsight to coordinate their distributed applications.

    Hope this helps. Do let us know if you any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest