How to Add a subqueue in yarn

Vamsi Anamaneni 1 Reputation point
2022-08-10T19:52:34.937+00:00

I already have queues setup on Yarn on HdInsight, they were setup with the Ambari UI.

I have a queue for sqoop that takes up 70% of the cluster.
However I have a few huge sqoop jobs with a lot of mappers that take up 100% of the queue and block other sqoop jobs from running.
I wanted to send all sqoop jobs greater than 512 sqoop mappers into a subqueue that had a max of 80% of the sqoop queue which would allow the smaller sqoop jobs to run concurrently and also the subqueue, and also allows the smaller sqoop job subqueue to use the resources of the larger sqoop job subqueue if theres resources available.

Is something like this possible?

I see subqueues in the yarn documentation: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
I don't see subqueues in Microsoft documentation: https://video2.skills-academy.com/en-us/azure/hdinsight/hdinsight-troubleshoot-yarn

How do I setup the subqueues on Ambari UI on HDInsight?

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
204 questions