Configuring yarn alerts in Azure Monitor

Revanth Thavidishetty 1

Hi All,

Is there any customized alerts to monitor the HDInsight yarn memory from Azure monitor.

1 answer

PRADEEPCHEEKATLA-MSFT 89,466 Reputation points Microsoft Employee

2020-06-15T06:27:15.45+00:00
@Revanth Thavidishetty , Welcome to the Microsoft Q&A platform.

You can monitor your HDInsight cluster by sending metrics and logs to a Log Analytics workspace.

With Azure Monitor, you can

View and explore cluster-specific dashboards

Query and analyze OSS logs and metrics

View VM-level performance counters, such as Memory and CPU counters

Get notified about important events by setting Azure Monitor alerts

Monitor multiple clusters from the same Log Analytics workspace

Example: Once the logs are in Azure Monitor, you can create a new alerts based on the conditions required.

Reference: Use Azure Monitor logs to monitor HDInsight clusters and Overview of alerts in Microsoft Azure

Hope this helps. Do let us know if you any further queries.

----------------------------------------------------------------------------------------

Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.
Please sign in to rate this answer.
Revanth Thavidishetty 1 Reputation point

2020-06-15T16:32:43.763+00:00

@PRADEEPCHEEKATLA-MSFT Thanks for the information. I am using the below query to get the yarn memory usage. metrics_resourcemanager_queue_root_CL | summarize AggregatedValue = avg(AvailableMB_d ) by bin(TimeGenerated, 10m) | where AggregatedValue > 90000

do you have any suggestion on the above query

PRADEEPCHEEKATLA-MSFT 89,466 Reputation points Microsoft Employee

2020-06-16T05:57:29.72+00:00

@Revanth Thavidishetty , What are the suggestion required on the above query?

You can also use the logs view in your Log Analytics workspace to query the metrics and tables directly.

HDInsight clusters emit several workload-specific tables of logs, such as log_resourcemanager_CL, log_spark_CL, log_kafkaserver_CL, log_jupyter_CL, log_regionserver_CL, and log_hmaster_CL.

On the metrics side, clusters emit several metrics tables, including metrics_sparkapps_CL, metrics_resourcemanager_queue_root_CL, metrics_kafka_CL, and metrics_hmaster_CL. For more information, please see our documentation, Query Azure Monitor logs to monitor HDInsight clusters. and "Azure HDInsight monitoring".

Hope this helps.

PRADEEPCHEEKATLA-MSFT 89,466 Reputation points Microsoft Employee

2020-06-17T08:25:21.187+00:00

@Revanth Thavidishetty , Just checking in to see if the answer(s) helped. If you found a response helpful, please “Accept Answer” and Up-Vote for the same which might be beneficial to other community members reading this thread. And, if you have any further queries do let us know.

Revanth Thavidishetty 1 Reputation point

2020-06-17T17:47:23.807+00:00

@PRADEEPCHEEKATLA-MSFT Could you please give me a query to monitor the yarn memory and for long running jobs in clusters.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Configuring yarn alerts in Azure Monitor

1 answer

Your answer