Unable to change spark.executor.heartbeatInterval parameter

Kurmann Simon (Helbling Technik) 21 Reputation points
2020-06-22T06:27:09.24+00:00

I try to run a Jupyter Notebook on HDInsight with Spark; after some time (observed: 15, 17, 30 minutes), execution fails with error message:
Error with 400 StatusCode: "requirement failed: Session isn't active."

According to Stack Overflow (https://stackoverflow.com/a/53306912, https://stackoverflow.com/a/38976472), one possible remedy is increasing the value of the Spark Config parameter spark.executor.heartbeatInterval. But I cannot find this parameter on the Ambari Spark Config page.

How can I change spark.executor.heartbeatInterval?

Azure HDInsight
Azure HDInsight
An Azure managed cluster service for open-source analytics.
207 questions
0 comments
Accepted answer
  1. HimanshuSinha-msft 19,381 Reputation points Microsoft Employee
    2020-06-23T22:39:04.697+00:00

    Hello @KurmannSimonHelblingTechnik-4973

    You can add this by going to to Ambari page ->Custom spark2-defaults ->Add property -> spark.executor.heartbeatInterval .

    Please do let me know how it goes .

    Thanks & stay safe

    Himanshu
    Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

    1 person found this answer helpful.
    0 comments

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest