Configuration related exception while trying to run a spark app in HDInsight 5.0 cluster

I am migrating from HDInsight 4.0 to 5.0. Locally, it works. However, when I ran spark jobs in HDInsight cluster, I got the below error. Any idea why "spark.nonjvm.error.forwarding.enabled" is registered multiple times?

Command to run spark job

casprod@hn0-caslin:/usr/hdp/current/spark3-client$ $SPARK_HOME/bin/spark-submit --master yarn --class org.apache.spark.deploy.dotnet.DotnetRunner --files abfs://testcontainer@lake1.dfs.core.windows.net/spark/dotnetApp/CaptiveAireSparkJobEngine.dll --conf spark.yarn.appMasterEnv.DOTNET_WORKER_DIR=/usr/local/bin/microsoft.spark.worker --jars abfs://testcontainer@lake1.dfs.core.windows.net/spark/dotnetApp/microsoft-spark-3-1_2.12-2.1.1.jar abfs://testcontainer@lake1.dfs.core.windows.net/spark/dotnetApp/publish.zip dotnetcorerunnerdll -a GenericJobApp -j abfs://testcontainer@lake1.dfs.core.windows.net/spark/casrtujob/


Error message

23/02/22 21:33:14 WARN SparkConf [main]: The configuration key 'spark.yarn.access.hadoopFileSystems' has been deprecated as of Spark 3.0 and may be removed in the future. Please use the new key 'spark.kerberos.access.hadoopFileSystems' instead.
23/02/22 21:33:14 WARN SparkConf [main]: The configuration key 'spark.yarn.access.hadoopFileSystems' has been deprecated as of Spark 3.0 and may be removed in the future. Please use the new key 'spark.kerberos.access.hadoopFileSystems' instead.
Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.apache.spark.deploy.dotnet.DotnetRunner$.main(DotnetRunner.scala:55)
        at org.apache.spark.deploy.dotnet.DotnetRunner.main(DotnetRunner.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:956)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:204)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1044)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1053)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalArgumentException: requirement failed: Config entry spark.nonjvm.error.forwarding.enabled already registered!
        at scala.Predef$.require(Predef.scala:281)
        at org.apache.spark.internal.config.ConfigEntry$.registerEntry(ConfigEntry.scala:283)
        at org.apache.spark.internal.config.ConfigEntry.<init>(ConfigEntry.scala:87)
        at org.apache.spark.internal.config.ConfigEntryWithDefault.<init>(ConfigEntry.scala:133)
        at org.apache.spark.internal.config.TypedConfigBuilder.createWithDefault(ConfigBuilder.scala:147)
        at org.apache.spark.internal.config.dotnet.Dotnet$.<init>(Dotnet.scala:21)
        at org.apache.spark.internal.config.dotnet.Dotnet$.<clinit>(Dotnet.scala)
        ... 14 more