Apache slf4j error when setting up spark pool

Daniel Pearson 20 Reputation points
2024-09-02T09:32:31.5366667+00:00

I am getting the following errors when my apache spark pool job is running. The main issue seems to be the slf4j error. How do I fix this:
User's image

Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/mnt/tmp
Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/mnt/tmp
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/5.2.20240522.3/spark3/jars/spark-streaming-kafka-0-10-assembly_2.12-3.3.1.5.2.20240522.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/5.2.20240522.3/spark3/jars/log4j-slf4j-impl-2.17.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Warning: Ignoring non-Spark config property: EntitySizes
Warning: Ignoring non-Spark config property: EntityNames
24/09/02 08:01:54 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/09/02 08:01:54 WARN AzureBlobFileSystemStore: checkDnsEntry: prdcpf11wukmefstor.dfs.core.windows.net not found in the file /etc/hosts.
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,858 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. NIKHILA NETHIKUNTA 1,760 Reputation points Microsoft Vendor
    2024-09-03T16:11:07.9733333+00:00

    @Daniel Pearson
    Thank you for the question and using Microsoft Q&A platform.

    The error message you are seeing indicates that there are multiple bindings for the Simple Logging Facade for Java (SLF4J) class path. This can happen when multiple libraries or dependencies are using different versions of SLF4J.

    To fix this issue, you can try excluding one of the SLF4J bindings from your class path.
    In your case, it looks like you have two bindings: one in the spark-streaming-kafka-0-10-assembly_2.12-3.3.1.5.2.20240522.3.jar file and another in the log4j-slf4j-impl-2.17.2.jar file.

    Please refer to these links for more help:
    https://stackoverflow.com/questions/14024756/slf4j-class-path-contains-multiple-slf4j-bindings
    https://community.cloudera.com/t5/Support-Questions/hadoop-SLF4J-Class-path-contains-multiple-SLF4J-bindings/td-p/239400
    https://www.slf4j.org/codes.html

    Hope this helps. Please let us know if you have any further questions.

    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.