How to fix the below issue which I am facing while installing the maven library with coordinates as 'com.databricks:spark-xml_2.12:0.14.0'

Dathathraiah 0

How to fix the below issue which I am facing while installing the maven library with coordinates as 'com.databricks:spark-xml_2.12:0.14.0' in ADF pipeline in azure databricks notebook activity

run failed with error message Library installation failed for library due to user error for maven { coordinates: "com.databricks:spark-xml_2.12:0.14.0" } Error messages: Library installation attempted on the driver node of cluster 1027-075217-qq4l2fsi and failed. Please refer to the following error message to fix the library or contact Databricks support. Error code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error message: Library resolution failed because unresolved dependency: com.databricks:spark-xml_2.12:0.14.0: not found

Vinodh247 22,951 Reputation points MVP

2024-10-31T10:26:13.0633333+00:00

Just checking if you had a chance to look at this. if so, can you please update it?

2 answers

Vinodh247 22,951 Reputation points MVP

2024-10-27T13:38:54.6066667+00:00
Hi Dathathraiah,

Thanks for reaching out to Microsoft Q&A.

The error you’re encountering indicates that the com.databricks:spark-xml_2.12:0.14.0 library couldn’t be found or resolved in the Maven repository Databricks is accessing.

Here are steps to troubleshoot and potentially resolve this:

Verify the Maven Repository Availability:

Databricks may not have access to the repository hosting spark-xml_2.12:0.14.0. Check if the library is hosted on Maven Central or another repository. You can manually download the library .jar file from Maven Central or other repository sources if it’s available.

Specify Repository Configuration:

Sometimes, Databricks clusters may need additional configuration to access non-default Maven repositories. If the library isn’t on Maven Central, specify the repository in the cluster's Advanced Options:

Go to Cluster configuration in Databricks.

Under Libraries > Advanced Options, you can add additional Maven repositories.

Try an Alternative Version:

If the library isn’t available in version 0.14.0, you could try another version such as 0.13.0 or the latest available version for Spark XML.

Install the Jar File Directly:

If you can download the .jar file manually, you can upload it to your Databricks workspace or DBFS and then attach it as a library to your cluster in ADF:

Use dbfs cp to move the .jar to the Databricks File System.

Attach the .jar file as a library in your Databricks notebook activity.

Check for Dependency Conflicts:

Conflicts between Spark and Scala versions might cause issues. Ensure compatibility by confirming that your Spark version in Databricks matches the expected version for spark-xml (ex: Spark 2.x or 3.x series).

If none of these resolve the issue, another option is to file a support ticket with Databricks, as they may provide more specific guidance on the DRIVER_LIBRARY_INSTALLATION_FAILURE error.

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.
Please sign in to rate this answer.
Hlaváček Martin 5 Reputation points

2024-11-01T06:36:55.5866667+00:00

Hello, I am specifically looking for option to update default maven repository to use our local repo within artifactory. So users dont need to specify repo url every time then are installing library. Your proposal in section "Specify Repository Configuration:" tends to be an option but I dont see such configuration within my databricks. I am workspace admin. I dont have section advanced option within clusters libraries section. Alse there is no section libraries within clusters advanced option setting (there is just Spark, Logging, InitScripts). Can you more elaborate how can I update or add default maven repository for cluster? Thank you

Vinodh247 22,951 Reputation points MVP

2024-11-01T07:31:52.4166667+00:00

To set up a default Maven repository for your Databricks clusters that doesn’t require users to manually specify the repository URL each time, you can use cluster init scripts. Here’s how to configure this using an init script that updates the Maven configuration to point to your Artifactory repository:

Step-by-Step Guide to Configure Artifactory as Default Maven Repository

Create the Init Script:

You’ll create a shell script that will update the spark-defaults.conf file or Maven configuration to point to your Artifactory URL.

Here’s a sample script (set_maven_repo.sh):

#!/bin/bash echo "Setting up custom Maven repository in Databricks cluster"

Add repository details to the Maven configuration

cat <<EOT >> /databricks/driver/conf/spark-env.sh export MAVEN_OPTS="-Dspark.jars.repositories=https://your-artifactory-url/artifactory/libs-release" EOT

Upload this script to your Databricks File System (DBFS):

dbfs cp /local/path/to/set_maven_repo.sh dbfs:/databricks/scripts/set_maven_repo.sh

Attach the Init Script to Your Cluster:

Go to your cluster configuration in Databricks.

Under Init Scripts (in the Advanced Options tab), add the path to the script you just uploaded: dbfs:/databricks/scripts/set_maven_repo.sh.

Restart the Cluster:

Restart your cluster to apply the configuration. This script will ensure that the Artifactory repository is used by default for all Maven installations.

This setup allows you to define a default Maven repository for the cluster, streamlining library installations across users without needing additional configurations on each install. If this approach doesn’t meet your needs, you may want to check if ur databricks account supports any premium features that allow direct maven repo configurations.

Vinodh247 22,951 Reputation points MVP

2024-11-01T07:33:29.43+00:00

To set up a default Maven repository for your Databricks clusters that doesn’t require users to manually specify the repository URL each time, you can use cluster init scripts. Here’s how to configure this using an init script that updates the Maven configuration to point to your Artifactory repository:

Step-by-Step Guide to Configure Artifactory as Default Maven Repository

Create the Init Script:

You’ll create a shell script that will update the spark-defaults.conf file or Maven configuration to point to your Artifactory URL.

Here’s a sample script (set_maven_repo.sh):

#!/bin/bash echo "Setting up custom Maven repository in Databricks cluster" Add repository details to the Maven configuration cat <<EOT >> /databricks/driver/conf/spark-env.sh export MAVEN_OPTS="-Dspark.jars.repositories=https://your-artifactory-url/artifactory/libs-release" EOT

Upload this script to your Databricks File System (DBFS):

dbfs cp /local/path/to/set_maven_repo.sh dbfs:/databricks/scripts/set_maven_repo.sh

Attach the Init Script to Your Cluster:

Go to your cluster configuration in Databricks.

Under Init Scripts (in the Advanced Options tab), add the path to the script you just uploaded: dbfs:/databricks/scripts/set_maven_repo.sh.

Restart the Cluster:

Restart your cluster to apply the configuration. This script will ensure that the Artifactory repository is used by default for all Maven installations.

This setup allows you to define a default Maven repository for the cluster, streamlining library installations across users without needing additional configurations on each install. If this approach doesn’t meet your needs, you may want to check if ur databricks account supports any premium features that allow direct maven repo configurations.To set up a default Maven repository for your Databricks clusters that doesn’t require users to manually specify the repository URL each time, you can use cluster init scripts. Here’s how to configure this using an init script that updates the Maven configuration to point to your Artifactory repository:

Step-by-Step Guide to Configure Artifactory as Default Maven Repository

Create the Init Script:

You’ll create a shell script that will update the spark-defaults.conf file or Maven configuration to point to your Artifactory URL.

Here’s a sample script (set_maven_repo.sh):

#!/bin/bash echo "Setting up custom Maven repository in Databricks cluster" Add repository details to the Maven configuration cat <<EOT >> /databricks/driver/conf/spark-env.sh export MAVEN_OPTS="-Dspark.jars.repositories=https://your-artifactory-url/artifactory/libs-release" EOT

Upload this script to your Databricks File System (DBFS):

dbfs cp /local/path/to/set_maven_repo.sh dbfs:/databricks/scripts/set_maven_repo.sh

Attach the Init Script to Your Cluster:

Go to your cluster configuration in Databricks.

Under Init Scripts (in the Advanced Options tab), add the path to the script you just uploaded: dbfs:/databricks/scripts/set_maven_repo.sh.

Restart the Cluster:

Restart your cluster to apply the configuration. This script will ensure that the Artifactory repository is used by default for all Maven installations.

This setup allows you to define a default Maven repository for the cluster, streamlining library installations across users without needing additional configurations on each install. If this approach doesn’t meet your needs, you may want to check if ur databricks account supports any premium features that allow direct maven repo configurations.

Hlaváček Martin 5 Reputation points

2024-11-01T08:20:27.35+00:00

Thank you very much, you saved my day! Just tested it and works as expected. I wasnt able to found this anywhere else.

Vinodh247 22,951 Reputation points MVP

2024-11-01T09:17:55.4666667+00:00

Hlaváček Martin: Glad it helped you and with your confirmation this answer can serve others who face the same issue in future. Please 'Accept as Answer' and click the 'Upvote'(Thumbs-up) button if the reply was helpful. This will help us to close this thread.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Amira Bedhiafi 26,101 Reputation points

2024-10-27T14:59:24.77+00:00

Have you checked if com.databricks:spark-xml_2.12:0.14.0 exists in the Maven repository? The latest version of spark-xml may be different, and 0.14.0 might not be available or may have been deprecated. You can search for the package on Maven Central or Databricks documentation to confirm the correct version.

Sometimes, certain packages are hosted in specific repositories (e.g., the Databricks Maven repository). You can add the repository link in the Maven configuration within the Databricks notebook activity in ADF. For Databricks-specific libraries, you can try adding:

Repository URL: https://databricks.jfrog.io/artifactory/maven/

If the library is unavailable through Maven, try downloading the jar file directly from a reliable source or building it manually. Once you have the jar file, upload it to Databricks and manually attach it to the cluster.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to fix the below issue which I am facing while installing the maven library with coordinates as 'com.databricks:spark-xml_2.12:0.14.0'

2 answers

Add repository details to the Maven configuration

Your answer