Azure Databricks fail to install Geospark libraries from Maven

1. Check your network connectivity - Yes, I have checked. No issues with network connectivity.
2. Check the Maven repository - How to verify that the Maven repository is up and running and that the GeoSpark library is available?
3. Check the Databricks cluster configuration: Yes, I have checked the Databricks cluster configuration. Since my cluster is Unity Catalog enabled, I have added maven library (org.datasyslab:geospark:1.3.1 & org.datasyslab:geospark-sql_2.3:1.3.1) in Allow List. I realized that the allowlist needs to apply to interactive clusters configured with Unity Catalog in ’shared’ access mode'. Once added I restarted the clusters for changes to take effect.

The issue still persists.

Can you suggest which different Maven repository?

@Anuj, Singh (Cognizant) - In case, if you are still experiencing the same issue. I would suggest you share the steps which you are trying to install and also try to download the jar file directly from the maven site: https://mvnrepository.com/artifact/org.datasyslab and upload manually and install.

Can you share the steps to download the jar file directly from the maven site: https://mvnrepository.com/artifact/org.datasyslab and upload manually and install

I have checked the Databricks cluster configuration: My cluster is Unity Catalog enabled; I have added maven library (org.datasyslab:geospark:1.3.1 & org.datasyslab:geospark-sql_2.3:1.3.1) in Allow List. I realized that the allowlist needs to apply to interactive clusters configured with Unity Catalog in ’shared’ access mode'. Once added I restarted the clusters for changes to take effect.

I am encountering issues. Can you suggest how to ensure that the Databricks Runtime Version is compatible with the library version we are trying to install. In my case, Databricks Runtime Version is 14.3 LTS (includes Apache Spark 3.5.0, Scala 2.12) and Library version org.datasyslab:geospark:1.3.1 & org.datasyslab:geospark-sql_2.3:1.3.1

• Please note I can consider to downgrading my Databricks Runtime Version to one that includes Apache Spark 2.X and Scala 2.3 . Please suggest what would be the correct Databricks Runtime Version

@Anuj, Singh (Cognizant) - There are different methods to install packages in Azure Databricks:

GUI Method

Method1: Using libraries

To make third-party or locally-built code available to notebooks and jobs running on your clusters, you can install a library. Libraries can be written in Python, Java, Scala, and R. You can upload Java, Scala, and Python libraries and point to external packages in PyPI, Maven, and CRAN repositories.

Steps to install third party libraries:

Step1: Create Databricks Cluster.

Step2: Select the cluster created.

Step3: Select Libraries => Install New => Select Library Source = "Maven" => Coordinates => Search Packages => Select Maven Central => Search for the package required. Example: (GDAL) => Select the version (3.0.0) required => Install

Notebook methods

Method2: Using Cluster-scoped init scripts

Cluster-scoped init scripts are init scripts defined in a cluster configuration. Cluster-scoped init scripts apply to both clusters you create and those created to run jobs. Since the scripts are part of the cluster configuration, cluster access control lets you control who can change the scripts.

Step1: Add the DBFS path dbfs:/databricks/scripts/gdal_install.sh to the cluster init scripts

# --- Run 1x to setup the init script. ---
# Restart cluster after running.
dbutils.fs.put("/databricks/scripts/gdal_install.sh","""
#!/bin/bash
sudo add-apt-repository ppa:ubuntugis/ppa
sudo apt-get update
sudo apt-get install -y cmake gdal-bin libgdal-dev python3-gdal""",
True)

Step2: Restart the cluster after running step1 for the first time.

Method3: Python packages are installed in the Spark container using pip install.

Using pip to install "psutil" library.

CLI & API Methods

Method4: Libraries CLI

You run Databricks libraries CLI subcommands by appending them to databricks libraries.

databricks libraries -h

Install a JAR from DBFS:

databricks libraries install --cluster-id $CLUSTER_ID --jar dbfs:/test-dir/test.jar

Method6: Libraries API

The Libraries API allows you to install and uninstall libraries and get the status of libraries on a cluster.

Install libraries on a cluster. The installation is asynchronous - it completes in the background after the request. 2.0/libraries/install

Example Request:

{
  "cluster_id": "10201-my-cluster",
  "libraries": [
    {
      "jar": "dbfs:/mnt/libraries/library.jar"
    },
    {
      "egg": "dbfs:/mnt/libraries/library.egg"
    },
    {
      "whl": "dbfs:/mnt/libraries/mlflow-0.0.1.dev0-py2-none-any.whl"
    },
    {
      "whl": "dbfs:/mnt/libraries/wheel-libraries.wheelhouse.zip"
    },
    {
      "maven": {
        "coordinates": "org.jsoup:jsoup:1.7.2",
        "exclusions": ["slf4j:slf4j"]
      }
    },
    {
      "pypi": {
        "package": "simplejson",
        "repo": "https://my-pypi-mirror.com"
      }
    },
    {
      "cran": {
        "package": "ada",
        "repo": "https://cran.us.r-project.org"
      }
    }
  ]
}

Hope this helps. Do let us know if you any further queries.