How to install jars /libraries related to spark-redis in azure databricks cluster?

Shivasai 21 Reputation points
2022-03-10T04:03:49.057+00:00

I am trying to connect to Azure cache for redis using python from azure databricks .

I have installed this package com.redislabs:spark-redis:2.3.0 from maven package in databricks. I have created a spark session with below code

SparkSession\
.builder\
.appName("myApp")\
.config("spark.redis.host", "my host")\
.config("spark.redis.port", "6379")\
.config("spark.redis.auth", "passwd")\
.getOrCreate()

But when I ran df.write.format("org.apache.spark.sql.redis").option("table", "people").option("key.column", "name").save()

I am getting below error.

Py4JJavaError: An error occurred while calling o390.save.
: java.lang.ClassNotFoundException:
Failed to find data source: org.apache.spark.sql.redis. Please find packages at
http://spark.apache.org/third-party-projects.html

Could you please let me know the detailed steps to install all necessary libraries/jars to access redis in databricks.

I have seen below code in spark-redis-python.md but I don't know how to run it in databricks.

$ ./bin/pyspark --jars <path-to>/spark-redis-<version>-jar-with-dependencies.jar

And also please let me know what is the latest spark-redis version.

Azure Cache for Redis
Azure Cache for Redis
An Azure service that provides access to a secure, dedicated Redis cache, managed by Microsoft.
251 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,175 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 89,466 Reputation points Microsoft Employee
    2022-03-10T11:21:30.653+00:00

    Hello @Shivasai ,

    Thanks for the question and using MS Q&A platform.

    Redis has a Spark Package that you can download and attach to your cluster

    The following notebook shows how to use Redis with Apache Spark in Azure Databricks.

    For more details, refer to Azure Databricks - Redis.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.