Autoloader in Synapse

Aditya Singh 160

I want to load data incrementally using Azure Synapse Analytics Notebook(dedicated spark pool) there is an option on databricks to load data incrementally which is autoloader so my question was can we use this option in Azure Synapse Analytics Notebook? This is my code :
This code works on Databricks but it not working on Azure Synapse Analytics Notebook:
Py4JJavaError: An error occurred while calling o4041.load. : java.lang.ClassNotFoundException: Failed to find data source: cloudFiles. Please find packages at https://spark.apache.org/third-party-projects.html

(spark.readStream
.format("cloudfiles")
.option("cloudfiles.format","csv")
.option("cloudfiles.schemaLocation",f"{output_path}/autoloader/schemalocation")
.load(f"{internal_path}")
.writeStream
.option("checkpointlocation",f"{output_path}/autoloader/checkpoint")
.option("path",f"{output_path}/autoloader/output")
.table("autoloader")
)

MADHUSUDAN PANWAR 86 Reputation points

2024-01-24T15:47:32.3533333+00:00

Hi Aditya, Did you find the solution? I am having a similar use case.
Aditya Singh 160 Reputation points

2024-01-25T07:28:55.8666667+00:00

Hi Madhusudan, No I am still looking for the solution or alternative
phemanth 11,125 Reputation points Microsoft Vendor

2024-01-25T10:07:50.0433333+00:00
@Aditya Singh
Thanks for the question and using MS Q&A platform.

the error message you received indicates that the cloudFiles data source is not found. This could be because the required package is not installed.

please try and install the package by running the following command in your Synapse workspace:

%pip install azure-storage-file-datalake

After installing the package, you can use the cloudFiles data source to load data incrementally in your Synapse notebook. Please note that the cloudFiles data source is only available in Delta Live Tables. If you are not using Delta Live Tables, you can use the COPY INTO statement to load data incrementally. if the issue doesn't resolve kindly provide more details about the error message you are receiving, such as the full stack trace, to help diagnose the issue further. I hope this helps! Let us know if you have any further questions.
Aditya Singh 160 Reputation points

2024-01-25T11:07:16.3066667+00:00

Still getting the same error @phemanth
phemanth 11,125 Reputation points Microsoft Vendor

2024-01-29T11:33:59.8633333+00:00

@Aditya Singh kindly provide more details about the error message you are receiving, such as the full stack trace, and screenshots, if possible, to help diagnose the issue further.
phemanth 11,125 Reputation points Microsoft Vendor

2024-01-30T10:05:30.3366667+00:00

@Aditya Singh We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Share via

Autoloader in Synapse

Your answer