Azure Synapse Analytics Pipeline Deployment Error Spark Configuration

Graeme Cash 76 Reputation points
2022-07-05T11:18:23.22+00:00

Hello

I have a YAML pipeline in Azure Dev Ops to deploy my Synapse Workspace from development to UAT. As soon as I add in a spark configuration the deployment fails with the following error...

[error]Encountered with exception:Error: For Artifact sparkConfiguration01: Failure in deployment: Skipped

This is a sample of the template generated after publishing in Synapse with a very basic configuration file...

        "name": "[concat(parameters('workspaceName'), '/sparkConfiguration01')]",  
        "type": "Microsoft.Synapse/workspaces/sparkConfigurations",  
        "apiVersion": "2019-06-01-preview",  
        "properties": {  
            "configs": {  
                "test": "1"  
            }

I haven't been able to get this working no matter what I try and I cannot get any additional detail about the error. Has anyone been able to get a Spark Configuration working with CI/CD?

Thanks

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,912 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Ingielewicz Michal 16 Reputation points
    2022-07-06T11:19:39.827+00:00

    We experience exactly the same. It's a BLOCKER in the Release part of CI/CD!
    We had to use the new spark configurations due to:
    Support is ending for user-provisioned Apache Spark configuration text files. Select a configuration from the Spark configuration dropdown list or create a new configuration in the manage hub in Synapse Studio. If no configuration is selected, jobs for this Spark pool will be run using the default configuration in the Spark system settings.

    So we've configured the Configuration, assigned it to the pools and it fails on the deployment like above with:

    Encountered with exception:Error: For Artifact elipseeSparkConfiguration: Failure in deployment: Skipped

    I've tried to work around it: exported the config from DEV, imported it to UAT - didn't help.
    I've assigned the imported config to the spark pools - didn't work either.

    There's no other workaround I can think of (other than removing the spark config, publishing, deploying and then re-applying the config manually again).

    Once again I feel like I'm Beta-Testing, very disappointing.

    1 person found this answer helpful.
  2. ShaikMaheer-MSFT 38,441 Reputation points Microsoft Employee
    2022-07-06T06:27:54.257+00:00

    Hi @Graeme Cash ,

    Thank you for posting query in Microsoft Q&A platform.

    Please note, Spark pools cannot be directly deployed in different environments using CI/CD process.

    Please feel free to check below offical documentation where its mentioned that Prepare pools before you migrate artifacts. If you have a SQL script or notebook attached to pools in the development workspace, use the same name for pools in different environments.
    https://video2.skills-academy.com/en-us/azure/synapse-analytics/cicd/continuous-integration-delivery#best-practices-for-cicd

    So kindly consider having your pool already ready in all environments before your deployment. Also, its best practice that to keep your pool name same.

    For any reason, if you wish you to keep your pool name different in different environments then you should consider including a step in your deployment process that executes a powershell script to replace pool names in template before deployment and then perform deployment. Click here to check more about that.

    Hope this helps. Please let us know if any further queries.

    -----
    Please consider hitting Accept Answer button. Accepted answers help community as well.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.