Optimizing Azure Synapse Analytics for Spark Pool and Mapping Data Flows

Aditya Singh 135 Reputation points
2024-08-29T08:21:28.3033333+00:00

I am encountering errors involving Livy sessions when running multiple pipelines in Synapse Notebook and mapping data flows. The errors are as follows:

For notebooks:

  • Operation on target forecast_constrained failed: Exception: Failed to create Livy session for executing notebook. LivySessionId: 1392Notebook: raw_forecast_constrained_outbound. --> : Livy session has failed. Session state: Error. Session yields Failed result.
  • Operation on target data_cleaning_saleshistory failed: Exception: Failed to create Livy session for executing notebook. LivySessionId: 219Notebook: data_cleaning_saleshistory. --> CLUSTER_CREATION_TIMED_OUT: Livy session has failed. Session state: Error. Error code: CLUSTER_CREATION_TIMED_OUT. [plugins.azx-asa-mspbi-dp-p-01.azxaspd365dpp1.219 WorkspaceType: CCID:<90b4f648-6a6f-4520-8943-a0161a9d9874>] The job has surpassed the max number of seconds it can be in ResourceAcquisition state [2000s], likely due to the cluster creation timing out, so ending the job. Source: Dependency.

For mapping data flows:

  • Hit unexpected exception, please retry. If the problem persists, please contact Azure Support"_
  • `{ "errorCode": "BadRequest", "message": "Operation on target data_quality_outbound failed: [traceId=plugins.AZX-ASA-MSPBI-DP-P-01.General-8-1-7f9e6b13-5d0a-4061-b875-3e2cc54d3abb.3367d738-cbfb-4bc4-af04-8119bb70666f WorkspaceType:
Azure Logic Apps
Azure Logic Apps
An Azure service that automates the access and use of data across clouds without writing code.
3,087 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,858 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,566 questions
Accepted answer
  1. NIKHILA NETHIKUNTA 1,760 Reputation points Microsoft Vendor
    2024-09-03T14:09:45.0666667+00:00

    @Aditya Singh
    Thank you for reaching out to the community forum with your query.

    We apologize for the inconvenience caused by the recent outage. We want to inform you that an outage was reported in the Europe region, which may have impacted your notebooks. However, the issue has now been resolved, and the service has been restored.

    This can happen due to various reasons like network issues, credential failures, or service outages. Azure Data Factory has an internal recovery process that monitors all the runs and starts them when it notices something went wrong.

    Hope this helps. Do let us know if you any further queries.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

    0 comments

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.