insert record from azure datafactory into a delta table generated in databricks

Antonio Nuzzo 11 Reputation points
2022-03-02T12:08:37.73+00:00

Hi team, i'm trying to insert value into a delta table from datafactory, the table has been created in databricks and in the table properties i setted "delta.enableChangeDataFeed=true", for this reason the protocol minWriterVersion of my delta table is equal to 4, when i tried to insert value from data factory as the simple dataflow below the pipeline return this error: "

{"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: at Sink 'sink1': com.databricks.sql.transaction.tahoe.actions.InvalidProtocolVersionException: Delta protocol version is too new for this version of the Databricks Runtime. Please upgrade to a newer release.","Details":"com.databricks.sql.transaction.tahoe.actions.InvalidProtocolVersionException: Delta protocol version is too new for this version of the Databricks Runtime. Please upgrade to a newer release.\n\tat com.databricks.sql.transaction.tahoe.DeltaLog.protocolWrite(DeltaLog.scala:523)\n\tat com.databricks.sql.transaction.tahoe.OptimisticTransactionImpl$class.prepareCommit(OptimisticTransaction.scala:319)\n\tat com.databricks.sql.transaction.tahoe.OptimisticTransaction.prepareCommit(OptimisticTransaction.scala:80)\n\tat com.databricks.sql.transaction.tahoe.OptimisticTransactionImplEdge$$anonfun$commit$1.apply$mcJ$sp(OptimisticTransactionImplEdge.scala:185)\n\tat com.databricks.sql.transaction.tahoe.OptimisticTransactionImplEdge$$anonfun$commit$1.apply(OptimisticTransactionImplEdge.scala:180)\n\tat com.databricks.sql.transaction.tahoe.OptimisticTransactionImplEdge$$anonfun$commit$1.apply(OptimisticTransactionImplEdge.scala:180)\n\tat com.databricks.logging.UsageLogging$$anonfun$recordOperation$1.apply(UsageLogging.scala:417)\n\tat com."}

i think that the reason of this error is that datafactory by default set protocol version minwriterversion=2, instead the miwriterversion of my delta table is setted to 4, i would know if there is a way to set the minwriterwriter of the datafactory command to 4 when i execute the pipeline.

179273-deltatableupdate.png

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,045 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,015 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. KranthiPakala-MSFT 46,437 Reputation points Microsoft Employee
    2022-03-04T22:28:14.367+00:00

    Hello @Antonio Nuzzo ,

    Thanks for the question and using MS Q&A platform.

    As I understand the ask here is that you have set the Delta protocol version minWriterVersion to 4 and you are receiving the above error while trying to update/insert the table using ADF. Please correct me if I'm not clear.

    By looking at the error message my understanding is that your Databricks Runtime version is not compatible with the protocol version minWriterVersion to 4. Error says Delta protocol version is too new for this version of the Databricks Runtime. Please upgrade to a newer release. Which means you will have to upgrade the Databricks runtime to a compatible Delta protocol version.

    For more info please refer to this document - Table versioning - Features by protocol version

    180218-image.png

    Please note that Delta Lake guarantees backward compatibility. A higher version of Databricks Runtime is always able to read data that was written by a lower version.

    Delta Lake will occasionally break forward compatibility. Lower versions of Databricks Runtime may not be able to read and write data that was written by a higher version of Databricks Runtime. If you try to read and write to a table with a version of Databricks Runtime that is too low, you’ll get an error telling you that you need to upgrade.

    Note: Protocol upgrades are irreversible, therefore it is recommended you upgrade specific tables only when needed, such as to opt-in to new features in Delta Lake.

    Hope this information helps. Do let us know if any further queries.

    -----------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.