增量实时表 CLI（旧版）

  • 项目

重要

本文档已过时，将来可能不会更新。

此信息适用于旧版 Databricks CLI 0.18 及更低版本。 Databricks 建议改用较新的 Databricks CLI 0.205 或更高版本。 请参阅什么是 Databricks CLI？。 若要查找你的 Databricks CLI 的版本，请运行 databricks -v。

要从 Databricks CLI 版本 0.18 或更低版本迁移到 Databricks CLI 版本 0.205 或更高版本，请参阅 Databricks CLI 迁移。

可以通过将 Delta Live Tables CLI 子命令追加到 databricks pipelines 后面来运行它们。 这些子命令调用增量实时表 API。

databricks pipelines -h

Usage: databricks pipelines [OPTIONS] COMMAND [ARGS]...

  Utility to interact with Databricks Delta Live Tables Pipelines.

Options:
  -v, --version   0.18.0
  --debug         Debug Mode. Shows full stack trace on error.
  --profile TEXT  CLI connection profile to use. The default profile is
                  "DEFAULT".

  -h, --help      Show this message and exit.

Commands:
  create  Creates a pipeline.
  delete  Deletes the pipeline and cancels any active updates.
  deploy  [Deprecated] This command is deprecated, use create and edit
          commands instead.  Creates or edits a pipeline specified by the
          pipeline settings.

  edit    Edits a pipeline.
  get     Gets a pipeline's current settings and status.
  list    Lists all pipelines and their statuses.
  reset   [Deprecated] Use the "start --full-refresh" command instead. Resets
          a pipeline so that data can be reprocessed from the beginning.

  run     [Deprecated] Use the "start" command instead. Starts a pipeline
          update.

  start   Starts a pipeline update.
  stop    Stops the pipeline by cancelling any active update.

创建管道

若要显示使用情况文档，请运行 databricks pipelines create --help。

使用情况

databricks pipelines create --settings create-pipeline.json

create-pipeline.json：

{
    "name": "Wikipedia pipeline (SQL)",
    "storage": "/Users/username/data",
    "clusters": [
      {
        "label": "default",
        "autoscale": {
          "min_workers": 1,
          "max_workers": 5
        }
      }
    ],
    "libraries": [
      {
        "notebook": {
          "path": "/Users/username/Delta Live Tables quickstart (SQL)"
        }
      }
    ],
    "target": "wikipedia_analysis",
    "continuous": false
}

Successfully created pipeline: https://<databricks-instance>/#joblist/pipelines/4ee6b5c2-7017-11ec-90d6-0242ac120003 with ID: 4ee6b5c2-7017-11ec-90d6-0242ac120003.

编辑管道

若要显示使用情况文档，请运行 databricks pipelines edit --help。

使用情况

databricks pipelines edit --settings edit-pipeline.json

edit-pipeline.json：

{
    "id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
    "name": "Wikipedia pipeline (SQL)",
    "storage": "/Users/username/data",
    "clusters": [
        {
            "label": "default",
            "autoscale": {
                "min_workers": 1,
                "max_workers": 3
            }
        }
    ],
    "libraries": [
        {
            "notebook": {
                "path": "/Users/username/Delta Live Tables quickstart (SQL)"
            }
        }
    ],
    "target": "wikipedia_quickstart_data",
    "continuous": false
}

Successfully edited pipeline settings: https://<databricks-instance>/#joblist/pipelines/4ee6b5c2-7017-11ec-90d6-0242ac120003.

删除管道

若要显示使用情况文档，请运行 databricks pipelines delete --help。

使用情况

databricks pipelines delete --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003

Pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003 deleted

获取有关管道的信息

若要显示使用情况文档，请运行 databricks pipelines get --help。

使用情况

databricks pipelines get --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003

{
  "pipeline_id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
  "spec": {
    "id": "4ee6b5c2-7017-11ec-90d6-0242ac120003",
    "name": "Wikipedia pipeline (SQL)",
    "storage": "/Users/username/data",
    "clusters": [
      {
        "label": "default",
        "autoscale": {
          "min_workers": 1,
          "max_workers": 5
        }
      }
    ],
    "libraries": [
      {
        "notebook": {
          "path": "/Users/username/Delta Live Tables quickstart (SQL)"
        }
      }
    ],
    "target": "wikipedia_analysis",
    "continuous": false
  },
  "state": "IDLE",
  "name": "Wikipedia pipeline (SQL)",
  "creator_user_name": "username@databricks.com",
  "last_modified": 1641600056265
}

列出所有管道及其状态信息

若要显示使用情况文档，请运行 databricks pipelines list --help。

使用情况

databricks pipelines list

[
  {
    "pipeline_id": "8a93e1b4-7024-11ec-90d6-0242ac120003",
    "state": "IDLE",
    "name": "Retail sales pipeline (SQL)",
    "latest_updates": [
      {
        "update_id": "9df03924-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-29T16:18:57.982Z"
      },
      {
        "update_id": "a57e601c-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-28T18:19:30.371Z"
      }
    ],
    "creator_user_name": "user@databricks.com"
  },
  {
    "pipeline_id": "b46e2670-7024-11ec-90d6-0242ac120003",
    "state": "IDLE",
    "name": "DLT quickstart (Python)",
    "latest_updates": [
      {
        "update_id": "bcd8fa2e-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-12-16T18:19:25.827Z"
      },
      {
        "update_id": "c2c7a2c8-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-29T22:22:32.586Z"
      },
      {
        "update_id": "c88d2818-7024-11ec-90d6-0242ac120003",
        "state": "COMPLETED",
        "creation_time": "2021-10-29T22:16:18.258Z"
      }
    ],
    "creator_user_name": "user@databricks.com"
  }
]

启动管道更新

若要显示使用情况文档，请运行 databricks pipelines start --help。

databricks pipelines start --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003

Started an update c6462e2e-beb7-4c24-a9d3-962eaa19781b for pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.

使用完全刷新启动管道更新

若要显示使用情况文档，请运行 databricks pipelines start --help。

使用情况

databricks pipelines start --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003 --full-refresh

Started an update ed885d3d-4fc0-44fd-b5dd-59471c789327 with full refresh for pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.

停止管道更新

若要显示使用情况文档，请运行 databricks pipelines stop --help。

使用情况

databricks pipelines stop --pipeline-id 4ee6b5c2-7017-11ec-90d6-0242ac120003

Stopped pipeline 4ee6b5c2-7017-11ec-90d6-0242ac120003.