Iceberg format in Azure Data Factory and Azure Synapse Analytics

  • Article

APPLIES TO: Azure Data Factory Azure Synapse Analytics

Tip

Try out Data Factory in Microsoft Fabric, an all-in-one analytics solution for enterprises. Microsoft Fabric covers everything from data movement to data science, real-time analytics, business intelligence, and reporting. Learn how to start a new trial for free!

Follow this article when you want to write the data into Iceberg format.

Iceberg format is supported for the following connectors:

You can use Iceberg dataset in Copy activity.

Dataset properties

For a full list of sections and properties available for defining datasets, see the Datasets article. This section provides a list of properties supported by the Iceberg format dataset.

Property Description Required
type The type property of the dataset must be set to Iceberg. Yes
location Location settings of the file(s). Each file-based connector has its own location type and supported properties under location. Yes

Below is an example of Iceberg dataset on Azure Data Lake Storage Gen2:

{
    "name": "IcebergDataset",
    "properties": {
        "type": "Iceberg",
        "linkedServiceName": {
            "referenceName": "<Azure Data Lake Storage Gen2 linked service name>",
            "type": "LinkedServiceReference"
        },
        "schema": [ < physical schema, optional, auto retrieved during authoring >
        ],
        "typeProperties": {
            "location": {
                "type": "AzureBlobFSLocation",
                "fileSystem": "filesystemname",
                "folderPath": "folder/subfolder",
            }
        }
    }
}

Copy activity properties

For a full list of sections and properties available for defining activities, see the Pipelines article. This section provides a list of properties supported by the Iceberg sink.

Iceberg as sink

The following properties are supported in the copy activity *sink* section.

Property Description Required
type The type property of the copy activity source must be set to IcebergSink. Yes
formatSettings A group of properties. Refer to Iceberg write settings table below. No
storeSettings A group of properties on how to write data to a data store. Each file-based connector has its own supported write settings under storeSettings. No

Supported Iceberg write settings under formatSettings:

Property Description Required
type The type of formatSettings must be set to IcebergWriteSettings. Yes

Here are some common connectors and formats related to the delimited text format:

Related content