Add PostgreSQL Database CDC source to an eventstream (preview)

This article shows you how to add a PostgreSQL Database Change Data Capture (CDC) source to an eventstream.

The PostgreSQL Database Change Data Capture (CDC) source connector for Microsoft Fabric event streams allows you to capture a snapshot of the current data in a PostgreSQL database. The connector then monitors and records any future row-level changes to this data. Once the changes are captured in the eventstream, you can process this CDC data in real-time and send it to different destinations within Fabric for further processing or analysis.

Prerequisites

• Access to the Fabric premium workspace with Contributor or higher permissions.

• Registered user access in the PostgreSQL database.

• Your PostgreSQL database must be publicly accessible and not be behind a firewall or secured in a virtual network.

• CDC enabled in the PostgreSQL database and tables.

  If you have Azure Database for PostgreSQL, follow the steps in the next section to enable CDC. For detailed information, see Logical replication and logical decoding - Azure Database for PostgreSQL - Flexible Server.

  For other PostgreSQL databases, see Debezium connector for PostgreSQL :: Debezium Documentation.

Enable CDC in your Azure Database for PostgreSQL

To enable CDC in your Azure Database for PostgreSQL Flexible Server, follow these steps:

1. On your Azure Database for PostgreSQL Flexible Server page in the Azure portal, select Server parameters in the navigation menu.

2. On the Server parameters page:

   • Set wal_level to logical.
   • Update the max_worker_processes to at least 16.

   

3. Save the changes and restart the server.

4. Confirm that your Azure Database for PostgreSQL Flexible Server instance allows public network traffic.

5. Grant the admin user replication permissions by running the following SQL statement. If you want to use other user account to connect your PostgreSQL DB to fetch CDC, please ensure the user is the table owner.

   ALTER ROLE <admin_user_or_table_owner_user> WITH REPLICATION;
   

Add Azure Database for PostgreSQL CDC as a source

1. Select Eventstream to create a new eventstream. Make sure the Enhanced Capabilities (preview) option is enabled.

   

2. On the next screen, select Add external source.

   

Configure and connect to Azure Database for PostgreSQL CDC

1. On the Select a data source screen, select Azure DB for PostgreSQL (CDC).

   

2. On the Connect page, select New connection.

   

3. In the Connection settings section, enter the following information.

   • Server: The server address of your PostgreSQL database, for example my-pgsql-server.postgres.database.azure.com.

   • Database: The database name, for example my_database.

     

4. Scroll down, and in the Connection credentials section, follow these steps.

   1. For Connection name, enter a name for the connection.

   2. For Authentication kind, select Basic.

      Note

      Currently, Fabric event streams supports only Basic authentication.

   3. Enter Username and Password for the database.

   4. Select Connect.

      

5. Now, on the Connect page, do these steps:

   1. For table, Enter a table name.

   2. For Port, enter the port number or leave the default value: 5432.

   3. Select Next at the bottom of the page.

      

6. On the Review and create screen, review the summary, and then select Add.

   

You can see the PostgreSQL Database CDC source added to your eventstream in Edit mode.

To implement this newly added PostgreSQL DB CDC source, select Publish. After you complete these steps, your PostgreSQL DB CDC source is available for visualization in the Live view.

Related content

Other connectors:

• Amazon Kinesis Data Streams
• Azure Cosmos DB
• Azure Event Hubs
• Azure IoT Hub
• Azure SQL Database Change Data Capture (CDC)
• Confluent Kafka
• Custom endpoint
• Google Cloud Pub/Sub
• MySQL Database CDC
• PostgreSQL Database CDC
• Sample data
• Azure Blob Storage events
• Fabric workspace event