Add Amazon Managed Streaming for Apache Kafka (MSK) as source in Real-Time hub (preview)

  • Article

This article describes how to add Amazon Streaming for Apache Kafka (MSK) as an event source in Fabric Real-Time hub.

Amazon MSK Kafka is a fully managed Kafka service that simplifies the setup, scaling, and management. By integrating Amazon MSK Kafka as a source within your eventstream, you can seamlessly bring the real-time events from your MSK Kafka and process it before routing them to multiple destinations within Fabric. 

Note

Real-Time hub is currently in preview.

Prerequisites

  • Get access to the Fabric premium workspace with Contributor or above permissions.
  • An Amazon MSK Kafka cluster in active status.
  • Your Amazon MSK Kafka cluster must be publicly accessible and not be behind a firewall or secured in a virtual network.

Launch Add source experience

  1. Sign in to Microsoft Fabric.

  2. Select Real-Time hub on the left navigation bar.

  3. On the Real-Time hub page, select + Add source in the top-right corner of the page.

    Screenshot that shows how to launch Real-Time hub In Microsoft Fabric.

Add Amazon Managed Streaming for Apache Kafka as a source

  1. On the Select a data source page, select Amazon MSK Kafka.

    Screenshot that shows the selection of Amazon MSK Kafka as the source type in the Get events wizard.

  2. On the Connect page, select New connection.

    Screenshot that shows the selection of the New connection link on the Connect page of the Get events wizard.

  3. In the Connection settings section, for Bootstrap Server, enter the public endpoint of your Kafka cluster.

    Screenshot that shows the selection of the Bootstrap server field on the Connect page of the Get events wizard.

    To get the public endpoint:

    Screenshot that shows the public endpoint of Amazon MSK cluster.

  4. In the Connection credentials section, If you have an existing connection to the Amazon MSK Kafka cluster, select it from the drop-down list for Connection. Otherwise, follow these steps:

    1. For Connection name, enter a name for the connection.
    2. For Authentication kind, confirm that API Key is selected.
    3. For Key and Secret, enter API key and key Secret for Amazon MSK Kafka cluster.

  5. Select Connect. 

  6. Now, on the Connect page, follow these steps.

    1. For Topic, enter the Kafka topic.

    2. For Consumer group, enter the consumer group of your Kafka cluster. This field provides you with a dedicated consumer group for getting events.

    3. Select Reset auto offset to specify where to start reading offsets if there's no commit.

    4. For Security protocol, the default value is SASL_SSL. The default SASL mechanism is SCRAM-SHA-512 and cannot be changed.

      Screenshot that shows the first page of the Amazon MSK Kafka connection settings.

  7. Select Next. On the Review and create screen, review the summary, and then select Add.

View data stream details

  1. On the Review and create page, if you select Open eventstream, the wizard opens the eventstream that it created for you with the selected Amazon Managed Streaming for Apache Kafka source. To close the wizard, select Close at the bottom of the page.

  2. In Real-Time hub, switch to the Data streams tab of Real-Time hub. Refresh the page. You should see the data stream created for you.

    For detailed steps, see View details of data streams in Fabric Real-Time hub.

Related content

To learn about consuming data streams, see the following articles: