How to configure Apache Kafka on HDInsight to automatically create topics

  • Article

By default, Apache Kafka on HDInsight doesn't enable automatic topic creation. You can enable auto topic creation for existing clusters using Apache Ambari. You can also enable auto topic creation when creating a new Kafka cluster using an Azure Resource Manager template.

Apache Ambari Web UI

To enable automatic topic creation on an existing cluster through the Ambari Web UI, use the following steps:

  1. From the Azure portal, select your Kafka cluster.

  2. From Cluster dashboards, select Ambari home.

    Image of the portal with cluster dashboard selected.

    When prompted, authenticate using the login (admin) credentials for the cluster. Instead, you can connect to Ambari directly from https://CLUSTERNAME.azurehdinsight.net/ where CLUSTERNAME is the name of your Kafka cluster.

  3. Select the Kafka service from the list on the left of the page.

    Apache Ambari service list tab.

  4. Select Configs in the middle of the page.

    Apache Ambari service configs tab.

  5. In the Filter field, enter a value of auto.create.

    Apache Ambari search filter field.

    This setting filters the list of properties and displays the auto.create.topics.enable setting.

  6. Change the value of auto.create.topics.enable to true, and then select Save. Add a note, and then select Save again.

    Image of the auto.create.topics.enable entry.

  7. Select the Kafka service, select Restart, and then select Restart all affected. When prompted, select Confirm restart all.

    `Apache Ambari restart all affected`.

Note

You can also set Ambari values through the Ambari REST API. This is generally more difficult, as you have to make multiple REST calls to retrieve the current configuration, modify it, etc. For more information, see the Managed HDInsight clusters using the Apache Ambari REST API document.

Resource Manager templates

When creating a Kafka cluster using an Azure Resource Manager template, you can directly set auto.create.topics.enable by adding it in a kafka-broker. The following JSON snippet demonstrates how to set this value to true:

"clusterDefinition": {
    "kind": "kafka",
    "configurations": {
        "gateway": {
            "restAuthCredential.isEnabled": true,
            "restAuthCredential.username": "[parameters('clusterLoginUserName')]",
            "restAuthCredential.password": "[parameters('clusterLoginPassword')]"
        },
        "kafka-broker": {
            "auto.create.topics.enable": "true"
        }
    }
}

Next Steps

In this document, you learned how to enable automatic topic creation for Apache Kafka on HDInsight. To learn more about working with Kafka, see the following links: