Azure IoT Operations built-in local MQTT broker

Azure IoT Operations features an enterprise-grade, standards-compliant MQTT broker that is scalable, highly available, and Kubernetes-native. It provides the messaging plane for Azure IoT Operations Preview, enables bi-directional edge/cloud communication and powers event-driven applications at the edge.

MQTT compliance

Message Queue Telemetry Transport (MQTT) has emerged as the lingua franca among protocols in the IoT space. MQTT's simple design allows a single broker to serve tens of thousands of clients simultaneously, with a lightweight publish-subscribe topic creation and management. Many IoT devices support MQTT natively out-of-the-box, with the long tail of IoT protocols being rationalized into MQTT by downstream translation gateways.

The MQTT broker underpins the messaging layer in IoT Operations and supports both MQTT v3.1.1 and MQTT v5. For more information about supported MQTT features, see MQTT feature support in MQTT broker.

Architecture

The MQTT broker has two major layers:

• Stateless frontend layer.
• Stateful and sharded backend layer.

The frontend layer handles client connections and requests and routes them to the backend. The backend layer partitions data by different keys, such as client ID for client sessions, and topic name for topic messages. It uses chain replication to replicate data within each partition.

The goals of the architecture are:

• Fault tolerance and isolation: Message publishing continues if backend pods fail and prevents failures from propagating to the rest of the system
• Failure recovery: Automatic failure recovery without operator intervention
• No message loss: Delivery of messages if at least one front-end pod and one backend pod in a partition is running
• Elastic scaling: Horizontal scaling of publishing and subscribing throughput to support edge and cloud deployments
• Consistent performance at scale: Limit message latency overhead due to chain-replication
• Operational simplicity: Minimum dependency on external components to simplify maintenance and complexity

Configuration

For configuration, the MQTT broker is composed of several Kubernetes custom resources that define different aspects of the broker's behavior and functionality.

• The main resource is Broker, which defines the global settings like cardinality, memory usage profile, and diagnostic settings.
• A Broker can have up to three BrokerListeners, each of which listens for incoming MQTT connections on the specified service type (NodePort, LoadBalancer, or ClusterIP). Each BrokerListener can have multiple ports.
• Each port within a BrokerListener can be associated with a BrokerAuthentication resource and a BrokerAuthorization resource. These are the authentication and authorization policies that determine which clients can connect to the port and what actions they can perform on the broker.

So, the relationship between Broker and BrokerListener is one-to-many, and the relationship between BrokerListener and BrokerAuthentication/BrokerAuthorization is many-to-many. The entity relationship diagram (ERD) for these resources is as follows:

By default, Azure IoT Operations deploys a default Broker, a default BrokerListener, and a default BrokerAuthentication. All of these resources are named default. Together, these resources provide a basic MQTT broker setup required for Azure IoT Operations to function. The default setup is as follows:

In a full deployment, you could have multiple BrokerListeners, each with multiple ports, and each port could have different BrokerAuthentication and BrokerAuthorization resources associated with it.

For example, starting from the default setup, you add:

• A LoadBalancer BrokerListener named example-lb-listener with two ports 1883 and 8883
• A NodePort BrokerListener named example-nodeport-listener with a single port 1884 (nodePort 31884)
• A BrokerAuthentication resource named example-authn, with a custom authentication method
• A BrokerAuthorization resource named example-authz, with your custom authorization settings

Then, if you configure all the new ports use the same BrokerAuthentication and BrokerAuthorization resources, the setup would look like:

This way, you keep the default setup intact and add new resources to customize the MQTT broker deployment to your needs.

Default Broker resource

Each Azure IoT Operations deployment can only have one Broker, and it must be named default. The default Broker resource is required for Azure IoT Operations to function. It's immutable and can't be modified after deployment.

Customize default Broker

Customizing the default Broker resource isn't required for most setups. The settings that require customization include:

• Cardinality: Determines the broker's capacity to handle more connections and messages, and it enhances high availability if there are pod or node failures.
• Memory profile: Sets the maximum memory usage of the broker and how to handle memory usage as the broker scales up.
• Disk-backed message buffer: Configuration for buffering messages to disk as RAM fills up.
• Diagnostics settings: Configuration for diagnostic settings like log level and metrics endpoint.
• Advanced MQTT client options: Configuration for advanced MQTT client options like session expiry, message expiry, and keep-alive settings.
• Encryption of internal traffic: Configuration for encrypting internal traffic between broker frontend and backend pods.

Customizing the default broker must be done during initial deployment time using Azure CLI or Azure portal. A new deployment is required if different Broker configuration settings are needed.

To customize the default Broker during deployment:

View default Broker settings

To view the settings for the default Broker:

az iot ops broker show --name default --instance <INSTANCE_NAME> --resource-group <RESOURCE_GROUP>

kubectl get broker default -n azure-iot-operations -o yaml

Next steps

Deploy Azure IoT Operations to an Arc-enabled Kubernetes cluster