Azure Container Networking Interface (CNI) Pod Subnet

Azure CNI Pod Subnet assigns IP addresses to pods from a separate subnet from your cluster Nodes. This feature is available in two modes: Dynamic IP Allocation and Static Block Allocation (Preview).

Prerequisites

• Review the prerequisites for configuring basic Azure CNI networking in AKS, as the same prerequisites apply to this article.

• Review the deployment parameters for configuring basic Azure CNI networking in AKS, as the same parameters apply.

• AKS Engine and DIY clusters aren't supported.

• Azure CLI version 2.37.0 or later and the aks-preview extension version 2.0.0b2 or later.

• Register the subscription-level feature flag for your subscription: 'Microsoft.ContainerService/AzureVnetScalePreview'.

• If you have an existing cluster, you need to enable the Container Insights for monitoring IP subnet usage add-on. You can enable Container Insights using the az aks enable-addons command, as shown in the following example:

  az aks enable-addons --addons monitoring --name <cluster-name> --resource-group <resource-group-name>
  

Dynamic IP allocation mode

Dynamic IP allocation helps mitigate pod IP address exhaustion issues by allocating pod IPs from a subnet that's separate from the subnet hosting the AKS cluster.

The dynamic IP allocation mode offers the following benefits:

• Better IP utilization: IPs are dynamically allocated to cluster Pods from the Pod subnet. This leads to better utilization of IPs in the cluster compared to the traditional CNI solution, which does static allocation of IPs for every node.
• Scalable and flexible: Node and pod subnets can be scaled independently. A single pod subnet can be shared across multiple node pools of a cluster or across multiple AKS clusters deployed in the same VNet. You can also configure a separate pod subnet for a node pool.
• High performance: Since pods are assigned VNet IPs, they have direct connectivity to other cluster pods and resources in the VNet. The solution supports very large clusters without any degradation in performance.
• Separate VNet policies for pods: Since pods have a separate subnet, you can configure separate VNet policies for them that are different from node policies. This enables many useful scenarios, such as allowing internet connectivity only for pods and not for nodes, fixing the source IP for pod in a node pool using an Azure NAT Gateway, and using network security groups (NSGs) to filter traffic between node pools.
• Kubernetes network policies: Both the Azure Network Policies and Calico work with this mode.

Plan IP addressing

With dynamic IP allocation, nodes and pods scale independently, so you can plan their address spaces separately. Since pod subnets can be configured to the granularity of a node pool, you can always add a new subnet when you add a node pool. The system pods in a cluster/node pool also receive IPs from the pod subnet, so this behavior needs to be accounted for.

IPs are allocated to nodes in batches of 16. Pod subnet IP allocation should be planned with a minimum of 16 IPs per node in the cluster, as the nodes request 16 IPs on startup and request another batch of 16 anytime there are <8 IPs unallocated in their allotment.

IP address planning for Kubernetes services and Docker Bridge remain unchanged.

Static block allocation mode (Preview)

Static block allocation helps mitigate potential pod subnet sizing and Azure address mapping limitations by assigning CIDR blocks to nodes rather than individual IPs.

The static block allocation mode offers the following benefits:

• Better IP scalability: CIDR blocks are statically allocated to the cluster nodes and are present for the lifetime of the node, as opposed to the traditional dynamic allocation of individual IPs with traditional CNI. This enables routing based on CIDR blocks and helps scale the cluster limit up to 1 million pods from the traditional 65K pods per cluster. Your Azure Virtual Network must be large enough to accommodate the scale of your cluster.
• Flexibility: Node and pod subnets can be scaled independently. A single pod subnet can be shared across multiple node pools of a cluster or across multiple AKS clusters deployed in the same VNet. You can also configure a separate pod subnet for a node pool.
• High performance: Since pods are assigned virtual network IPs, they have direct connectivity to other cluster pods and resources in the VNet.
• Separate VNet policies for pods: Since pods have a separate subnet, you can configure separate VNet policies for them that are different from node policies. This enables many useful scenarios such as allowing internet connectivity only for pods and not for nodes, fixing the source IP for pod in a node pool using an Azure NAT Gateway, and using NSGs to filter traffic between node pools.
• Kubernetes network policies: Cilium, Azure NPM, and Calico work with this solution.

Limitations

Below are some of the limitations of using Azure CNI Static Block allocation:

• Minimum Kubernetes Version required is 1.28
• Maximum subnet size supported is x.x.x.x/12 ~ 1 million IPs
• Not supported for Windows node pools
• Not supported for Cilium Data Plane
• Only a single mode of operation can be used per subnet. If a subnet uses Static Block allocation mode, it cannot be use Dynamic IP allocation mode in a different cluster or node pool with the same subnet and vice versa.
• Only supported in new clusters or when adding node pools with a different subnet to existing clusters. Migrating or updating existing clusters or node pools is not supported.
• Across all the CIDR blocks assigned to a node in the node pool, one IP will be selected as the primary IP of the node. Thus, for network administrators selecting the --max-pods value try to use the calculation below to best serve your needs and have optimal usage of IPs in the subnet:

max_pods = (N * 16) - 1 where N is any positive integer and N > 0

Region availability

This feature is not available in the following regions:

• US South
• East US 2
• West US
• West US 2

Plan IP addressing

With static block allocation, nodes and pods scale independently, so you can plan their address spaces separately. Since pod subnets can be configured to the granularity of a node pool, you can always add a new subnet when you add a node pool. The system pods in a cluster/node pool also receive IPs from the pod subnet, so this behavior needs to be accounted for.

CIDR blocks of /28 (16 IPs) are allocated to nodes based on your --max-pods configuration for your node pool, which defines the maximum number of pods per node. 1 IP is reserved on each node from all the available IPs on that node for internal purposes.

While planning your IPs, it's important to define your --max-pods configuration using the following calculation: max_pods_per_node = (16 * N) - 1, where N is any positive integer greater than 0.

Ideal values with no IP wastage would require the max pods value to conform to the above expression.

See the following example cases:

IP address planning for Kubernetes services remains unchanged.