@@ -9,6 +9,10 @@ AKS preview features are available on a self-service, opt-in basis. Previews are  This article helps you get started with Azure Kubernetes Application Network for AKS. It provides step-by-step instructions to create an Application Network resource and connect an AKS cluster to it as a member cluster. +Note++This quickstart creates a public AKS cluster. For an eligible private AKS cluster, follow [Connect private AKS clusters](private-connectivity) to check availability and choose the subnet and join options. Public clusters can't use the private-connect path.+ ## Prerequisites  - An Azure subscription. If you don't have one, create a [free account](https://azure.microsoft.com/pricing/purchase-options/azure-account?cid=msft_learn).@@ -20,6 +24,40 @@ This article helps you get started with Azure Kubernetes Application Network for     - [OIDC issuer](/en-us/azure/aks/use-oidc-issuer) is enabled.     - [Managed Kubernetes Gateway API installation](/en-us/azure/aks/managed-gateway-api) is enabled. - Verify support for your existing Kubernetes cluster using the [support policy](supported-versions).+- Meet the network access requirements for your cluster's API server and data plane.++## Network access requirements++Application Network needs connectivity in both directions. Allowing one path doesn't enable the other:++| Connection | Required access |+| --- | --- |+| Application Network managed control plane to the member's Kubernetes API server | For a public API server, permit the Application Network source addresses through any API-server access restrictions. For a private API server, follow the [private-connectivity requirements](private-connectivity). |+| Member cluster data plane to its Application Network managed control plane (MCP) | Allow DNS resolution and outbound TLS connections to the member's MCP FQDN on **TCP 443**. These connections carry configuration discovery and workload certificate requests. |++Meet these requirements in addition to the [standard AKS outbound network requirements](../aks/outbound-rules-control-egress).++### Public API-server authorized IP ranges++If a public AKS cluster uses [API-server authorized IP ranges](../aks/api-server-authorized-ip-ranges), the allowed sources must include Application Network's managed control-plane egress addresses. Application Network uses the **`CloudAppLink`** service tag for these addresses.++Important++Using a service-tag name in AKS authorized IP ranges requires separate enablement for `Microsoft.ContainerService/EnableServiceTagAuthorizedIPPreview`. Public API Server VNet Integration configurations can have additional service-tag availability requirements. Confirm the supported configuration with the Application Network and AKS teams before using the tag.++AppNet `PublicPreview` or `PrivatePath` registration doesn't enable AKS service-tag support. Don't assume that `CloudAppLink` is available through the standard public service-tag catalog or that the AKS authorized IP ranges setting accepts tag names without this enablement.++When updating authorized IP ranges, retain the existing approved administrator, automation, and cluster egress ranges. An update replaces the configured list; adding Application Network access must not remove those entries. If service-tag entries aren't available for your subscription, obtain the supported source-address configuration from the service team rather than using a guessed CIDR or opening the API server to all sources.++This requirement applies to a **public** API server. It doesn't use `connectivityProfile.privateConnect` or enable the private-connect path for public AKS clusters.++### Data plane outbound access to the MCP++If the member cluster has a firewall, proxy, network policy, or other egress restriction, allow its data plane to resolve and reach the assigned MCP hostname over TLS on TCP 443. This requirement applies to both public and private member clusters. Private connectivity to the Kubernetes API server doesn't replace the data-plane-to-MCP connection.++Use the member's actual hostname, not a copied example, a fixed IP address, or a metrics URL including its scheme and path. See Get the managed control plane hostname for the current endpoint discovery method.++Plan this access before onboarding. If initial provisioning is blocked before the assigned endpoint is available, coordinate the hostname and bootstrap access requirements with the Application Network team. Don't treat an empty endpoint as permission to use an unrestricted outbound rule.  ## Set environment variables @@ -125,6 +163,8 @@ If you have an existing cluster that already meets these requirements, you can s  When you join a member cluster to Azure Kubernetes Application Network, you can specify one of the following upgrade modes for that cluster: **fully-managed** or **self-managed**. The upgrade mode determines how minor version upgrades of Azure Kubernetes Application Network are applied to the member cluster. If you don't specify an upgrade mode during member join, it defaults to `SelfManaged`. For more information about upgrade modes, see [Configure upgrades for Azure Kubernetes Application Network members](upgrades). +For multi-cluster deployments, [choose your connectivity topology](multi-cluster-connectivity) before joining members. Network names describe whether members share direct pod-IP connectivity or communicate through east-west gateways.+ Note  You can't change a member's parent Application Network, location, subscription, or resource group after you join it. To change any of these, remove the member and rejoin it with the new values.@@ -142,7 +182,7 @@ You can't change a member's parent Application Network, location, subscription,  ## Verify member cluster connectivity -- After joining the cluster, you can verify connectivity and view member details using the [`az appnet member show`](/en-us/cli/azure/appnet/member#az-appnet-member-show) command.+- After joining the cluster, check member provisioning and view its details by using the [`az appnet member show`](/en-us/cli/azure/appnet/member#az-appnet-member-show) command. A successful ARM operation alone doesn't verify application traffic or all network paths.      ```azurecli     az appnet member show --resource-group $APPNET_RG --appnet-name $APPNET_NAME --member-name $APPNET_MEMBER_NAME@@ -183,6 +223,23 @@ You can't change a member's parent Application Network, location, subscription,     }     ``` +### Get the managed control plane hostname++- Read the member's endpoint URL by using the [`az appnet member show`](/en-us/cli/azure/appnet/member#az-appnet-member-show) command:++    ```azurecli+    az appnet member show \+        --resource-group $APPNET_RG \+        --appnet-name $APPNET_NAME \+        --member-name $APPNET_MEMBER_NAME \+        --query properties.observabilityProfile.metrics.metricsEndpoint \+        -o tsv+    ```++    `metricsEndpoint` is a read-only metrics URL. Its hostname currently identifies the same MCP endpoint used for configuration discovery and certificate requests. For an FQDN-based egress rule, use only that hostname, without `https://`, a path, or a port suffix, and permit TCP 443.++    Don't set this property in a join or update request. If the returned value is empty during initial or failed provisioning, the endpoint isn't exposed yet. Follow the bootstrap access guidance rather than inventing a destination.+ ## List Azure Kubernetes Application Network members  Multiple clusters can join an Azure Kubernetes Application Network.

@@ -1,25 +1,18 @@ # GPU observability in Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Learn -Efficient placement and optimization of GPU workloads often requires visibility into resource utilization and performance. Managed GPU metrics on AKS (preview) provide automated collection and exposure of GPU utilization, memory, and performance data across NVIDIA GPU-enabled node pools. This enables platform administrators to optimize cluster resources and developers to tune and debug workloads with limited manual instrumentation.+Efficient placement and optimization of GPU workloads often requires visibility into resource utilization and performance. Managed GPU metrics on AKS provide automated collection and exposure of GPU utilization, memory, and performance data across NVIDIA GPU-enabled node pools. This capability enables platform administrators to optimize cluster resources and developers to tune and debug workloads with limited manual instrumentation. -In this article, you learn about GPU metrics collected by the NVIDIA Data Center GPU Manager [(DCGM) exporter](https://github.com/NVIDIA/dcgm-exporter/tree/main) with [a fully managed GPU-enabled node pool (preview)](aks-managed-gpu-nodes) in Azure Kubernetes Service (AKS).--Important--AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:--- [AKS support policies](/en-us/azure/aks/support-policies)-- [Azure support FAQ](/en-us/azure/aks/faq)+In this article, you learn about GPU metrics collected by the NVIDIA Data Center GPU Manager [(DCGM) exporter](https://github.com/NVIDIA/dcgm-exporter/tree/main) with [a managed GPU-enabled node pool](aks-managed-gpu-nodes) in Azure Kubernetes Service (AKS).  ## Prerequisites -- An AKS cluster with [a fully managed GPU-enabled node pool (preview)](aks-managed-gpu-nodes) and ensure that the GPUs are schedulable.+- An AKS cluster with [a managed GPU-enabled node pool](aks-managed-gpu-nodes) and ensure that the GPUs are schedulable. - A [sample GPU workload](use-nvidia-gpu#run-a-gpu-enabled-workload) deployed to your node pool. - [Azure Managed Prometheus](/en-us/azure/azure-monitor/containers/kubernetes-monitoring-enable?tabs=cli#enable-with-cli) enabled on your AKS cluster.  ## Verify that managed GPU components are installed -After creating your managed NVIDIA GPU node pool (preview) following [these instructions](aks-managed-gpu-nodes), confirm that the GPU software components were installed with the [az aks nodepool show](/en-us/cli/azure/aks/nodepool#az-aks-nodepool-show) command:+After creating your managed NVIDIA GPU node pool following [these instructions](aks-managed-gpu-nodes), confirm that the GPU software components were installed by using the [az aks nodepool show](/en-us/cli/azure/aks/nodepool#az-aks-nodepool-show) command:  ```azurecli az aks nodepool show \

@@ -1,12 +1,12 @@-# Troubleshoot AKS API Server Authorized IP Range Access - Azure | Microsoft Learn+# Troubleshoot AKS API server authorized IP range access - Azure | Microsoft Learn  ## Summary -This article discusses how to resolve a scenario in which you can't use authorized IP address ranges to access the API server for a Microsoft Azure Kubernetes Service (AKS) cluster.+This article helps you troubleshoot AKS API server access problems when authorized IP address ranges block your connection. Follow the steps to update the allowed ranges and restore `kubectl` access to your cluster.  ## Symptoms -If you try to create or manage resources in an AKS cluster, you can't access the cluster API server. When you run `kubectl`, you receive the following error message:+If you try to create or manage resources in an AKS cluster, you can't access the cluster API server. When you run `kubectl`, you receive the following error message.  > > Unable to connect to the server: dial tcp x.x.x.x:443: i/o timeout@@ -17,8 +17,8 @@ You configured your AKS cluster to access the cluster API server by using author  ## Solution -Make sure that when you run the [az aks create](/en-us/cli/azure/aks#az-aks-create) or [az aks update](/en-us/cli/azure/aks#az-aks-update) command in [Azure CLI](/en-us/cli/azure/install-azure-cli), the `--api-server-authorized-ip-ranges` parameter includes the IP addresses or IP address ranges of the automation, development, or tooling systems that are being used.+Make sure that when you run the [az aks create](/en-us/cli/azure/aks#az-aks-create) or [az aks update](/en-us/cli/azure/aks#az-aks-update) command in [Azure CLI](/en-us/cli/azure/install-azure-cli), the `--api-server-authorized-ip-ranges` parameter includes the IP addresses or IP address ranges of the automation, development, or tooling systems that you're using. -## More information+## References  - [How to find my IP to include in --api-server-authorized-ip-ranges?](/en-us/azure/aks/api-server-authorized-ip-ranges#how-to-find-my-ip-to-include-in---api-server-authorized-ip-ranges)

@@ -196,7 +196,7 @@ Fleet Manager resource placement supports the following placement policy types f  #### PickFixed placement type -Use `PickFixed` to select the clusters by name. Supply names in the `clusterNames` array.+Use `PickFixed` to select the clusters by name. Supply names in the `clusterNames` array. An empty or omitted array selects no clusters.  ::: zone pivot="cluster-scope" @@ -249,6 +249,16 @@ spec:  ::: zone-end +To create a placement without target clusters or scale an existing placement down to zero clusters, use the following policy under `spec`:++```yaml+policy:+  placementType: PickFixed+  clusterNames: []+```++When you clear the array on an existing placement, Fleet Manager removes the resources previously distributed by that placement from the member clusters. The placement object and selected resources on the hub cluster remain. To distribute the resources again, add target cluster names back to `clusterNames`.+ #### PickAll placement type  Use `PickAll` to distribute resources across all member clusters, or all clusters that match a criteria you specify.@@ -699,7 +709,7 @@ Each toleration is used to tolerate one or more specific taints applied on a `Me ::: zone pivot="cluster-scope"  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata:   name: test-ns

@@ -6,19 +6,12 @@ Azure Kubernetes Service (AKS) is a managed Kubernetes service that you can use - Add a Linux-based GPU node pool with managed GPU nodes enabled. - Verify that AKS installed and configured the GPU software stack for scheduling. -Fully managed GPU nodes are a preview feature. When you enable managed GPU nodes, AKS installs and manages the NVIDIA GPU driver, NVIDIA Kubernetes device plugin, Data Center GPU Manager (DCGM) metrics exporter, and GPU health monitoring components for the GPU node pool. Managed GPU nodes also integrate real-time GPU metrics for ingestion in Azure Managed Prometheus and Azure Monitor managed service for Prometheus. For more information, see [Create a fully managed GPU node pool on Azure Kubernetes Service (AKS) (preview)](../aks-managed-gpu-nodes) and [GPU observability in Azure Kubernetes Service (AKS)](../monitor-gpu-metrics).+When you enable managed GPU nodes, AKS installs and manages the NVIDIA GPU driver, NVIDIA Kubernetes device plugin, Data Center GPU Manager (DCGM) metrics exporter, and GPU health monitoring components for the GPU node pool. Managed GPU nodes also integrate real-time GPU metrics for ingestion in Azure Managed Prometheus and Azure Monitor managed service for Prometheus. For more information, see [Create a managed GPU node pool on Azure Kubernetes Service (AKS)](../aks-managed-gpu-nodes) and [GPU observability in Azure Kubernetes Service (AKS)](../monitor-gpu-metrics).  Note  This article includes steps to deploy a cluster for evaluation purposes only. Before you deploy a production-ready cluster, familiarize yourself with the [baseline reference architecture](/en-us/azure/architecture/reference-architectures/containers/aks/baseline-aks?toc=/azure/aks/toc.json&bc=/azure/aks/breadcrumb/toc.json) to consider how it aligns with your business requirements. -Important--AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:--- [AKS support policies](/en-us/azure/aks/support-policies)-- [Azure support FAQ](/en-us/azure/aks/faq)- ## Before you begin  This quickstart assumes a basic understanding of Kubernetes concepts. For more information, see [Kubernetes core concepts for Azure Kubernetes Service (AKS)](../concepts-clusters-workloads).@@ -34,46 +27,12 @@ This quickstart assumes a basic understanding of Kubernetes concepts. For more i     - When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see [Use and manage extensions with the Azure CLI](/en-us/cli/azure/azure-cli-extensions-overview).     - Run [az version](/en-us/cli/azure/reference-index?#az-version) to find the version and dependent libraries that are installed. To upgrade to the latest version, run [az upgrade](/en-us/cli/azure/reference-index?#az-upgrade). -- You need Azure CLI version 2.85.0 or later. To find the version, run `az --version`. If you need to install or upgrade Azure CLI, see [Install Azure CLI](/en-us/cli/azure/install-azure-cli).+- You need Azure CLI version 2.92.0 or later. To find the version, run `az --version`. If you need to install or upgrade Azure CLI, see [Install Azure CLI](/en-us/cli/azure/install-azure-cli). - Ensure that the identity you're using to create your cluster has the appropriate minimum permissions. For more information on access and identity for AKS, see [Access and identity options for Azure Kubernetes Service (AKS)](../concepts-identity). - If you have multiple Azure subscriptions, select the appropriate subscription ID for billing by using the [az account set](/en-us/cli/azure/account#az-account-set) command. For more information, see [How to manage Azure subscriptions - Azure CLI](/en-us/cli/azure/manage-azure-subscriptions-azure-cli?tabs=bash#change-the-active-subscription). - Depending on your Azure subscription, you might need to request a vCPU quota increase for the GPU-enabled VM family you use in this quickstart. For more information, see [Increase VM-family vCPU quotas](/en-us/azure/quotas/per-vm-quota-requests). - GPU-enabled VM sizes contain specialized hardware subject to higher pricing and regional availability. For more information, see [GPU optimized virtual machine sizes](/en-us/azure/virtual-machines/sizes-gpu). -## Install the `aks-preview` CLI extension--Install the `aks-preview` CLI extension by using the [az extension add](/en-us/cli/azure/extension#az-extension-add) command.--```azurecli-az extension add --name aks-preview-```--Update the extension to ensure you have the latest version by using the [az extension update](/en-us/cli/azure/extension#az-extension-update) command.--```azurecli-az extension update --name aks-preview-```--## Register the preview feature--Register the `ManagedGPUExperiencePreview` feature flag in your subscription by using the [az feature register](/en-us/cli/azure/feature#az-feature-register) command.--```azurecli-az feature register --namespace Microsoft.ContainerService --name ManagedGPUExperiencePreview-```--It takes a few minutes for the status to show **Registered**. Verify the registration status by using the [az feature show](/en-us/cli/azure/feature#az-feature-show) command.--```azurecli-az feature show --namespace Microsoft.ContainerService --name ManagedGPUExperiencePreview --query properties.state-```--When the status shows **Registered**, refresh the registration of the `Microsoft.ContainerService` resource provider by using the [az provider register](/en-us/cli/azure/provider#az-provider-register) command.--```azurecli-az provider register --namespace Microsoft.ContainerService-```- ## Define environment variables  Define the following environment variables for use throughout this quickstart.

@@ -144,14 +144,14 @@ See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.m  The following Kubernetes distributions **have been validated in conformance testing**. This means we have explicitly validated that the **Azure Policy Extension installs correctly and functions as expected** on these platforms. -- **Supported distributions with conformance validation**: AKS on Azure Local (AKS enabled by Azure Arc), Azure Red Hat OpenShift (ARO), Openshift Cloud Platform (OCP), Kind, Rancher Government (RKE2), Minikube, K3s, AKS Edge, TKG (VMware Tanzu Kubernetes Grid)+- **Supported distributions with conformance validation**: AKS on Azure Local (AKS enabled by Azure Arc), Azure Red Hat OpenShift (ARO), Openshift Cloud Platform (OCP), Kind, Rancher Government (RKE2), Minikube, K3s, AKS Edge, TKG (VMware Tanzu Kubernetes Grid), Canonical Kubernetes Distribution.  The following Kubernetes distributions **have NOT been validated in conformance testing**. This means Azure Policy extension installation is supported, but **there is no guarantee of full functionality** or behavioral consistency until conformance validation is complete. -- **Supported distributions without conformance validation**: EKS (Amazon Elastic Kubernetes Service), GKE (Google Kubernetes Engine), RKE (Rancher Kubernetes Engine), Nexus Baremetal+- **Supported distributions without conformance validation**: EKS (Amazon Elastic Kubernetes Service), GKE (Google Kubernetes Engine), RKE (Rancher Kubernetes Engine), Nexus Baremetal, kubeadm  >-> kubeadm is currently not supported in Azure Policy extension. RKE (Rancher Kubernetes Engine) is now deprecated. Please use Rancher Government (RKE2) instead.+> RKE (Rancher Kubernetes Engine) is now deprecated. Please use Rancher Government (RKE2) instead.  Azure Policy extends [Gatekeeper](https://github.com/open-policy-agent/gatekeeper), an admission controller webhook for [Open Policy Agent](https://www.openpolicyagent.org/) (OPA). Use Gatekeeper with OPA to consistently apply centralized, at-scale enforcements and safeguards on your clusters. @@ -161,28 +161,28 @@ For more information, see [Understand Azure Policy for Kubernetes clusters](/en-  Azure Red Hat OpenShift (ARO) clusters ship with **Guardrails pre-installed**. These guardrails **conflict with the Azure Policy Extension** and must be disabled before installation. -To disable ARO Guardrails, run the following commands in order after connecting ARO cluster to Azure:+To disable ARO guardrails, add the following configuration flag when installing the policy extension on an ARO cluster:  ```bash-kubectl patch cluster.aro.openshift.io cluster --type json -p '[{ "op": "replace", "path": "/spec/operatorflags/aro.guardrails.deploy.managed", "value":"false" }]'--kubectl patch cluster.aro.openshift.io cluster --type json -p '[{ "op": "replace", "path": "/spec/operatorflags/aro.guardrails.enabled", "value":"false" }]'--kubectl delete ns openshift-azure-guardrails--kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration,clusterrole,clusterrolebinding,crd -o name | grep gatekeeper | xargs -r kubectl delete+--configuration-settings AdoptExistingResources=true ``` -Once guardrails are disabled, you may proceed with installing the Azure Policy Extension.--To install the Azure Policy Extension:+Full command:  ```bash-az k8s-extension create --cluster-type connectedClusters --cluster-name <cluster name> --resource-group <resource group name> --extension-type Microsoft.PolicyInsights --name <extension name>+az k8s-extension create --cluster-type connectedClusters --cluster-name <cluster name> --resource-group <resource group name> --extension-type Microsoft.PolicyInsights --name <extension name> --configuration-settings AdoptExistingResources=true ```  ### Azure Policy Extension Release Notes +#### 1.21.0++Security improvements. Fixed GKE and EKS installation bug.++- Released: Nov 2026+- Kubernetes: 1.30++- Gatekeeper: 3.23.1+ #### 1.20.2  Security improvements. Resolved eMudhra issue.

@@ -242,7 +242,7 @@ Assignment of the managed identity immediately after creation may result in an e     az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_NAME     helm install alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller \          --namespace $HELM_NAMESPACE \-         --version 1.12.4 \+         --version 1.13.0 \          --set albController.namespace=$CONTROLLER_NAMESPACE \          --set albController.podIdentity.clientID=$(az identity show -g $RESOURCE_GROUP -n $IDENTITY_RESOURCE_NAME --query clientId -o tsv)     ```@@ -262,7 +262,7 @@ Assignment of the managed identity immediately after creation may result in an e     # Install Helm chart     helm install alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller `         --namespace $HELM_NAMESPACE `-        --version 1.12.4 `+        --version 1.13.0 `         --set albController.namespace=$CONTROLLER_NAMESPACE `         --set albController.podIdentity.clientID=$clientId     ```@@ -284,7 +284,7 @@ Assignment of the managed identity immediately after creation may result in an e     az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_NAME     helm upgrade alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller \         --namespace $HELM_NAMESPACE \-        --version 1.12.4 \+        --version 1.13.0 \         --set albController.namespace=$CONTROLLER_NAMESPACE \         --set albController.podIdentity.clientID=$(az identity show -g $RESOURCE_GROUP -n $IDENTITY_RESOURCE_NAME --query clientId -o tsv)     ```@@ -304,7 +304,7 @@ Assignment of the managed identity immediately after creation may result in an e     # Upgrade Helm chart     helm upgrade alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller `         --namespace $HELM_NAMESPACE `-        --version 1.12.4 `+        --version 1.13.0 `         --set albController.namespace=$CONTROLLER_NAMESPACE `         --set albController.podIdentity.clientID=$clientId     ```

@@ -180,7 +180,7 @@ BackendTLSPolicyConfig defines the policy specification for the Backend TLS Poli | `CommonTLSPolicy`*CommonTLSPolicy* | (Members of `CommonTLSPolicy` are embedded into this type.) | | `sni`* string * | *(Optional)*<br>Sni is the server name to use for the TLS connection to the backend. | | `ports`*[]BackendTLSPolicyPort* | Ports specifies the list of ports where the policy is applied. |-| `clientCertificateRef`* Gateway API .SecretObjectReference * | *(Optional)*<br>ClientCertificateRef is the reference to the client certificate to use for the TLS connection to the backend. |+| `clientCertificateRef`*[Gateway API .SecretObjectReference](https://gateway-api.sigs.k8s.io/reference/api-spec/1.6/spec/#secretobjectreference)* | *(Optional)*<br>ClientCertificateRef is the reference to the client certificate to use for the TLS connection to the backend. |  ### BackendTLSPolicyPort @@ -232,7 +232,7 @@ CommonTLSPolicyVerify defines the schema for the CommonTLSPolicyVerify API.  | Field | Description | | --- | --- |-| `caCertificateRef`* Gateway API .SecretObjectReference * | CaCertificateRef is the CA certificate used to verify peer certificate. |+| `caCertificateRef`*[Gateway API .SecretObjectReference](https://gateway-api.sigs.k8s.io/reference/api-spec/1.6/spec/#secretobjectreference)* | CaCertificateRef is the CA certificate used to verify peer certificate. | | `subjectAltName`* string * | *(Optional)*<br>SubjectAltName is the subject alternative name used to verify peer certificate. |  ### CustomTargetRef@@ -243,7 +243,7 @@ CustomTargetRef is a reference to a custom resource that isn’t part of the Kub  | Field | Description | | --- | --- |-| `NamespacedPolicyTargetReference`* Gateway API .NamespacedPolicyTargetReference * | (Members of `NamespacedPolicyTargetReference` are embedded into this type.) |+| `NamespacedPolicyTargetReference`*[Gateway API .NamespacedPolicyTargetReference](https://gateway-api.sigs.k8s.io/reference/api-spec/1.6/spec/#namespacedpolicytargetreference)* | (Members of `NamespacedPolicyTargetReference` are embedded into this type.) | | `sectionNames`* []string * | *(Optional)*<br>SectionNames is the name of the section within the target resource. When unspecified, this targetRef targets the entire resource. In the following resources, SectionNames is interpreted as the following:<br><br>- Gateway: Listener Name<br>- Service: Port Name<br><br><br>If a SectionNames is specified, but doesn’t exist on the targeted object, the Policy fails to attach, and the policy implementation will record a `ResolvedRefs` or similar Condition in the Policy’s status. |  ### FrontendTLSPolicy@@ -367,7 +367,7 @@ HTTPPathModifier defines configuration for path modifiers.  | Field | Description | | --- | --- |-| `type`*HTTPPathModifierType* | Type defines the type of path modifier. More types may be added in a future release of the API.<br><br>Values may be added to this enum, implementations must ensure unknown values won’t cause a crash.<br><br>Unknown values here must result in the implementation setting the Accepted Condition for the rule to be false |+| `type`*HTTPPathModifierType* | Type defines the type of path modifier. More types may be added in a future release of the API.<br><br>This enum might be extended in the future. Implementations must handle unknown values without crashing.<br><br>Unknown values here must result in the implementation setting the Accepted Condition for the rule to be false | | `replaceFullPath`* string * | *(Optional)*<br>ReplaceFullPath specifies the value with which to replace the full path of a request during a rewrite or redirect. | | `replacePrefixMatch`* string * | *(Optional)*<br>ReplacePrefixMatch specifies the value with which to replace the prefix match of a request during a rewrite or redirect. For example, a request to “/foo/bar” with a prefix match of “/foo” and a ReplacePrefixMatch of “/xyz” would be modified to “/xyz/bar”.<br><br>This matches the behavior of the PathPrefix match type. This matches full path elements. A path element refers to the list of labels in the path split by the `/` separator. When specified, a trailing `/` is ignored. For example, the paths `/abc`, `/abc/`, and `/abc/def` would all match the prefix `/abc`, but the path `/abcd` wouldn’t.<br><br>ReplacePrefixMatch is only compatible with a `PathPrefix` HTTPRouteMatch. Using any other HTTPRouteMatch type on the same HTTPRouteRule results in the implementation setting the Accepted Condition for the Route to `status: False`.<br><br><br>| Request Path | Prefix Match | Replace Prefix | Modified Path |<br>| --- | --- | --- | --- |<br>| /foo/bar | /foo | /xyz | /xyz/bar |<br>| /foo/bar | /foo | /xyz/ | /xyz/bar |<br>| /foo/bar | /foo/ | /xyz | /xyz/bar |<br>| /foo/bar | /foo/ | /xyz/ | /xyz/bar |<br>| /foo | /foo | /xyz | /xyz |<br>| /foo/ | /foo | /xyz | /xyz/ |<br>| /foo/bar | /foo |  | /bar |<br>| /foo/ | /foo |  | / |<br>| /foo | /foo |  | / |<br>| /foo/ | /foo | / | / |<br>| /foo | /foo | / | / | | | --- | --- |@@ -641,7 +641,7 @@ LoadAwareConfig defines the configuration for load aware routing. | `blackoutPeriod`*[Kubernetes meta/v1.Duration](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#Duration)* | *(Optional)*<br>An endpoint must report load metrics continuously for at least this long before the endpoint metrics will be used to influence load balancing decisions. Takes effect both immediately after we establish a connection to an endpoint and after `metricExpirationPeriod` has elapsed. Default is 10 seconds. | | `metricExpirationPeriod`*[Kubernetes meta/v1.Duration](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#Duration)* | *(Optional)*<br>If the endpoint does not report load metrics for this duration, metrics will stop being used to influence load balancing decisions. Default is 3 minutes. | | `errorUtilizationPenalty`* string * | *(Optional)*<br>The multiplier used to adjust endpoint weights with the error rated calculated based on the reported `rps_fractional` and `eps` load metrics. Must not be a negative value. Default is 1.0. |-| `namedMetrics`* []string * | *(Optional)*<br>A list of custom metrics reported by endpoints to be used for reporting utilization and influencing load balancing decisions. Utilization will be computed by taking the max of the values of metrics specified in this list. |+| `namedMetrics`* []string * | *(Optional)*<br>A list of custom metrics reported by endpoints to be used for reporting utilization and influencing load balancing decisions. Compute utilization by taking the max of the values of metrics specified in this list. |  ### LoadBalancingConfig @@ -676,7 +676,7 @@ MTLSPolicyVerify defines the schema for the MTLSPolicyVerify API.  | Field | Description | | --- | --- |-| `caCertificateRef`* Gateway API .SecretObjectReference * | CaCertificateRef is the CA certificate used to verify peer certificate. |+| `caCertificateRef`*[Gateway API .SecretObjectReference](https://gateway-api.sigs.k8s.io/reference/api-spec/1.6/spec/#secretobjectreference)* | CaCertificateRef is the CA certificate used to verify peer certificate. | | `subjectAltNames`* []string * | *(Optional)*<br>SubjectAltNames is the list of subject alternative names used to verify peer certificate. |  ### PolicyConditionReason (`string` alias)@@ -700,6 +700,7 @@ PolicyConditionReason is the type of reason used for different ALB related polic | "NoTargetReference" |  | | "OperationFailed" |  | | "OverrideNotSupported" |  |+| "PartiallyResolvedTargetReference" |  | | "Pending" |  | | "Programmed" |  | | "ProgrammingFailed" |  |@@ -780,7 +781,7 @@ Redirect defines a filter that redirects a request. This MUST NOT be used on the | `hostname`*PreciseHostname* | *(Optional)*<br>Hostname is the hostname to be used in the value of the `Location` header in the response. When empty, the hostname in the `Host` header of the request is used. | | `path`*HTTPPathModifier* | *(Optional)*<br>Path defines parameters used to modify the path of the incoming request. The modified path is then used to construct the `Location` header. When empty, the request path is used as-is. | | `port`*PortNumber* | *(Optional)*<br>Port is the port to be used in the value of the `Location` header in the response.<br><br>If no port is specified, the redirect port MUST be derived using the following rules:<br><br>- If redirect scheme is not-empty, the redirect port MUST be the well-known port associated with the redirect scheme. Specifically “http” to port 80 and “https” to port 443. If the redirect scheme doesn’t have a well-known port, the listener port of the Gateway SHOULD be used.<br>- If redirect scheme is empty, the redirect port MUST be the Gateway Listener port.<br><br><br>Implementations SHOULD NOT add the port number in the ‘Location’ header in the following cases:<br><br>- A Location header that uses HTTP (whether that is determined via the Listener protocol or the Scheme field) *and* use port 80.<br>- A Location header that uses HTTPS (whether that is determined via the Listener protocol or the Scheme field) *and* use port 443. |-| `statusCode`* int * | *(Optional)*<br>StatusCode is the HTTP status code to be used in response.<br><br>Values may be added to this enum, implementations must ensure that unknown values won’t cause a crash. |+| `statusCode`* int * | *(Optional)*<br>StatusCode is the HTTP status code to be used in response.<br><br>This enum might be extended in the future. Implementations must handle unknown values without crashing. |  ### RewriteType (`string` alias) 

@@ -1,18 +1,20 @@-# Fix AKS time-out errors from custom NSG blocks - Azure | Microsoft Learn+# Troubleshoot AKS time-out errors from custom NSG blocks - Azure | Microsoft Learn  ## Summary -When you access an application that's hosted on an Azure Kubernetes Service (AKS) cluster, you receive a "Timed out" error message. This error can occur even if the application is running and the rest of the configuration appears to be correct.+AKS time-out errors can occur when a custom network security group (NSG) blocks traffic to an application that's hosted on an Azure Kubernetes Service (AKS) cluster. You receive a "Timed out" error message even if the application is running and the rest of the configuration appears correct. This article helps you identify the cause and allow the traffic.  ## Prerequisites +Ensure you have the following prerequisites:+ - The Kubernetes [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) tool, or a similar tool, to connect to the cluster. To install kubectl by using [Azure CLI](/en-us/cli/azure/install-azure-cli), run the [az aks install-cli](/en-us/cli/azure/aks#az-aks-install-cli) command. - The Client URL ([cURL](https://www.tecmint.com/install-curl-in-linux/)) tool, or a similar command-line tool. - The [apt-get](https://linux.die.net/man/8/apt-get) command-line tool for handling packages.  ## Symptoms -If you run the following [kubectl get](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get) and cURL commands, you experience "Timed out" errors that resemble the following console output:+If you run the following [kubectl get](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get) and cURL commands, you experience "Timed out" errors that resemble the following console output.  ```console $ kubectl get pods@@ -33,11 +35,11 @@ curl: (28) Failed to connect to 10.81.x.x port 80 after 21033 ms: Timed out  ## Cause -If you experience the same "Timed out" error every time, that usually suggests that a network component is blocking the traffic.+If you experience the same "Timed out" error every time, that error usually suggests that a network component is blocking the traffic. -To troubleshoot this issue, you can start by checking access to the pod, and then move on to the client in an *inside-out* approach.+To troubleshoot this issue, start by checking access to the pod, and then move on to the client in an *inside-out* approach. -To check the pod, run the following `kubectl get` and [kubectl describe](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe) commands:+To check the pod, run the following `kubectl get` and [kubectl describe](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe) commands.  ```console $ kubectl get pods -o wide@@ -59,7 +61,7 @@ Events:  Based on this output, the pod seems to be running correctly, without any restarts. -Open a test pod to check access to the application pod. Run the following `kubectl get`, [kubectl run](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#run), `apt-get`, and cURL commands:+Open a test pod to check access to the application pod. Run the following `kubectl get`, [kubectl run](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#run), `apt-get`, and cURL commands.  ```console $ kubectl get pods -o wide  # Get the pod IP address.@@ -94,17 +96,17 @@ HTTP/1.1 200 OK  The pod is accessible directly. Therefore, the application is running. -The defined service is a `LoadBalancer` type. This means that the request flow from the end client to the pod will be as follows:+The defined service is a `LoadBalancer` type. This type means that the request flow from the end client to the pod is as follows.  > > Client >> Load balancer >> AKS node >> Application pod -In this request flow, we can block the traffic through the following components:+In this request flow, you can block the traffic through the following components:  - Network policies in the cluster - The network security group (NSG) for the AKS subnet and AKS node -To check network policy, run the following `kubectl get` command:+To check the network policy, run the following `kubectl get` command.  ```console $ kubectl get networkpolicy --all-namespaces@@ -112,7 +114,7 @@ NAMESPACE     NAME                 POD-SELECTOR             AGE kube-system   konnectivity-agent   app=konnectivity-agent   3h8m ``` -Only the AKS default policy exists. Therefore, network policy doesn't seem to be blocking the traffic.+Only the AKS default policy exists. Therefore, the network policy doesn't seem to be blocking the traffic.  To check the NSGs and their associated rules by using AKS, follow these steps: @@ -120,26 +122,26 @@ To check the NSGs and their associated rules by using AKS, follow these steps: 2. In the list of scale set instances, select the one that you're using. 3. In the menu pane of your scale set instance, select `Networking`. -The **Networking** page for the scale set instance appears. In the **Inbound port rules** tab, two sets of rules are displayed that are based on the two NSGs that act on the scale set instance:+The **Networking** page for the scale set instance appears. In the **Inbound port rules** tab, you see two sets of rules that are based on the two NSGs that act on the scale set instance: -- The first set is composed of NSG rules at the subnet level. These rules are displayed under the following note heading:+- The first set is composed of NSG rules at the subnet level. These rules are displayed under the following note heading.  > > Network security group *<my-aks-nsg>* (attached to subnet: *<my-aks-subnet>*) -    This arrangement is common if a custom virtual network and custom subnet for the AKS cluster are used. The set of rules at the subnet level might resemble the following table.+    This arrangement is common if you use a custom virtual network and custom subnet for the AKS cluster. The set of rules at the subnet level might resemble the following table.      | Priority | Name | Port | Protocol | Source | Destination | Action |     | --- | --- | --- | --- | --- | --- | --- |     | 65000 | AllowVnetInBound | Any | Any | VirtualNetwork | VirtualNetwork | Allow |     | 65001 | AllowAzureLoadBalancerInBound | Any | Any | AzureLoadBalancer | Any | Allow |     | 65500 | DenyAllInBound | Any | Any | Any | Any | Deny |-- The second set is composed of NSG rules at the network adapter level. These rules are displayed under the following note heading:+- The second set is composed of NSG rules at the network adapter level. You see these rules under the following note heading.  > > Network security group *aks-agentpool-<agentpool-number>-nsg* (attached to network interface: *aks-agentpool-<vm-scale-set-number>-vmss*) -    This NSG is applied by the AKS cluster, and it's managed by AKS. The corresponding set of rules might resemble the following table.+    This NSG is applied by the AKS cluster, and AKS manages it. The corresponding set of rules might resemble the following table.      | Priority | Name | Port | Protocol | Source | Destination | Action |     | --- | --- | --- | --- | --- | --- | --- |@@ -148,13 +150,13 @@ The **Networking** page for the scale set instance appears. In the **Inbound por     | 65001 | AllowAzureLoadBalancerInBound | Any | Any | AzureLoadBalancer | Any | Allow |     | 65500 | DenyAllInBound | Any | Any | Any | Any | Deny | -At the network adapter level, there's an NSG inbound rule for TCP at IP address 10.81.*x*.*x* on port 80 (highlighted in the table). However, an equivalent rule is missing from the rules for the NSG at the subnet level.+At the network adapter level, there's an NSG inbound rule for TCP at IP address 10.81.*x*.*x* on port 80 (highlighted in the preceding table). However, the rules for the NSG at the subnet level don't include an equivalent rule. -Why didn't AKS apply the rule to the custom NSG? Because AKS doesn't apply NSGs to its subnet, and it won't modify any of the NSGs that are associated with that subnet. AKS will modify the NSGs only at the network adapter level. For more information, see [Can I configure NSGs with AKS?](/en-us/azure/aks/faq#can-i-configure-nsgs-with-aks).+Why didn't AKS apply the rule to the custom NSG? AKS doesn't apply NSGs to its subnet, and it doesn't modify any of the NSGs that are associated with that subnet. AKS only modifies the NSGs at the network adapter level. For more information, see [Can I configure NSGs with AKS?](/en-us/azure/aks/faq#can-i-configure-nsgs-with-aks).  ## Solution -If the application is enabled for access on a certain port, you must make sure that the custom NSG allows that port as an `Inbound` rule. After the appropriate rule is added in the custom NSG at the subnet level, the application is accessible.+If the application is enabled for access on a certain port, ensure that the custom NSG allows that port as an `Inbound` rule. After you add the appropriate rule in the custom NSG at the subnet level, the application is accessible.  ```console $ curl -Iv http://10.81.x.x

@@ -31,7 +31,7 @@ Prior to migration, it's important to identify any dependencies on Application G  Such dependencies include: -- Private IP+- Private IP *(now available in public preview — see [Private ingress with Application Gateway for Containers](private-ingress))* - Ports other than 80 and 443  Here's a summarized list of AGIC annotations and whether Application Gateway for Containers has translated such capabilities to either Gateway or Ingress API.@@ -50,7 +50,7 @@ Here's a summarized list of AGIC annotations and whether Application Gateway for | [Session Affinity / Sticky Sessions](migrate-from-agic-to-agc#session-affinity) | appgw.ingress.kubernetes.io/cookie-based-affinity | [Session affinity in Gateway API](migrate-from-agic-to-agc#session-affinity-in-gateway-api) | [Session affinity in Ingress API](migrate-from-agic-to-agc#session-affinity-in-ingress-api) | | [Request timeout](migrate-from-agic-to-agc#request-timeout) | appgw.ingress.kubernetes.io/request-timeout | [RoutePolicy](api-specification-kubernetes#alb.networking.azure.io/v1.RouteTimeouts) | [IngressBackendSettings](api-specification-kubernetes#alb.networking.azure.io/v1.IngressTimeouts) | | [Frontend port other than 80 and 443](migrate-from-agic-to-agc#frontend-port-override) | appgw.ingress.kubernetes.io/override-frontend-port | Not supported | Not supported |-| [Private frontend](migrate-from-agic-to-agc#private-frontend) | appgw.ingress.kubernetes.io/use-private-ip | Not supported | Not supported |+| [Private frontend](migrate-from-agic-to-agc#private-frontend) | appgw.ingress.kubernetes.io/use-private-ip | [Public preview](private-ingress) | [Public preview](private-ingress) | | [Web Application Firewall (WAF)](migrate-from-agic-to-agc#waf) | appgw.ingress.kubernetes.io/waf-policy-for-path | [Web Application Firewall (WAF)](web-application-firewall) | Not supported | | [Custom health probe](migrate-from-agic-to-agc#custom-health-probes) | appgw.ingress.kubernetes.io/health-probe-hostname | [HealthCheckPolicy](migrate-from-agic-to-agc#healthcheckpolicy) | [HealthCheckPolicy](migrate-from-agic-to-agc#healthcheckpolicy) | | [Custom health probe](migrate-from-agic-to-agc#custom-health-probes) | appgw.ingress.kubernetes.io/health-probe-port | [HealthCheckPolicy](migrate-from-agic-to-agc#healthcheckpolicy) | [HealthCheckPolicy](migrate-from-agic-to-agc#healthcheckpolicy) |@@ -283,7 +283,9 @@ AGIC annotation  Application Gateway for Containers implementation -Private IP frontend isn't currently supported by Application Gateway for Containers.+Private frontend support is available in **public preview** for Application Gateway for Containers. Unlike AGIC's annotation-based private IP assignment, Application Gateway for Containers implements private ingress through Azure Private Link private endpoints, which enables connectivity across virtual networks, subscriptions, and Microsoft Entra tenants.++For configuration details, see [Private ingress with Application Gateway for Containers](private-ingress).  ### WAF 

@@ -1,4 +1,4 @@-# Troubleshoot AKS image pull errors - Azure | Microsoft Learn+# Troubleshoot AKS image pull errors from Azure Container Registry - Azure | Microsoft Learn  Note @@ -6,7 +6,9 @@ Was this article helpful? Your input is important to us. Please use the **Feedba  ## Summary -When you use Microsoft Azure Container Registry together with Azure Kubernetes Service (AKS), you need to establish an authentication mechanism. Set up the AKS to Container Registry integration by using a few simple Azure CLI or Azure PowerShell commands. This integration assigns either the [Container Registry Repository Reader role](/en-us/azure/role-based-access-control/built-in-roles#container-registry-repository-reader) (for attribute-based access control (ABAC)-enabled registries) or the [AcrPull role](/en-us/azure/role-based-access-control/built-in-roles#acrpull) (for non-ABAC-enabled registries) for the kubelet identity that's associated with the AKS cluster to pull images from a container registry. For more information about the necessary ACR role to assign based on whether your ACR registry is ABAC-enabled, see [Microsoft Entra attribute-based access control (ABAC) for repository permissions](/en-us/azure/container-registry/container-registry-rbac-abac-repository-permissions?tabs=azure-portal).+This article helps you troubleshoot image pull errors from Azure Container Registry to an Azure Kubernetes Service (AKS) cluster. It provides guidance for resolving common issues such as authentication failures, network connectivity problems, and misconfigured roles.++When you use Azure Container Registry together with Azure Kubernetes Service (AKS), you need to establish an authentication mechanism. Set up the AKS to Container Registry integration by using a few simple Azure CLI or Azure PowerShell commands. This integration assigns either the [Container Registry Repository Reader role](/en-us/azure/role-based-access-control/built-in-roles#container-registry-repository-reader) (for attribute-based access control (ABAC)-enabled registries) or the [AcrPull role](/en-us/azure/role-based-access-control/built-in-roles#acrpull) (for non-ABAC-enabled registries) for the kubelet identity that's associated with the AKS cluster to pull images from a container registry. For more information about the necessary ACR role to assign based on whether your ACR registry is ABAC-enabled, see [Microsoft Entra attribute-based access control (ABAC) for repository permissions](/en-us/azure/container-registry/container-registry-rbac-abac-repository-permissions?tabs=azure-portal).  In some cases, pulling images from a container registry to an AKS cluster fails. This article provides guidance for troubleshooting the most common errors that you encounter when you pull images from a container registry to an AKS cluster. @@ -271,7 +273,7 @@ The `403 Forbidden` error message appears as follows:  ### Solution 1: Make sure the AKS virtual network link is set in the container registry's Private DNS zone -If the network interface of the container registry's private endpoint and the AKS cluster are in different virtual networks, make sure that the [virtual network link](/en-us/azure/dns/private-dns-virtual-network-links) for the AKS cluster's virtual network is set in the private DNS zone of the container registry. (That link is named **privatelink.azurecr.io** by default.) If the virtual network link isn't in the private DNS zone of the container registry, add it using one of the following methods:+If the network interface of the container registry's private endpoint and the AKS cluster are in different virtual networks, make sure that the [virtual network link](/en-us/azure/dns/private-dns-virtual-network-links) for the AKS cluster's virtual network is set in the private DNS zone of the container registry. That link is named **privatelink.azurecr.io** by default. If the virtual network link isn't in the private DNS zone of the container registry, add it by using one of the following methods:  - Use the Azure portal: @@ -327,7 +329,7 @@ If the AKS cluster connects publicly to the container registry (**not** through          If **Public network access** is set to **Disabled**, switch it to **Selected networks** first. -        [![Screenshot showing how to add AKS Load Balancer's public IP address to address range.](media/cannot-pull-image-from-acr-to-aks-cluster/add-aks-load-balancer-public-ip.png)](media/cannot-pull-image-from-acr-to-aks-cluster/add-aks-load-balancer-public-ip.png#lightbox)+        [![Screenshot of the Networking page in the Azure portal showing the AKS load balancer's public IP address added to the address range.](media/cannot-pull-image-from-acr-to-aks-cluster/add-aks-load-balancer-public-ip.png)](media/cannot-pull-image-from-acr-to-aks-cluster/add-aks-load-balancer-public-ip.png#lightbox)  ## Cause 4: `i/o timeout` error @@ -341,9 +343,9 @@ The `i/o timeout` error occurs only when you [connect privately to a container r  ### Solution 1: Make sure virtual network peering is used -If the network interface of the container registry's private endpoint and the AKS cluster are in different virtual networks, make sure that [virtual network peering](/en-us/azure/virtual-network/virtual-network-peering-overview) is used for both virtual networks. You can check virtual network peering by running the Azure CLI command `az network vnet peering list --resource-group <MyResourceGroup> --vnet-name <MyVirtualNetwork> --output table` or in the Azure portal by selecting the **VNETs** &gt; **Peerings** in the **Settings** panel. For more information about listing all peerings of a specified virtual network, see [az network vnet peering list](/en-us/cli/azure/network/vnet/peering#az-network-vnet-peering-list).+If the network interface of the container registry's private endpoint and the AKS cluster are in different virtual networks, ensure that you use [virtual network peering](/en-us/azure/virtual-network/virtual-network-peering-overview) for both virtual networks. You can check virtual network peering by running the Azure CLI command `az network vnet peering list --resource-group <MyResourceGroup> --vnet-name <MyVirtualNetwork> --output table` or in the Azure portal by selecting the **VNETs** &gt; **Peerings** in the **Settings** panel. For more information about listing all peerings of a specified virtual network, see [az network vnet peering list](/en-us/cli/azure/network/vnet/peering#az-network-vnet-peering-list). -If the virtual network peering is used for both virtual networks, make sure that the status is **Connected**. If the status is [Disconnected](/en-us/azure/virtual-network/virtual-network-troubleshoot-peering-issues#the-peering-status-is-disconnected), delete the peering from both virtual networks, and then recreate it. If the status is **Connected**, see the troubleshooting guide: [The peering status is "Connected"](/en-us/azure/virtual-network/virtual-network-troubleshoot-peering-issues#the-peering-status-is-connected).+If you use virtual network peering for both virtual networks, ensure that the status is **Connected**. If the status is [Disconnected](/en-us/azure/virtual-network/virtual-network-troubleshoot-peering-issues#the-peering-status-is-disconnected), delete the peering from both virtual networks, and then recreate it. If the status is **Connected**, see the troubleshooting guide: [The peering status is "Connected"](/en-us/azure/virtual-network/virtual-network-troubleshoot-peering-issues#the-peering-status-is-connected).  For further troubleshooting, connect to one of the AKS nodes or [pods](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#exec) and then test the connectivity with the container registry at the TCP level by using the telnet or netcat utility. Check the IP address with the `nslookup <acrname>.azurecr.io` command and then run the `telnet <ip-address-of-the-container-registry> 443` command. @@ -351,7 +353,7 @@ For more information about connecting to AKS nodes, see [Connect with SSH to Azu  ### Solution 2: Use Azure Firewall -If the network interface of the container registry's private endpoint and the AKS cluster are in different virtual networks (in addition to virtual network peering), you can use Azure Firewall to set up a [Hub-spoke network topology in Azure](/en-us/azure/architecture/reference-architectures/hybrid-networking/hub-spoke?tabs=cli). When you set up the firewall rule, you need to use network rules to explicitly allow the [outbound connection](/en-us/azure/firewall/rule-processing#outbound-connectivity) to the container registry private endpoint IP addresses.+If the network interface of the container registry's private endpoint and the AKS cluster are in different virtual networks (in addition to virtual network peering), you can use Azure Firewall to set up a [Hub-spoke network topology in Azure](/en-us/azure/architecture/reference-architectures/hybrid-networking/hub-spoke?tabs=cli). When you set up the firewall rule, use network rules to explicitly allow the [outbound connection](/en-us/azure/firewall/rule-processing#outbound-connectivity) to the container registry private endpoint IP addresses.  ## Cause 5: No match for platform in manifest @@ -359,11 +361,11 @@ The host operating system (node OS) is incompatible with the image that you use  `Failed to pull image "\<acrname>.azurecr.io/\<repository:tag>"\:\ [\ &nbsp; rpc error:\ &nbsp; code = NotFound\ &nbsp; desc = failed to pull and unpack image "\<acrname>.azurecr.io/\<repository:tag>": **no match for platform in manifest**: not found,\ ]` -This error occurs for an image that's pulled from any source if the image is incompatible with the host OS. The error isn't limited to images that you pull from the container registry.+This error occurs for an image that you pull from any source if the image is incompatible with the host OS. The error isn't limited to images that you pull from the container registry.  ### Solution: Configure the nodeSelector field correctly in your pod or deployment -Specify the correct `nodeSelector` field in the configuration settings of your pod or deployment. The correct value for this field's `kubernetes.io/os` setting ensures that the pod is scheduled on the correct type of node. The following table shows how to set the `kubernetes.io/os` setting in YAML:+Specify the correct `nodeSelector` field in the configuration settings of your pod or deployment. The correct value for this field's `kubernetes.io/os` setting ensures that the pod is scheduled on the correct type of node. The following table shows how to set the `kubernetes.io.os` setting in YAML:  | Container type | YAML setting | | --- | --- |@@ -449,7 +451,7 @@ The following events and messages might occur when pulling images from ACR to AK  ### Solution: Update Azure Virtual Network Appliance (NVAs) by using validated routes -If you receive this `net/http: TLS handshake` time-out, check your network configuration, including firewall, proxy, and network connectivity settings.+If you receive this `net/http: TLS handshake` timeout, check your network configuration, including firewall, proxy, and network connectivity settings.  ## Cause 8: Data endpoint DNS or egress blocked @@ -465,9 +467,9 @@ or:  `Get "https://<region>.data.mcr.microsoft.com/...": connection reset by peer` -### Solution 1: Add the data endpoint to your firewall or egress allowlist+### Solution 1: Add the data endpoint to your firewall or egress allow list -If you use Azure Firewall, a proxy, or an NVA for egress filtering, add the data endpoints to the allowlist:+If you use Azure Firewall, a proxy, or an NVA for egress filtering, add the data endpoints to the allow list:  - For ACR: allow `*.blob.core.windows.net` (or the specific `<registry-name>.<region>.data.azurecr.io` host) on TCP 443. - For MCR: allow `*.data.mcr.microsoft.com` on TCP 443.@@ -520,7 +522,7 @@ containers:   imagePullPolicy: Always ``` -Refer to [Impage Pull Policy Behavior](https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy) for more details.+Refer to [Image Pull Policy Behavior](https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy) for more details.  ### Solution 3: Force a rollout restart to re-resolve the tag @@ -581,15 +583,15 @@ If this call times out, check that no NVA, network security group, or custom rou  ## Cause 11: Public or third-party registry rate limits -When pulling images from Docker Hub, GitHub Container Registry (`ghcr.io`), or other public registries, the AKS cluster shares a rate limit with all traffic from the node's outbound IP address. If the limit is exceeded, the pull fails with:+When you pull images from Docker Hub, GitHub Container Registry (`ghcr.io`), or other public registries, the AKS cluster shares a rate limit with all traffic from the node's outbound IP address. If you exceed the limit, the pull fails with:  `Failed to pull image "docker.io/library/nginx:latest": ... 429 Too Many Requests`  or: -`You have reached your pull rate limit. You may increase the limit by authenticating...`+`You have reached your pull rate limit. You can increase the limit by authenticating...` -This commonly affects clusters that:+This limit commonly affects clusters that:  - Pull many different public images during node scale-up. - Use `imagePullPolicy: Always` with frequently restarting pods.@@ -607,7 +609,7 @@ Then update your deployments to reference `<myacr>.azurecr.io/nginx:latest`.  ### Solution 2: Use ACR Artifact Cache (pull-through cache) -Configure [ACR Artifact Cache](/en-us/azure/container-registry/container-registry-artifact-cache) to automatically cache upstream images. This provides a pull-through proxy that caches images from Docker Hub, `ghcr.io`, and other registries inside your ACR.+Configure [ACR Artifact Cache](/en-us/azure/container-registry/container-registry-artifact-cache) to automatically cache upstream images. This setup provides a pull-through proxy that caches images from Docker Hub, `ghcr.io`, and other registries inside your ACR.  ### Solution 3: Authenticate with the upstream registry 

@@ -10,12 +10,6 @@ Note  If you aren't already familiar with Fleet Manager's resource placement concepts, read the [cluster-scoped resource placement overview](concepts-resource-placement) and the [namespace-scoped resource placement overview](concepts-namespace-scoped-resource-propagation) before reading this article. -Important--Azure Kubernetes Fleet Manager preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. Azure Kubernetes Fleet Manager previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use.--Our data plane previews are released via our v1beta1 API. If you don't see preview objects, check that you're requesting the v1beta1 API when you interact with the Fleet Manager hub cluster.- ## Workload takeover options  The starting point for workload takeover is to deploy the workload manifests onto your Fleet Manager [hub cluster](access-fleet-hub-cluster-kubernetes-api). Once deployed you define a `ClusterResourcePlacement` or `ResourcePlacement` that specifies an explicit takeover behavior using the `whenToTakeOver` property.@@ -37,10 +31,12 @@ You can use an optional `comparisonOptions` property to fine-tune how `whenToTak  In order to check for existing workloads, start by adding an `applyStrategy` with a `whenToTakeOver` value of `Never` as shown in the following examples. +::: zone pivot="cluster-scope"+ ### ClusterResourcePlacement example  ```yml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata:   name: web-2-crp@@ -69,10 +65,14 @@ Apply the `ClusterResourcePlacement` to your Fleet Manager hub cluster. kubectl apply -f web-2-crp.yaml ``` +::: zone-end++::: zone pivot="namespace-scope"+ ### ResourcePlacement example  ```yml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ResourcePlacement metadata:   name: web-2-rp@@ -100,26 +100,32 @@ Apply the `ResourcePlacement` to your Fleet Manager hub cluster. kubectl apply -f web-2-rp.yaml ``` +::: zone-end+ ### Check placement status  Fleet Manager attempts to place the workload and when it finds a matching workload on a target member cluster returns a `failedPlacement` response.  You can determine which member clusters failed placement due to a clashing workload using the following commands. The [jq command](https://github.com/jqlang/jq) is used to format the output. -**For ClusterResourcePlacement:**+::: zone pivot="cluster-scope"  ```bash-kubectl get clusterresourceplacement.v1beta1.placement.kubernetes-fleet.io web-2-crp -o jsonpath='{.status.placementStatuses}' \+kubectl get clusterresourceplacement.v1.placement.kubernetes-fleet.io web-2-crp -o jsonpath='{.status.placementStatuses}' \     | jq '[.[] | select (.failedPlacements != null)] | map({clusterName, failedPlacements})' ``` -**For ResourcePlacement:**+::: zone-end++::: zone pivot="namespace-scope"  ```bash-kubectl get resourceplacement.v1beta1.placement.kubernetes-fleet.io web-2-rp -n web-2 -o jsonpath='{.status.placementStatuses}' \+kubectl get resourceplacement.v1.placement.kubernetes-fleet.io web-2-rp -n web-2 -o jsonpath='{.status.placementStatuses}' \     | jq '[.[] | select (.failedPlacements != null)] | map({clusterName, failedPlacements})' ``` +::: zone-end+ Each cluster that fails placement due to an existing workload returns an entry similar to the following sample.  ```json@@ -148,10 +154,12 @@ To proceed with the take over of an existing workload you can modify the existin  Fleet Manager applies the hub cluster workload in place of the existing workload on the target cluster when there are no differences between managed fields on both workloads. Extra fields are ignored. +::: zone pivot="cluster-scope"+ ### ClusterResourcePlacement example  ```yml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata:   name: web-2-crp@@ -181,10 +189,14 @@ Apply the `ClusterResourcePlacement` to your Fleet Manager hub cluster. kubectl apply -f web-2-crp.yaml ``` +::: zone-end++::: zone pivot="namespace-scope"+ ### ResourcePlacement example  ```yml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ResourcePlacement metadata:   name: web-2-rp@@ -213,26 +225,32 @@ Apply the `ResourcePlacement` to your Fleet Manager hub cluster. kubectl apply -f web-2-rp.yaml ``` +::: zone-end+ ### Check for failed placements  Fleet Manager attempts to place the workload, overwriting target member clusters that have matching fields.  Placements can still fail, so check for extra `failedPlacement` messages. Determine which member clusters failed using the following commands. The [jq command](https://github.com/jqlang/jq) is used to format the output. -**For ClusterResourcePlacement:**+::: zone pivot="cluster-scope"  ```bash-kubectl get clusterresourceplacement.v1beta1.placement.kubernetes-fleet.io web-2-crp -o jsonpath='{.status.placementStatuses}' \+kubectl get clusterresourceplacement.v1.placement.kubernetes-fleet.io web-2-crp -o jsonpath='{.status.placementStatuses}' \     | jq '[.[] | select (.failedPlacements != null)] | map({clusterName, failedPlacements})' ``` -**For ResourcePlacement:**+::: zone-end++::: zone pivot="namespace-scope"  ```bash-kubectl get resourceplacement.v1beta1.placement.kubernetes-fleet.io web-2-rp -n web-2 -o jsonpath='{.status.placementStatuses}' \+kubectl get resourceplacement.v1.placement.kubernetes-fleet.io web-2-rp -n web-2 -o jsonpath='{.status.placementStatuses}' \     | jq '[.[] | select (.failedPlacements != null)] | map({clusterName, failedPlacements})' ``` +::: zone-end+ Each cluster that failed placement due to a configuration difference returns an entry similar to the following sample.  ```json

@@ -18,7 +18,7 @@ These CRDs provide a more structured and Kubernetes-native way to package resour The `ClusterResourceEnvelope` is a cluster-scoped resource that can only wrap other cluster-scoped resources. For example:  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourceEnvelope metadata:   name: example@@ -58,7 +58,7 @@ data: The `ResourceEnvelope` is a namespace-scoped resource that can only wrap namespace-scoped resources. For example:  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ResourceEnvelope metadata:   name: example@@ -102,7 +102,7 @@ We apply our envelope objects on the hub cluster and then use a `ClusterResource Here's an example of a `ClusterResourcePlacement` (CRP) that propagates a `ResourceEnvelope` to a member cluster, please note that since the `ResourceEnvelope` is namespace-scoped, the CRP just needs to select the namespace that contains the envelope object:  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata:   name: crp-with-envelope@@ -126,7 +126,7 @@ spec: Here's an example of a `ClusterResourcePlacement` (CRP) that propagates a `ClusterResourceEnvelope` to a member cluster, please note that since the `ClusterResourceEnvelope` is cluster-scoped, the ClusterResourcePlacement object just needs to select the envelope object itself:  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata:   name: crp-with-cluster-envelope@@ -139,7 +139,7 @@ spec:   - group: placement.kubernetes-fleet.io     kind: ClusterResourceEnvelope     name: example-    version: v1beta1+    version: v1   revisionHistoryLimit: 10   strategy:     type: RollingUpdate@@ -160,7 +160,7 @@ status:     kind: ResourceEnvelope     name: example     namespace: app-    version: v1beta1+    version: v1   conditions:   - lastTransitionTime: "2023-11-30T19:54:13Z"     message: found all the clusters needed as specified by the scheduling policy@@ -280,7 +280,7 @@ status:     kind: ResourceEnvelope     name: example     namespace: app-    version: v1beta1+    version: v1 ```  ### CRP status with failed ClusterResourceEnvelope resource:@@ -329,5 +329,5 @@ status:   - group: placement.kubernetes-fleet.io     kind: ClusterResourceEnvelope     name: example-    version: v1beta1+    version: v1 ```

@@ -248,7 +248,7 @@ Note | `check-capacity.autoscaling.x-k8s.io` | Checks whether the cluster has enough resources for the specified pods. If the resources are available, the cluster autoscaler reserves them for the `ProvisioningRequest` for 10 minutes to prevent other `ProvisioningRequests` from claiming the same resources. | | `best-effort-atomic-scale-up.autoscaling.x-k8s.io` | The cluster autoscaler makes a best-effort attempt to provision the full requested resources in a single scale-up operation. If it can't provision all of it, the `ProvisioningRequest` fails, and the cluster autoscaler retries it after a back-off period. | -For an end-to-end example that demonstrates Kueue integration with the cluster autoscaler and `ProvisioningRequest`, see [Configure Kueue with the cluster autoscaler on AKS](configure-kueue-with-cluster-autoscaler).+For an end-to-end example that demonstrates Kueue integration with the cluster autoscaler and `ProvisioningRequest`, see [Use managed Kueue with the cluster autoscaler on AKS](use-managed-kueue-extension-with-the-cluster-autoscaler-on-aks).  ## Retrieve cluster autoscaler logs and status 

@@ -6,12 +6,14 @@ This article describes how to fix issues that occur when you can't connect to an  ## Prerequisites +Ensure that you meet the following prerequisites:+ - [Azure CLI](/en-us/cli/azure/install-azure-cli). - The Kubernetes cluster command-line tool ([kubectl](https://kubernetes.io/docs/tasks/tools/)). You can alternatively install kubectl by running the [az aks install-cli](/en-us/cli/azure/aks#az-aks-install-cli) command in Azure CLI.  ## Symptoms -During a cluster connection attempt, an error message similar to the following text appears:+During a cluster connection attempt, an error message similar to the following text appears.  ```output Unable to connect to the server: dial tcp [::1]:8080: connectex: No connection could be made because the target machine actively refused it.@@ -26,26 +28,26 @@ The [kubectl tool](https://kubernetes.io/docs/reference/kubectl/overview/) and o - The [az aks get-credentials](/en-us/cli/azure/aks#az-aks-get-credentials) command in Azure CLI, which is used to get access credentials for a managed Kubernetes cluster, modifies the *~/.kube/config* file. - The kubectl command uses the [kubeconfig (kubectl configuration) file](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) in the *$HOME/.kube* directory. -So what happens during an attempted Kubernetes session depends on the user who's running the kubectl command. If you've signed in as user A, and executed both commands, here's what happens:+So what happens during an attempted Kubernetes session depends on the user who's running the kubectl command. If you sign in as user A and run both commands, here's what happens:  - The `az aks get-credentials` command tries to add the new kubeconfig parameters in the *C:\Users\A\.kube\config* file. - The kubectl command tries to search the *C:\Users\A\.kube\config* file. -But for kubectl, if the pointer to the kubeconfig file has changed, the file that's used for accessing the cluster is supposed to be in a different location.+But for kubectl, if the pointer to the kubeconfig file changes, the file that's used for accessing the cluster is in a different location.  Note  A kubeconfig file is a reference to a file that contains configuration parameters for accessing Kubernetes clusters. It doesn't necessarily refer to a file that's named *kubeconfig*. -The error occurs if one of the following scenarios occurs:+The error occurs if one of the following scenarios occurs. -## Cause 1: The *config* file doesn't exist+## Cause 1: The config file doesn't exist -The *config* file doesn't exist on your machine.+The config file doesn't exist on your machine. -### Solution: Save the credentials+### Solution: Save the credentials to the config file -Load the *config* file by running the `az aks get-credentials` command in Azure CLI, which saves the credentials. If you don't want to use the default location, specify the `--file <config-file-location>` parameter with the location of *config* (for example, *~/Dir1/Dir2/config* or *C:\Dir1\Dir2\config*).+Load the config file by running the `az aks get-credentials` command in Azure CLI, which saves the credentials. If you don't want to use the default location, specify the `--file <config-file-location>` parameter with the location of the config file (for example, *~/Dir1/Dir2/config* or *C:\Dir1\Dir2\config*).  ```azurecli az aks get-credentials --resource-group <cluster-resource-group> \@@ -53,15 +55,15 @@ az aks get-credentials --resource-group <cluster-resource-group> \     [--file <config-file-location>] ``` -## Cause 2: The *config* file is in the wrong directory+## Cause 2: The config file is in the wrong directory -The *config* file is on your machine, but it's in a different directory from where the `az aks get-credentials` command and/or the kubectl tool expects it to be.+The config file is on your machine, but it's in a different directory from where the `az aks get-credentials` command and the kubectl tool expect it to be. -### Solution: Move the *config* file, save the credentials again, or change the KUBECONFIG environment variable+### Solution: Move the config file, save the credentials again, or change the KUBECONFIG environment variable  Take one or more of the following actions: -- Move the *config* file to the directory you want it to be in.+- Move the config file to the directory you want. - Run the `az aks get-credentials` command. Specify the location you want if it isn't the default location.      ```azurecli@@ -71,16 +73,16 @@ Take one or more of the following actions:     ``` - Use one of the following options to change where kubectl looks for configuration parameters: -    - Modify the `KUBECONFIG` environment variable to point to the *config* file's current location. For more information, see [Set the KUBECONFIG environment variable](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable).+    - Modify the `KUBECONFIG` environment variable to point to the config file's current location. For more information, see [Set the KUBECONFIG environment variable](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable).     - Run the [kubectl config](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#config) command with the `--kubeconfig=<config-file-location>` parameter. -## Cause 3: The *config* file has expired or is corrupted+## Cause 3: The config file is expired or corrupted -The *config* file is on your machine. It's also in the expected directory for the `az aks get-credentials` command and the kubectl tool. But the file is expired or corrupted.+The config file is on your machine. It's also in the expected directory for the `az aks get-credentials` command and the kubectl tool. But the file is expired or corrupted.  ### Solution: Reestablish the credentials -Reestablish the credentials, because the existing credentials might be expired or corrupted. In that case, you may run the `az aks get-credentials` command with the `--overwrite-existing` parameter.+Reestablish the credentials because the existing credentials might be expired or corrupted. In that case, run the `az aks get-credentials` command with the `--overwrite-existing` parameter.  ```azurecli az aks get-credentials --resource-group <cluster-resource-group> \

@@ -22,9 +22,9 @@ For managed operator installation outside this Terraform workflow, see [Install  ## What is Kueue? -[Kueue](https://kueue.sigs.k8s.io/) is a Kubernetes-native job queueing controller that manages workload admission based on resource quotas. Instead of letting every submitted job consume resources immediately, Kueue gates admission against defined quotas - jobs that fit run, jobs that don't wait in line. For a detailed overview of Kueue concepts and configuration, see [Kueue overview on AKS](kueue-overview).+[Kueue](https://kueue.sigs.k8s.io/) is a Kubernetes-native job queueing controller that manages workload admission based on resource quotas. Instead of letting every submitted job consume resources immediately, Kueue gates admission against defined quotas - jobs that fit run, jobs that don't wait in line. For a detailed overview of Kueue concepts and configuration, see [Managed Kueue extension overview](managed-kueue-extension-overview). -For managed installation and operation, see [Install the managed Kueue extension on AKS (preview)](kueue-overview).+For managed installation and operation, see [Install the managed Kueue extension on AKS (preview)](install-the-managed-kueue-extension-on-aks).  ## Solution architecture 

@@ -1,4 +1,4 @@-# Detecting and managing workload drift with Azure Kubernetes Fleet Manager resource placement | Microsoft Learn+# Managing workload drift with Azure Kubernetes Fleet Manager resource placement | Microsoft Learn  **Applies to**: ✔️ Fleet Manager with hub cluster @@ -10,24 +10,20 @@ Note  If you aren't already familiar with Fleet Manager's resource placement concepts, read the [cluster-scoped resource placement overview](concepts-resource-placement) and the [namespace-scoped resource placement overview](concepts-namespace-scoped-resource-propagation) before reading this article. -Important--Azure Kubernetes Fleet Manager preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. Azure Kubernetes Fleet Manager previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use.--Our data plane previews are released via our v1beta1 API. If you don't see preview objects, check that you're requesting the v1beta1 API when you interact with the Fleet Manager hub cluster.- ## Detect differences across a fleet  This section provides an overview of the `ReportDiff` apply mode, which can be used to evaluate configuration state of placed workload across a fleet at any time.  Using the `ReportDiff` mode, Fleet Manager checks for configuration differences between the [hub cluster](access-fleet-hub-cluster-kubernetes-api) workload definition and corresponding placed workloads on the member clusters, reporting the results in the resource placement status. +::: zone pivot="cluster-scope"+ ### ClusterResourcePlacement example  In the following sample, Fleet Manager reports differences for the namespace `web` for any member cluster with the placed namespace.  ```yml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata:   name: crp-reportdiff-sample@@ -56,12 +52,16 @@ Apply the `ClusterResourcePlacement` to your Fleet Manager hub cluster. kubectl apply -f crp-reportdiff-sample.yaml ``` +::: zone-end++::: zone pivot="namespace-scope"+ ### ResourcePlacement example  In the following sample, Fleet Manager reports differences for a deployment in the `web` namespace.  ```yml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ResourcePlacement metadata:   name: rp-reportdiff-sample@@ -89,28 +89,38 @@ Apply the `ResourcePlacement` to your Fleet Manager hub cluster. kubectl apply -f rp-reportdiff-sample.yaml ``` +::: zone-end+ ### Report on drifted clusters  Using the following commands you can determine which clusters have configuration drift. The [jq command](https://github.com/jqlang/jq) is used to format the output. +::: zone pivot="cluster-scope"+ **For ClusterResourcePlacement:**  Replace `crp-reportdiff-sample` with the name of your `ClusterResourcePlacement`.  ```bash-kubectl get clusterresourceplacement.v1beta1.placement.kubernetes-fleet.io crp-reportdiff-sample -o jsonpath='{.status.placementStatuses}' \+kubectl get clusterresourceplacement.v1.placement.kubernetes-fleet.io crp-reportdiff-sample -o jsonpath='{.status.placementStatuses}' \     | jq '[.[] | select (.diffedPlacements != null)] | map({clusterName, diffedPlacements})' ``` +::: zone-end++::: zone pivot="namespace-scope"+ **For ResourcePlacement:**  Replace `rp-reportdiff-sample` with the name of your `ResourcePlacement` and `web` with your namespace.  ```bash-kubectl get resourceplacement.v1beta1.placement.kubernetes-fleet.io rp-reportdiff-sample -n web -o jsonpath='{.status.placementStatuses}' \+kubectl get resourceplacement.v1.placement.kubernetes-fleet.io rp-reportdiff-sample -n web -o jsonpath='{.status.placementStatuses}' \     | jq '[.[] | select (.diffedPlacements != null)] | map({clusterName, diffedPlacements})' ``` +::: zone-end+ The command results in a response containing JSON similar to the following snippet. In this sample, we can see that someone has directly overwritten the owner label on this cluster.  ```json@@ -178,12 +188,14 @@ You can use an optional `comparisonOption` property to fine-tune how `whenToAppl - `PartialComparison`: only fields that are present on the hub cluster workload and on the target cluster workload are used for value comparison. Any extra unmanaged fields on the target cluster workload are ignored. This behavior is the default without an explicit `comparisonOption` setting. - `FullComparison`: all fields on the workload definition on the Fleet hub cluster must be present on the selected member cluster. If the target cluster has any extra unmanaged fields, then it fails comparison. +::: zone pivot="cluster-scope"+ ### ClusterResourcePlacement example  In the following sample, if a change is found in any field (managed or unmanaged), the placement fails.  ```yml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata:   name: crp-deploy-drift-sample@@ -207,12 +219,16 @@ spec:       unavailablePeriodSeconds: 1 ``` +::: zone-end++::: zone pivot="namespace-scope"+ ### ResourcePlacement example  In the following sample, if a change is found in any field (managed or unmanaged), the placement fails.  ```yml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ResourcePlacement metadata:   name: rp-deploy-drift-sample@@ -235,6 +251,8 @@ spec:       unavailablePeriodSeconds: 1 ``` +::: zone-end+ ### Placement behavior summary  The following table summarizes the placement behavior depending on the `whenToApply` and `comparisonOption` values selected.

@@ -1,8 +1,8 @@-# Tunnel connectivity issues - Azure | Microsoft Learn+# Troubleshoot tunnel connectivity issues in AKS - Azure | Microsoft Learn  ## Summary -Tunnel connectivity problems in Azure Kubernetes Service (AKS) can disrupt secure communication between cluster nodes and the control plane. This article explains how to troubleshoot and resolve these problems to restore expected cluster operations.+Tunnel connectivity issues in Azure Kubernetes Service (AKS) can disrupt secure communication between cluster nodes and the control plane. This article helps you troubleshoot and resolve these issues so that you can restore kubectl logs, exec, and port-forward functionality.  [Tunnel connectivity is used specifically for API server to kubelet communication](/en-us/azure/aks/faq#how-does-the-managed-control-plane-communicate-with-my-nodes-) (not general pod-to-pod) or ingress traffic. @@ -10,7 +10,7 @@ Tunnel connectivity problems in Azure Kubernetes Service (AKS) can disrupt secur  Note -Previously, the AKS tunnel component was `tunnel-front`. It's now migrated to the [Konnectivity service](https://kubernetes.io/docs/concepts/architecture/control-plane-node-communication/#konnectivity-service), an upstream Kubernetes component. For more information about this migration, see the [AKS release notes](https://github.com/Azure/AKS/releases/tag/2021-10-28).+Previously, the AKS tunnel component was `tunnel-front`. It migrated to the [Konnectivity service](https://kubernetes.io/docs/concepts/architecture/control-plane-node-communication/#konnectivity-service), an upstream Kubernetes component. For more information about this migration, see the [AKS release notes](https://github.com/Azure/AKS/releases/tag/2021-10-28).  [API Server VNet Integration](/en-us/azure/aks/api-server-vnet-integration) enables nodes to communicate directly with the API server through an internal load balancer, without requiring a tunnel. Therefore, the `konnectivity-agent` component isn't expected in clusters where Konnectivity isn't required. @@ -61,7 +61,7 @@ az vmss run-command invoke --resource-group <infra-or-MC-resource-group> \  If you use an NSG and you have specific restrictions, make sure that you add a security rule that allows traffic for port 10250 at the virtual network level. The following [Azure portal](https://portal.azure.com) image shows an example security rule: -![Screenshot of the Add inbound security rule pane in the Azure portal. The Destination port ranges box is set to 10250 for the new security rule.](media/tunnel-connectivity-issues/add-inbound-security-rule.png)+[![Screenshot of the Add inbound security rule pane in the Azure portal. The Destination port ranges box is set to 10250 for the new security rule.](media/tunnel-connectivity-issues/add-inbound-security-rule.png)](media/tunnel-connectivity-issues/add-inbound-security-rule.png#lightbox)  If you want to be more restrictive, you can allow access to port 10250 at the subnet level only. @@ -213,7 +213,7 @@ To use service diagnostics to view the SNAT ports, follow these steps: 5. Under **SNAT Connection and Port Allocation**, select **View details**. 6. If necessary, use the **Time Range** button to customize the time frame. -### Solution 4a: Make sure the application uses connection pooling+### Solution 4a: Ensure the application uses connection pooling  This behavior might occur because an application isn't reusing existing connections. Don't create one outbound connection per request. Such a configuration can cause connection exhaustion. Check whether the application code follows best practices and uses connection pooling. Most libraries support connection pooling. Therefore, you shouldn't have to create a new outbound connection per request. @@ -233,7 +233,7 @@ Note  This cause applies only to the `Konnectivity-agent` pods. -### Solution 6: Cluster Proportional Autoscaler for Konnectivity Agent+### Solution 5: Cluster Proportional Autoscaler for Konnectivity Agent  To manage scalability challenges in large clusters, implement the Cluster Proportional Autoscaler for your Konnectivity Agents. This approach aligns with industry standards and best practices. It ensures optimal resource usage and enhanced performance. @@ -298,6 +298,55 @@ kubectl get pod <pod-name> -n kube-system -o yaml | grep -A5 "resources:" kubectl edit deployment konnectivity-agent -n kube-system ``` +## Cause 6: Excessive API server traffic caused by Grafana Alloy log collection++### Symptoms++The AKS cluster experiences intermittent Konnectivity tunnel failures or degraded tunnel connectivity.++You might observe one or more of the following symptoms:++- Konnectivity agents become intermittently unhealthy or unresponsive.+- Kubernetes API server requests experience increased latency or timeouts.+- `kubectl` commands intermittently fail or take significantly longer than expected.+- Admission webhooks time out or fail.+- Aggregated API requests, such as `metrics.k8s.io`, fail intermittently.+- API server responses stall during periods of high log collection activity.++### Cause++This issue can occur when **Grafana Alloy is configured to collect container logs through the Kubernetes API server**, rather than reading them directly from the cluster nodes.++When Grafana Alloy uses API-based log collection, log retrieval requests route through the Kubernetes API server and the Konnectivity tunnel to reach the kubelets.++In clusters generating large volumes of logs, this configuration can introduce significant traffic through the tunnel infrastructure.++The resulting load can saturate Konnectivity tunnel resources, causing delayed or stalled API server responses and intermittent failures in operations that depend on tunnel connectivity.++Note++This issue is related to the log collection method used by Grafana Alloy, not to Loki itself. Loki is the log aggregation backend.++### Solution: Configure Grafana Alloy to collect logs directly from nodes++To prevent excessive traffic through the Kubernetes API server and Konnectivity tunnel, configure Grafana Alloy to collect container logs directly from the AKS nodes.++Use node-level log collection instead of Kubernetes API-based log streaming.++With this configuration:++1. Grafana Alloy runs on the cluster nodes, typically as a DaemonSet.+2. Alloy reads container logs directly from the node filesystem.+3. Collected logs are forwarded to the configured Loki endpoint.+4. Log collection no longer relies on the API server and Konnectivity tunnel for retrieving container log contents.++This approach reduces API server and Konnectivity tunnel traffic and helps prevent tunnel saturation caused by high-volume log collection.++For configuration guidance, see the Grafana documentation:++- [Grafana Alloy - Collect Kubernetes logs](https://grafana.com/docs/alloy/latest/collect/logs-in-kubernetes/)+- [Grafana Alloy - loki.source.file](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.source.file/)+ **Third-party contact disclaimer**  Microsoft provides third-party contact information to help you find additional information about this topic. This contact information may change without notice. Microsoft does not guarantee the accuracy of third-party contact information.

@@ -12,6 +12,75 @@ These updates cover security information related to the following AKS components - Azure Kubernetes Service Node Image (AKS Node Image) - Azure Kubernetes Service add-ons (AKS add-ons) +## AKS-2026-0008 AKS advisory for CVE-2026-111497++**Published date**: November 3, 2026++### Description++CVE-2026-111497 affects the Brotli decoder. Applications that decompress malicious Brotli data can expose memory, crash, or allow remote code execution.++### Affected components++- **AKS node OS:** Node components don't decompress untrusted Brotli data, so the installed library isn't exposed through this path.+- **AKS-managed Istio:** The add-on uses an unaffected Envoy version. No action is needed for this CVE.+- **Customer-managed workloads:** Applications, runtimes, and proxies might be affected if they use a vulnerable Brotli decoder.++### Resolutions++Rebuild affected container images with patched libraries and runtimes. Update affected customer-managed proxies. Redeploy the updated workloads. Updating the node OS doesn't fix packages inside containers.++Node OS update availability:++| Update | Version | Availability |+| --- | --- | --- |+| Azure Linux 3.0 node image (VHD) | `202610.23.0` | Includes the fix. Available in all regions listed in the [Azure Linux release tracker](https://releases.aks.azure.com/AzureLinux). |+| Ubuntu 22.04/24.04 node image (VHD) | `202610.30.0` (planned) | Not yet available. Monitor the [Ubuntu node image release tracker](https://releases.aks.azure.com/Ubuntu) for regional availability. |+| Ubuntu [`SecurityPatch` channel](../auto-upgrade-node-os-image#available-node-os-upgrade-channels) | `20261028T000000Z` | Rolling out. AKS expects availability in all regions by November 12, 2026. Track the [Ubuntu security patch rollout](https://releases.aks.azure.com/UbuntuSecurityPatch). |++Enable the [`SecurityPatch` node OS upgrade channel](../auto-upgrade-node-os-image#set-the-node-os-autoupgrade-channel-on-an-existing-cluster) to receive Ubuntu security patches. Use `NodeImage` or [upgrade node images manually](../upgrade-node-image) to apply a VHD when available in your region.++### References++- [Brotli vulnerability and Ubuntu fixes](https://ubuntu.com/security/CVE-2026-111497).+- [Upstream Istio advisory](https://istio.io/latest/news/security/istio-security-2026-007/).++## AKS-2026-0007 AKS advisory and mitigation guide for CVE-2026-99999 and CVE-2026-98432++**Published date**: November 2, 2026++### Description++This security bulletin covers two vulnerabilities in the Linux kernel's IPv4 networking subsystem that affect AKS Linux nodes.++CVE-2026-98432 is a use-after-free vulnerability in the code that sets up incoming IPv4 TCP connections. This vulnerability can let an unauthenticated remote attacker run commands as root on Linux servers with a multipath TCP (MPTCP) listener.++CVE-2026-99999 lets a local attacker or untrusted guest virtual machine send packets with malformed checksum metadata. This vulnerability can allow arbitrary code execution on the host, a denial-of-service crash, or disclosure of sensitive memory.++### References++- [CVE-2026-99999](https://ubuntu.com/security/CVE-2026-99999).+- [CVE-2026-98432](https://ubuntu.com/security/CVE-2026-98432).++### Affected components++This advisory applies to AKS node images.++**Affected versions**++- Azure Linux: node images earlier than VHD `202610.23.0`.+- Ubuntu: all currently supported Ubuntu node images.++**Resolutions**++- **Azure Linux:** The fix is in kernel `6.6.159.1-1`, which ships in VHD `202610.23.0`. This image is available in all regions listed in the [Azure Linux release tracker](https://releases.aks.azure.com/AzureLinux). [Upgrade your node images](../upgrade-node-image) to apply the fix. Consider the [`SecurityPatch` node OS upgrade channel](../auto-upgrade-node-os-image#available-node-os-upgrade-channels) to receive security fixes faster.+- **Ubuntu:** AKS is waiting for a fix from upstream Ubuntu. A VHD release containing the official fix is tentatively expected after November 15, 2026. This bulletin will be updated when it's available.++    - An interim fix using Ubuntu's `proposed` repository rolls out in this week's VHD release. This bulletin will be updated as soon as the VHD rollout begins.+    - **Exceptions:**The interim fix doesn't cover the following:+        - **Ubuntu FIPS node images:** FIPS kernels won't be available early. They'll be included only after Canonical releases the official kernels.+        - **Kata Gen2:** The guest (UVM) kernel remains on version `.52` and doesn't include the fix until the official release is available.+ ## AKS-2026-0006 OpenSSL end-of-life scanner findings in AKS-managed add-on images  **Published date**: October 21, 2026

@@ -48,7 +48,7 @@ The following table lists the feature differences between the different ingress | **API** | Ingress API | Ingress API and Gateway API | [Istio Ingress API](https://istio.io/latest/docs/reference/config/networking/gateway/) | | **Hosting** | In-cluster | Azure hosted | In-cluster | | **Scaling** | Autoscaling | Autoscaling | Autoscaling |-| **Load balancing** | Internal/External | External | Internal/External |+| **Load balancing** | Internal/External | Internal/External | Internal/External | | **SSL termination** | In-cluster | Yes: Offloading and E2E SSL | In-cluster | | **mTLS** | N/A | Yes: frontend and backend | Yes | | **Static IP address** | Yes | FQDN (no static IP) | N/A |@@ -62,7 +62,7 @@ The following table lists the different scenarios where you might use each ingre | Ingress option | When to use | | --- | --- | | **Managed NGINX - Application Routing add-on** | • In-cluster hosted, customizable, and scalable NGINX ingress controllers.  • Basic load balancing and routing capabilities.  • Internal and external load balancer configuration.  • Static IP address configuration.  • Integration with Azure Key Vault for certificate management.  • Integration with Azure DNS Zones for public and private DNS management.  • Supports the Ingress API. |-| **Application Gateway for Containers** | • Azure hosted ingress gateway.  • Flexible deployment strategies managed by the controller or bring your own Application Gateway for Containers.  • Advanced traffic management features such as automatic retries, availability zone resiliency, mutual authentication (mTLS) to backend target, traffic splitting / weighted round robin, and autoscaling.  • Integration with Azure Key Vault for certificate management.  • Integration with Azure DNS Zones for public and private DNS management.  • Supports the Ingress and Gateway APIs. |+| **Application Gateway for Containers** | • Azure hosted ingress gateway.  • Web Application Firewall (WAF) support for application security.  • AI gateway capabilities for inference and Model Context Protocol (MCP) workloads.  • Flexible deployment strategies managed by the controller or bring your own Application Gateway for Containers.  • Advanced traffic management features such as automatic retries, availability zone resiliency, mutual authentication (mTLS) to backend target, traffic splitting / weighted round robin, and autoscaling.  • Integration with Azure Key Vault for certificate management.  • Integration with Azure DNS Zones for public and private DNS management.  • Supports the Ingress and Gateway APIs. | | **Istio Ingress Gateway** | • Based on Envoy, when using with Istio for a service mesh.  • Advanced traffic management features such as rate limiting and circuit breaking.  • Support for mTLS. |  Note

@@ -17,7 +17,7 @@ If using the AKS add-on, updates are automatically applied to the cluster.  | ALB Controller Version | Gateway API Version | Gateway API Inference Extension Version | Minimum Kubernetes Version | Release notes | | --- | --- | --- | --- | --- |-| 1.12.4 | v1.5.1 | v1.3.1 | v1.32 | Security patches |+| 1.13.0 | v1.6.2 | v1.6.2 | v1.32 | New features: JWT validation, MCP gateway, Gateway API ListenerSet, LLM-D support in Gateway API Inference Extension, Gateway API v1.6.2, Gateway API Inference Extension 1.6.2, security patches, [WAF sectionName handling fix](https://github.com/Azure/AKS/issues/5996), and additional bug fixes |  ## AKS add-on releases @@ -34,6 +34,7 @@ Reference: [AKS Managed Gateway API installation](/en-us/azure/aks/managed-gatew  | ALB Controller Version | Gateway API Version | Gateway API Inference Extension Version | Minimum Kubernetes Version | Release Notes | | --- | --- | --- | --- | --- |+| 1.12.4 | v1.5.1 | v1.3.1 | v1.32 | Security patches | | 1.12.2 | v1.5.1 | v1.3.1 | v1.32 | Security patches | | 1.12.1 | v1.5.1 | v1.3.1 | v1.32 | Security patches, bug fixes [helm chart pre-delete tolerations](https://github.com/Azure/AKS/issues/5729)[session affinity cookie fix](https://github.com/Azure/AKS/issues/5805), request mirror | | 1.11.7 | v1.5.1 | v1.3.1 | v1.27 | Security patches |

@@ -70,6 +70,8 @@ az aks nodepool scale --cluster-name <cluster-name> \  Define the gateway configuration by creating a `StaticGatewayConfiguration` custom resource. This configuration specifies which node pool and public IP prefix to use. +If you supply an existing public IP prefix, configure its access and namespace authorization before applying the manifest.+ | Field | Required | Description | | --- | --- | --- | | `gatewayNodepoolName` | Yes | Name of the gateway node pool that handles egress traffic. |@@ -105,15 +107,46 @@ kubectl get staticgatewayconfiguration <gateway-config-name> \     --output jsonpath='{.status.egressIpPrefix}' ``` -Tip+### Use your own public IP prefix++To use an existing public IP prefix, set `spec.publicIpPrefixId` to its Azure resource ID.++#### Grant Azure access to the cluster identity++Grant the AKS cluster identity the **Network Contributor** role at the existing public IP prefix's resource scope.++#### Authorize the namespace on the prefix++Ask the prefix administrator to set the Azure resource tag `kube-egress-gateway-allowed-namespaces` to the Kubernetes namespaces approved to use the prefix. The tag accepts comma- or semicolon-separated namespace names. For example, `team-a,team-b` authorizes configurations in those two namespaces. The value `*` authorizes all namespaces; use explicit namespace names when access should be limited. -If you omit `publicIpPrefixId`, the controller creates a public IP prefix. Run `kubectl describe StaticGatewayConfiguration <gateway-config-name> --namespace <namespace>` and check `Egress Ip Prefix` in the status. To use an existing prefix, specify its resource ID in `publicIpPrefixId` and grant the AKS cluster identity the Network Contributor role.+Set the tag:++```azurecli+PREFIX_ID=$(az network public-ip prefix show \+    --resource-group <prefix-resource-group> \+    --name <prefix-name> \+    --query id \+    --output tsv)++az tag update \+    --resource-id "$PREFIX_ID" \+    --operation Merge \+    --tags "kube-egress-gateway-allowed-namespaces=team-a,team-b"+```++Replace `team-a,team-b` with the complete list of approved namespaces, including any existing entries that should remain authorized. The merge operation preserves other resource tags, but replaces the value of this tag.++Important++Permission to modify this tag grants control over namespace authorization. Limit tag changes to trusted administrators. The tag doesn't grant Azure access to the cluster identity or Kubernetes access to create `StaticGatewayConfiguration` resources; configure those permissions separately.++This tag applies only when you supply an existing public IP prefix through `spec.publicIpPrefixId`. It isn't required for automatically created public IP prefixes or private IP configurations with `spec.provisionPublicIps: false`.  ## Configure Static Egress Gateway with private IP addresses  Important -Static private IP support is in preview and requires clusters running Kubernetes version 1.34 or later.+Static private IP support requires clusters running Kubernetes version 1.34 or later.  If you need to keep egress traffic on private addresses, create a gateway node pool that uses the `VirtualMachines` VM set type. Node public IPs are disabled by default. The `StaticGatewayConfiguration` resource in the following steps separately disables public IP provisioning for egress. @@ -207,6 +240,23 @@ In the `status` section, verify the following fields: - `gatewayServerProfile.ip` and `gatewayServerProfile.port` identify the gateway endpoint. - `gatewayServerProfile.publicKey` contains the public key for the WireGuard configuration, which defines the encrypted tunnel between pods and the gateway node pool. +### Check BYO prefix authorization warnings++A Kubernetes Warning event with reason `BYOPrefixAuthorizationRequired` on a `StaticGatewayConfiguration` indicates that its BYO prefix doesn't authorize the configuration's namespace.++View events for a configuration:++```bash+kubectl describe staticgatewayconfiguration <gateway-config-name> -n <namespace>+```++List authorization warnings in a namespace:++```bash+kubectl get events -n <namespace> \+    --field-selector involvedObject.kind=StaticGatewayConfiguration,reason=BYOPrefixAuthorizationRequired+```+ ## Delete a gateway node pool  Before you delete a gateway node pool, remove the gateway annotation from pod templates and replace the existing pods, or delete the workloads that use it. Then, delete every `StaticGatewayConfiguration` that references the node pool. The command waits for the controller's cleanup finalizers to complete before it returns:

@@ -2,13 +2,6 @@  This article provides best practices for monitoring and interpreting GPU signals on Azure Kubernetes Service (AKS). Instead of looking at NVIDIA GPU metrics in isolation, you correlate signals across utilization, memory, and workload context to improve long-term performance and node efficiency. -Important--AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:--- [AKS support policies](/en-us/azure/aks/support-policies)-- [Azure support FAQ](/en-us/azure/aks/faq)- ## Understand GPU utilization versus saturation  Don't treat the NVIDIA DCGM metric `DCGM_FI_DEV_GPU_UTIL` as a direct efficiency score. `DCGM_FI_DEV_GPU_UTIL` only indicates how often kernels are active, so it doesn't tell you whether the workload is compute-efficient. You get more accurate guidance by correlating utilization signals instead of reading them independently. Compare `DCGM_FI_DEV_GPU_UTIL` with `DCGM_FI_PROF_SM_ACTIVE`, and then compare `DCGM_FI_PROF_SM_ACTIVE` with `DCGM_FI_PROF_DRAM_ACTIVE` to identify whether your bottleneck is compute, memory, or launch and synchronization overhead.

@@ -1090,7 +1090,7 @@ status: Here's a comprehensive example showing the complete status of a ResourcePlacement:  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ResourcePlacement metadata:   name: web-app-placement

@@ -1,81 +1,139 @@ # Troubleshoot Azure Kubernetes Service (AKS) Workloads with Natural Language in AKS Desktop (preview) - Azure Kubernetes Service | Microsoft Learn -AKS desktop includes an AI-powered troubleshooting assistant that provides natural language diagnostics for your AKS workloads. By securely connecting to your cluster, the assistant can analyze logs, events, and metrics to identify issues and recommend resolutions in plain language. This allows developers and operators to quickly understand and fix problems without needing deep Kubernetes expertise. This article walks you through how to enable and use the AI troubleshooting assistant in AKS desktop.+The AI Assistant in AKS desktop provides enhanced natural language troubleshooting for AKS workloads. Powered by the Headlamp AI Assistant, it delivers AKS-specific skills, cluster-aware observability, guided troubleshooting workflows, and seamless integration with your existing GitHub Copilot entitlement. -Important+After securely connecting to your cluster, the assistant combines language models with AKS-specific context, logs, events, and metrics to identify issues and recommend resolutions in plain language. This approach allows developers and operators to quickly understand and fix problems without needing deep Kubernetes expertise. This article walks you through how to enable and use the AI Assistant in AKS desktop. -AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:+## Prerequisites -- [AKS support policies](/en-us/azure/aks/support-policies)-- [Azure support FAQ](/en-us/azure/aks/faq)+Before you enable the AI Assistant in AKS desktop, make sure you have: -## What is the AI troubleshooting assistant?+- [AKS desktop v0.10.0 or later installed](aks-desktop-overview#install-aks-desktop).+- Access to one or more Kubernetes clusters that you want to investigate.+- The AI Assistant plugin enabled in **Settings** > **Plugins**. The plugin is enabled by default.+- One of the following provider configurations:+    - **GitHub Copilot Auto Detect (recommended)**: A GitHub Copilot subscription, [GitHub CLI (`gh`)](https://cli.github.com/) installed on your machine, and GitHub CLI authenticated with your GitHub account.+    - **Another supported AI provider**: Access to a supported provider or OpenAI-compatible endpoint. -The AKS desktop troubleshooting assistant is built into [AKS desktop](aks-desktop-overview) and provides a guided experience for diagnosing issues in your cluster. With the assistant, you can:+## What is the AI Assistant?++The AI Assistant is bundled with AKS desktop and provides a guided experience for diagnosing issues in your cluster. The assistant helps you: -- Select a project, namespace, or workload to investigate. - Run scoped diagnostics that analyze logs, events, and metrics. - Receive AI-generated summaries with clear reasoning and evidence. - Preview recommended actions and apply them with confidence.-- Use your own language models or connect to Azure managed options.-- Work alongside existing dashboards in AKS desktop for a cohesive troubleshooting experience.+- Use GitHub Copilot when available for a streamlined setup experience.+- Use cluster-aware observability and AKS-specific skills to bring additional context and best practices into investigations.++## How does the AI Assistant work? -Warning+You have two options for enabling the AI Assistant in AKS desktop: -This plugin is in early development and isn't ready for production use. Using it might incur costs from your AI provider.+- Use **GitHub Copilot (recommended)**. If GitHub CLI is installed and authenticated with an account that has a GitHub Copilot subscription, AKS desktop can detect and configure GitHub Copilot as the provider.+- **Another supported AI provider**: Manually configure a supported provider or OpenAI-compatible endpoint. -## How does the AI troubleshooting assistant work?+For either option, AKS desktop can provide cluster context and enabled AKS skills to the configured provider. -You have two options for enabling the AI troubleshooting assistant in AKS desktop:+The AI Assistant runs as part of AKS desktop and doesn't deploy an agent or other workload to your cluster. AKS desktop connects to the selected cluster through the Kubernetes API by using the credentials in your kubeconfig. The assistant is limited to the operations allowed by your Kubernetes RBAC permissions. -- Use the **[Azure AKS Agentic CLI Agent](https://blog.aks.azure.com/2025/08/15/cli-agent-for-aks)** (recommended for AKS): This is an AI-powered tool that helps diagnose, troubleshoot, and optimize AKS clusters using natural language queries. It provides root cause analysis by connecting to your AKS cluster (using your permissions) and remediation suggestions. This is backed by an LLM of your choice.-- Use the **AKS desktop Agent**: AKS desktop acts as an Agent itself, providing context to your model of your choice.+## Enable GitHub Copilot (Auto Detect) -### Which option should I choose?+1. Install GitHub CLI (`gh`) on your machine and sign in to your GitHub account.+2. In AKS desktop, go to **Settings** > **Plugins** > **AI Assistant**.+3. Select **Auto Detect**.+4. Verify that GitHub Copilot appears as the configured provider. -Asking a standalone LLM a question is similar to asking an expert for advice based only on what you tell them. In contrast, the AKS Agentic CLI integrates the HolmesGPT agentic framework with the AKS Model Context Protocol (MCP) server. Combined, they know which tools, troubleshooting techniques, and other MCP servers to use, enabling the agent to securely access live cluster state, logs, metrics, and Azure resource data to perform iterative, context-aware investigations within Role-Based Access Control (RBAC) boundaries rather than relying solely on static user-provided input.+    ![Screenshot of AKS desktop AI Assistant settings showing the Auto Detect button.](media/aks-desktop-app/aks-desktop-github-copilot-auto-detect.png) -By default, the AKS Agentic CLI connects to a hosted LLM endpoint using a customer‑provided API key, but organizations with strict data residency or compliance requirements might instead choose to deploy a model within their own environment.+## Configure your own AI provider -In the following sections, we provide end-to-end steps for setting up both options, but we recommend the AKS Agentic CLI for the best experience with AKS desktop.+1. In AKS desktop, go to **Settings** > **Plugins** > **AI Assistant**.+2. Select **Add Provider**.+3. Select the AI provider you want to use.+4. Enter the required configuration information for your provider.+5. Verify that the provider appears in the configured providers list. -## Enable the AI troubleshooting assistant in AKS desktop+    ![Screenshot of the AKS desktop provider selection dialog showing the supported AI providers.](media/aks-desktop-app/aks-desktop-select-ai-provider.png) -1. Go to **Settings** > **Plugins**, and make sure the AI Assistant plugin is enabled.-2. Select **AI Assistant** in the list of plugins.-3. Toggle **Preview** to the enabled state. After this, the AI Assistant icon appears in the top right.+## Enable Kubernetes API requests for the assistant -## Configure a provider for the AI troubleshooting assistant+The **Kubernetes API Request** setting allows the AI Assistant to make live Kubernetes API requests against your cluster. When enabled, the assistant can query resource state, pod logs, events, and metrics to provide context-aware analysis. If disabled, the assistant can only reason from information you manually provide in the chat. -1. Go to **Settings** > **Plugins** > **AI Assistant** > **Add provider**.+Read requests can run automatically. Requests that create, update, or delete Kubernetes resources require your approval and are still subject to your Kubernetes RBAC permissions. -    ![Screenshot of the AKS desktop Settings panel showing the AI Assistant plugin with the Enable toggle.](media/aks-desktop-app/aks-desktop-enable-ai-assistant.png)-2. Choose your provider option:+1. Go to **Settings** > **Plugins** > **AI Assistant**.+2. Enable the **Kubernetes API Request** toggle. -    - **Use the Azure AKS Agentic CLI Agent** (recommended for AKS): Install the [Agentic CLI Agent](agentic-cli-for-aks-install) on your cluster, configure it to connect to a model, and ensure the cluster running the Agentic CLI Agent is a registered cluster in AKS desktop (you should see it in your cluster list).-    - **Use an existing model**: Configure the provider so that AKS desktop can act as an agent and send cluster context to your existing model.+    ![Screenshot of AKS desktop AI Tools settings showing Kubernetes API Request enabled.](media/aks-desktop-app/aks-desktop-kubernetes-api-request.png) -## Enable Kubernetes API requests for the assistant+## Enable MCP servers -The **AI tool Kubernetes Requests** setting allows the AI Assistant to make live Kubernetes API requests against your cluster. When enabled, the assistant can query resource state, pod logs, events, and metrics to provide context-aware analysis. If disabled, the assistant can only reason from information you manually provide in the chat.+Model Context Protocol (MCP) servers allow the AI Assistant to connect to external tools and data sources. MCP servers extend the capabilities of the AI Assistant by providing additional context and actions that can be used during troubleshooting and investigation workflows. -1. Go to **Settings** > **Plugins** > **AI Assistant**.-2. Enable the **AI tool Kubernetes Requests** toggle.+AKS desktop supports connections to MCP servers, but MCP servers aren't enabled by default.++MCP servers are optional and aren't required for the AI Assistant's built-in AKS troubleshooting and cluster-aware observability. Configure an MCP server only when you need to connect the AI Assistant to a specific external tool or data source.++### Choose an MCP server++Choose a server based on the capability you need:++- Use Flux Operator MCP to investigate Flux GitOps resources and workflows.+- Use Inspektor Gadget for eBPF tracing and runtime diagnostics.++Before you configure an MCP server in AKS desktop, install the server and confirm that it can run as a local command. If you don't need one of these additional integrations, continue using the AI Assistant without an MCP server.++### Configure an MCP server++1. In AKS desktop, go to **Settings** > **Plugins** > **AI Assistant**.+2. View **MCP Servers**.+3. Turn on **Enable MCP Servers**.+4. Select **Add MCP Server**.+5. Enter the required configuration information for the MCP server.+6. Configure the approval settings for the server.++    ![Screenshot of AKS desktop MCP Servers settings showing the Enable MCP Servers toggle and Add MCP Server button.](media/aks-desktop-app/aks-desktop-mcp-servers.png)++### MCP server approvals++When an MCP server performs an action on behalf of the AI Assistant, AKS desktop can require approval before the action is executed. Configure approval settings for each MCP server.++By default, **Auto Approve Actions** is disabled. The AI Assistant displays each requested action in the chat window and requires your approval before the MCP server can execute it.++Enable **Auto Approve Actions** for a specific MCP server to automatically approve requests from that server. When enabled, the AI Assistant can use MCP tools without prompting for approval for each request.++## Enable skill repositories++Skills provide instructions and supporting information for specific tasks. AKS desktop includes an AKS skill from the Azure Skills repository, which is enabled by default. You can enable other skill repositories or add a custom repository.++1. In AKS desktop, go to **Settings** > **Plugins** > **AI Assistant**.+2. Review the list of built-in skill repositories.+3. Use the toggle next to each repository to enable or disable repositories.+4. Select **Add Repository** to add a custom repository.+5. Enter the repository URL and save your changes.++    ![Screenshot of AKS desktop skill settings showing the Azure Skills repository configured with the azure-kubernetes path and enabled.](media/aks-desktop-app/aks-desktop-azure-skills-repository.png)++## Proactive diagnosis++Proactive diagnosis is a separate, opt-in feature that's disabled by default. When enabled, it uses one-shot requests through Headlamp's Kubernetes API proxy to poll selected clusters for recent events with a type of `Warning` or `Error` rather than continuously watching the clusters. -## Set the AI Assistant to use the AKS Agentic CLI Agent+The first check occurs shortly after the AI backend is ready, and the scheduler checks once per minute. When qualifying events are found, the assistant diagnoses up to three of the most recent distinct affected workloads sequentially. After a diagnosis cycle completes, a 20-minute cooldown begins before another cycle can run. If no qualifying events are found, no diagnosis cycle starts, so the lightweight event query can run again at the next one-minute check. Successful results are cached to avoid repeatedly diagnosing the same resource. -1. Open the AI Assistant.-2. Select **Agent**.-3. Select the cluster where the Agentic CLI Agent is running.+### Enable proactive diagnosis -    ![Screenshot of the AKS desktop AI Assistant panel with Agent mode selected and a cluster dropdown.](media/aks-desktop-app/aks-desktop-set-agent-mode.png)+1. In AKS desktop, go to **Settings** > **Plugins** > **AI Assistant**.+2. Turn on the **Proactive Diagnosis** toggle.+3. Review detected problems and recommendations surfaced by the AI Assistant.+4. Select a recommendation to investigate or begin troubleshooting. -## Investigate degraded resources+    ![Screenshot of AKS desktop AI Assistant settings showing the Proactive Diagnosis toggle.](media/aks-desktop-app/aks-desktop-proactive-diagnosis.png) -The assistant is context-aware. When you see errors or warnings highlighted in the interface, select the resource and ask the assistant for an explanation and recommended resolution.+## Disable the AI Assistant -1. Go to the cluster resource (for example a service) or go to the AKS desktop Project and select the workload.-2. Select the Kubernetes resource that's degraded or showing errors in the event log.-3. Select **Agent** and start chatting.+1. Go to **Settings** > **Plugins**.+2. Turn off **AI Assistant**.+3. Select **Save & Apply**.+4. Reload AKS desktop if prompted. -    ![Screenshot of a chat with natural language feature in AKS desktop.](media/aks-desktop-app/aks-desktop-ai-assistant.png)+    ![Screenshot of the AKS desktop plugin list showing the AI Assistant toggle turned off.](media/aks-desktop-app/aks-desktop-disable-ai-assistant.png)

@@ -1,18 +1,22 @@-# Client IP address can't access the API server - Azure | Microsoft Learn+# Client IP address cant access the API server for AKS - Azure | Microsoft Learn  ## Summary -This article explains how to troubleshoot when a client IP address can't access the AKS API server so you can reconnect to your Azure Kubernetes Service (AKS) cluster.+This article explains how to troubleshoot when a client IP address can't access the API server for Azure Kubernetes Service (AKS) so you can restore connectivity to your cluster.  ## Prerequisites +Ensure the following prerequisites are met before troubleshooting access to the AKS API server:+ - [Azure CLI](/en-us/cli/azure/install-azure-cli). - The client URL ([curl](https://techcommunity.microsoft.com/t5/containers/tar-and-curl-come-to-windows/ba-p/382409)) tool.  ## Symptoms +The following symptoms indicate that your client IP address can't access the AKS API server.+ # [Azure portal](#tab/azure-portal)-When you try to access Kubernetes resources such as mamespaces and workloads from the Azure portal, you might encounter the following errors:+When you try to access Kubernetes resources such as namespaces and workloads from the [Azure portal](https://portal.azure.com), you might encounter the following error.  > > Network error@@ -22,7 +26,7 @@ When you try to access Kubernetes resources such as mamespaces and workloads fro [![Screenshot of the AKS portal view showing a network error when connecting to the API server from namespaces and workloads.](media/client-ip-address-cannot-access-api-server/network-error.png)](media/client-ip-address-cannot-access-api-server/network-error.png#lightbox)  # [Azure CLI](#tab/azure-cli)-When you try to connect to a cluster using the Azure CLI, you might see the following errors:+When you try to connect to a cluster by using Azure CLI, you might see the following errors.  ```output "Unhandled Error" err="couldn't get current server API group list: Get \"https://<API-SERVER-FQDN>:443/api?timeout=32s\": dial tcp <API-SERVER-IP>:443: i/o timeout"@@ -36,7 +40,7 @@ Unable to connect to the server: dial tcp <API-SERVER-IP>:443: connectex: A conn  ## Cause -[API server-authorized IP ranges](/en-us/azure/aks/api-server-authorized-ip-ranges) may have been enabled on the cluster's API server, but the client's IP address wasn't included in the IP ranges. To check whether this feature has been enabled, see if the following [az aks show](/en-us/cli/azure/aks#az-aks-show) command in Azure CLI produces a list of IP ranges:+You might have enabled [API server-authorized IP ranges](/en-us/azure/aks/api-server-authorized-ip-ranges) on the cluster's API server, but you didn't include the client's IP address in the IP ranges. To check whether this feature is enabled, see if the following [az aks show](/en-us/cli/azure/aks#az-aks-show) command in Azure CLI returns a list of IP ranges.  ```azurecli az aks show --resource-group ${RG_NAME} \@@ -50,20 +54,23 @@ Look at the cluster's API server-authorized ranges, and add your client's IP add  Note -1. Do you access the API server from a corporate network where traffic is routed through a proxy server or firewall? Then ask your network administrator before you add your client IP address to the list of authorized ranges for the API server.-2. Also ask your cluster administrator before you add your client IP address, because there might be security concerns with adding a temporary IP address to the list of authorized ranges.+Do you access the API server from a corporate network where traffic routes through a proxy server or firewall? If so, ask your network administrator before you add your client IP address to the list of authorized ranges for the API server. Also, ask your cluster administrator before you add your client IP address, because there might be security concerns with adding a temporary IP address to the list of authorized ranges.  # [Azure portal](#tab/azure-portal)-1. Navigate to the cluster from the Azure portal.-2. In the left menu, locate **Settings** and then select **Networking**.+Follow these steps to add your client IP address to the authorized IP ranges for the AKS API server:++1. Go to the cluster in the Azure portal.+2. In the left menu, find **Settings** and then select **Networking**. 3. On the **Networking** page, select the **Overview** tab. 4. Select **Manage** under **Resource settings**.-5. In the **Authorized IP ranges** pane, add your client IP address as shown in the following screenshot:+5. In the **Authorized IP ranges** pane, add your client IP address as shown in the following screenshot. -    [![Screenshot of Authorized-ip-ranges pane.](media/client-ip-address-cannot-access-api-server/authorized-ip-ranges.png)](media/client-ip-address-cannot-access-api-server/authorized-ip-ranges.png#lightbox)+    [![Screenshot of the Authorized IP ranges pane in the Azure portal where you add a client IP address to the AKS API server.](media/client-ip-address-cannot-access-api-server/authorized-ip-ranges.png)](media/client-ip-address-cannot-access-api-server/authorized-ip-ranges.png#lightbox)  # [Azure CLI](#tab/azure-cli)-1. Get your client IP address by running this [curl](https://curl.se/docs/manpage.html) command:+Follow these steps to add your client IP address to the authorized IP ranges for the AKS API server using Azure CLI:++1. Get your client IP address by running this [curl](https://curl.se/docs/manpage.html) command.      ```azurecli     export CLIENT_IP=$(curl --silent https://ipinfo.io/ip | tr -d '\n')@@ -75,7 +82,7 @@ Note     ```output     0.255.127.63     ```-2. Update the API server-authorized range with the [az aks update](/en-us/cli/azure/aks#az-aks-update) command in Azure CLI, using your client IP address:+2. Update the API server-authorized range with the [az aks update](/en-us/cli/azure/aks#az-aks-update) command in Azure CLI, using your client IP address.      ```azurecli     az aks update --resource-group $RG_NAME \

@@ -1,10 +1,10 @@-# Resolve az aks command invoke failures - Azure | Microsoft Learn+# Resolve az aks command invoke failures for AKS - Azure | Microsoft Learn  ## Summary -This article explains how to resolve [az aks command invoke](/en-us/cli/azure/aks/command#az-aks-command-invoke) failures in Azure CLI so you can restore access to your AKS cluster, especially a [private AKS cluster](/en-us/azure/aks/command-invoke).+This article explains how to resolve [az aks command invoke](/en-us/cli/azure/aks/command#az-aks-command-invoke) failures in Azure CLI so you can restore access to your Azure Kubernetes Service (AKS) cluster, especially a [private AKS cluster](/en-us/azure/aks/command-invoke). -Other connection methods need to use extra configuration components, as shown in the following table.+Other connection methods need extra configuration components, as shown in the following table.  | Connection methods | Extra configuration component | | --- | --- |@@ -18,30 +18,32 @@ When you run the `az aks command invoke` command, Azure CLI automatically create  ## Prerequisites +Ensure you have the following prerequisites installed and configured:+ - [Azure CLI](/en-us/cli/azure/install-azure-cli). - The Kubernetes [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) tool. To install kubectl by using Azure CLI, run the [az aks install-cli](/en-us/cli/azure/aks#az-aks-install-cli) command.  ## Symptoms -The following table lists common `az aks command invoke` error messages. Each error message has a link to the section that describes why the error is occurring, and how to fix it.+The following table lists common `az aks command invoke` error messages. Each error message links to the section that describes why the error occurs and how to fix it.  | Error message | Link | | --- | --- |-| Operation returned an invalid status 'Not Found' | Cause 1: The pod can't be created because of node or resource constraints |+| Operation returned an invalid status 'Not Found' | Cause 1: Node or resource constraints prevent pod creation | | Failed to run command in managed cluster due to kubernetes failure. details: admission webhook "validation.gatekeeper.sh" denied the request: &lt;policy-specific-message&gt; | Cause 2: Azure Policy doesn't allow the pod creation | | Error from server (Forbidden): namespaces is forbidden: User "&lt;ID&gt;" cannot list resource "&lt;resource&gt;" in API group "" at the cluster scope | Cause 3: Required roles aren't granted | | Failed to connect to MSI. Please make sure MSI is configured correctly.Get Token request returned: Response [400]; | Cause 4: There's a Cloud Shell issue | -## Cause 1: The pod can't be created because of node or resource constraints+### Cause 1: Node or resource constraints prevent pod creation -The operation returns a `Not Found` status because the `command-<ID>` pod can't reach a successful state, such as `Running`. (In many cases, the pod stays in the `Pending` state.) In this case, the nodes aren't able to schedule the pod. This scenario can have different causes, such as the following causes:+The operation returns a `Not Found` status because the `command-<ID>` pod can't reach a successful state, such as `Running`. In many cases, the pod stays in the `Pending` state. In this case, the nodes can't schedule the pod. This scenario can have different causes, such as the following causes:  - Resource constraints - Nodes that have a `NotReady` or `SchedulingDisabled` state - Nodes that have taints that the pod can't tolerate - Other causes -Here are sample error messages for `KubernetesPerformanceError` or `KubernetesOperationError`:+Here are sample error messages for `KubernetesPerformanceError` or `KubernetesOperationError`.  ```output (KubernetesPerformanceError) Failed to run command due to cluster perf issue, container command-357ebsdfsd342869 in aks-command namespace did not start within 30s on your cluster, retry may helps. If issue persist, you may need to tune your cluster with better performance (larger node/paid tier).@@ -49,22 +51,33 @@ Code: KubernetesPerformanceError Message: Failed to run command due to cluster perf issue, container command-357ebc50d40c47a4a247ab6e067d2869 in aks-command namespace did not start within 30s on your cluster, retry may helps. If issue persist, you may need to tune your cluster with better performance (larger node/paid tier). ``` -#### Solution 1: Change the configuration so that you can schedule and run the pod+When you run `az aks command invoke`, AKS creates a `command-<ID>` pod in the `aks-command` namespace. The cluster must schedule the pod onto an eligible node.++The pod can remain in a `Pending` state if no node satisfies its scheduling requirements. Possible reasons include:++- Nodes are `NotReady` or marked as unschedulable.+- Nodes don't have sufficient allocatable CPU or memory to satisfy the pod's resource requests.+- Nodes have taints that the `command-<ID>` pod doesn't tolerate.+- Nodes don't satisfy the pod's node selector, affinity, or other scheduling constraints.++### Solution 1: Change the configuration so you can schedule and run the pod -Make sure that the `command-<ID>` pod can be scheduled and run by adjusting the configuration. For example:+Adjust the configuration to ensure that the cluster can schedule and run the `command-<ID>` pod. The following examples can help you resolve this problem: -- Increase the node pool size and make sure it has no pod secluding constraints like taints so that the `command-<ID>` pod can be deployed.-- Adjust resource requests and limits in your pod specifications.+- Increase the node pool size if the cluster needs more capacity, and ensure that an eligible node doesn't have scheduling constraints, such as untolerated taints, that prevent the cluster from scheduling the `command-<ID>` pod.+- Ensure that eligible nodes have sufficient allocatable CPU and memory for the `command-<ID>` pod. If necessary, scale the node pool or adjust the resource requests of other workloads to make capacity available. -## Cause 2: Azure Policy doesn't allow the pod creation+This guidance avoids implying that you should modify the resource requests or limits of the automatically generated `command-<ID>` pod.++### Cause 2: Azure Policy doesn't allow the pod creation  If you have specific Azure policies, the `az aks command invoke` command can fail because of a disallowed configuration in the `command-<ID>` pod. For example, you might have an Azure policy that requires a read-only root file system or other specific configuration. -#### Solution 2: Exempt the namespace for policies that prohibit pod creation+### Solution 2: Exempt the namespace for policies that prohibit pod creation -We recommend that you exempt the `aks-command` namespace for the associated Azure policies that don't allow the pod creation. For more information about exemption, see [Understand scope in Azure Policy](/en-us/azure/governance/policy/concepts/scope#:%7E:text=Exemption%20%2D%20A%20resource,Exemption%20definition.)+Exempt the `aks-command` namespace for the associated Azure policies that don't allow the pod creation. For more information about exemption, see [Understand scope in Azure Policy](/en-us/azure/governance/policy/concepts/scope#:%7E:text=Exemption%20%2D%20A%20resource,Exemption%20definition). -To exempt an Azure Policy:+To exempt an Azure Policy, follow these steps:  1. In the [Azure portal](https://portal.azure.com), search for and select **Policy**. 2. In the **Policy** navigation pane, locate the **Authoring** section, and then select **Assignments**.@@ -74,19 +87,19 @@ To exempt an Azure Policy: 6. Clear the **Only show parameters that need input or review** option. 7. In the **Namespace exclusions** box, add the *aks-command* namespace to the list of namespaces to be excluded. -Alternatively, if the policy isn't a built-in policy, you can check the configuration of the `command-<ID>` pod, and adjust the policy as necessary. To explore the pod's YAML configuration, run the following command:+Alternatively, if the policy isn't a built-in policy, check the configuration of the `command-<ID>` pod, and adjust the policy as necessary. To explore the pod's YAML configuration, run the following command.  ```bash kubectl get pods command-<ID> --namespace aks-command --output yaml ``` -You can exempt the `aks-command` namespace from restrictive policies by running the following command:+You can exempt the `aks-command` namespace from restrictive policies by running the following command.  ```bash az policy exemption create --name ExemptAksCommand --scope /subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.ContainerService/managedClusters/{aks-cluster} --policyAssignment /subscriptions/{subscription-id}/providers/Microsoft.Authorization/policyAssignments/{policy-assignment-id} ``` -## Cause 3: Required roles aren't granted+### Cause 3: Required roles aren't granted  To use the `az aks command invoke` command, you must have access to the following roles on the cluster: @@ -95,31 +108,40 @@ To use the `az aks command invoke` command, you must have access to the followin  If you don't have these roles, the `az aks command invoke` command can't retrieve the required information. -#### Solution 3: Add the required roles+### Solution 3: Add the required roles  To resolve this issue, follow these steps:  1. Add the `Microsoft.ContainerService/managedClusters/runCommand/action` and `Microsoft.ContainerService/managedClusters/commandResults/read` roles.-2. Assign the necessary roles to the user:+2. Assign the necessary roles to the user.      ```bash     az role assignment create --assignee {user-principal-name} --role "Azure Kubernetes Service Cluster User Role" --scope /subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.ContainerService/managedClusters/{aks-cluster}     ``` -## Cause 4: There's a Cloud Shell issue+    Ensure that the Azure RBAC role assigned to the user includes both of the following permissions:++>+> - `Microsoft.ContainerService/managedClusters/runCommand/action`+> - `Microsoft.ContainerService/managedClusters/commandResults/read`+>+>+> Review the [Azure built-in roles for Containers](/en-us/azure/role-based-access-control/built-in-roles/containers) documentation to determine whether an existing built-in role provides both permissions. If an appropriate built-in role isn't available, or if the available built-in role grants more permissions than required, create a custom Azure RBAC role containing both actions and assign it at the AKS cluster scope.++### Cause 4: There's a Cloud Shell issue -The `az aks command invoke` command isn't processed as expected when it's run directly in the [Azure Cloud Shell](/en-us/azure/cloud-shell/overview) environment. This is a known issue in Cloud Shell.+The `az aks command invoke` command isn't processed as expected when you run it directly in the [Azure Cloud Shell](/en-us/azure/cloud-shell/overview) environment. This behavior is a known issue in Cloud Shell. -#### Solution 4a: Run the az login command first+### Solution 4a: Run the az login command first -In Cloud Shell, run the [az login](/en-us/cli/azure/reference-index#az-login) command before you run the `az aks command invoke` command. For example:+In Cloud Shell, run the [az login](/en-us/cli/azure/reference-index#az-login) command before you run the `az aks command invoke` command. The following example shows how to do this.  ```bash az login az aks command invoke --resource-group {resource-group} --name {aks-cluster} --command "kubectl get pods" ``` -#### Solution 4b: Run the command on a local computer or a virtual machine+### Solution 4b: Run the command on a local computer or a virtual machine  Run the `az aks command invoke` command on a local computer or any virtual machine (VM) that has Azure CLI installed. 

@@ -9,23 +9,30 @@ AKS preview features are available on a self-service, opt-in basis. Previews are  Azure Kubernetes Application Network releases minor versions roughly once a quarter. Minor version releases include new features, improvements, and component updates. Each minor version has a specific Istio minor version built in, though consecutive Application Network versions might share the same Istio version. Patch releases include fixes for bugs and security vulnerabilities. -This article lists available Application Network versions along with their compatible Azure Kubernetes Service (AKS) versions.+This article lists Application Network versions, their support lifecycle, and their compatible Azure Kubernetes Service (AKS) versions.  ## Supported Application Network versions and compatible AKS versions -The following table outlines available Azure Kubernetes Application Network versions, their corresponding Istio versions, end-of-life dates, and compatible AKS versions:+The following table lists Azure Kubernetes Application Network versions, their corresponding Istio versions, end-of-life information, compatible AKS versions, and fully managed release channels. Recently retired versions remain in the table for reference and are no longer supported. Version availability and release-channel mappings vary by region and AKS version. Check available versions in your region before onboarding or planning an upgrade. -| Application Network version | Istio version | End of life | Compatible AKS versions |-| --- | --- | --- | --- |-| 1.1 | 1.26 | April 2026 | 1.30, 1.31, 1.32, 1.33, 1.34 |-| 1.2 | 1.27 | May 2026 | 1.30, 1.31, 1.32, 1.33, 1.34, 1.35 |-| 1.3 | 1.28 | ~Aug 2026 (expected) | 1.30, 1.31, 1.32, 1.33, 1.34, 1.35 |-| 1.4 | 1.29 | ~Sep 2026 (expected) | 1.31, 1.32, 1.33, 1.34, 1.35, 1.36 |-| 1.5 | 1.30 | ~Dec 2026 (expected) | 1.32, 1.33, 1.34, 1.35, 1.36 |+| Application Network version | Istio version | End of life | Compatible AKS versions | Fully managed release channel |+| --- | --- | --- | --- | --- |+| 1.1 (retired) | 1.26 | April 2026 | 1.30, 1.31, 1.32, 1.33, 1.34 | None |+| 1.2 (retired) | 1.27 | May 2026 | 1.30, 1.31, 1.32, 1.33, 1.34, 1.35 | None |+| 1.3 (retired) | 1.28 | Sept 2026 | 1.30, 1.31, 1.32, 1.33, 1.34, 1.35 | None |+| 1.4 | 1.29 | Not confirmed | 1.31, 1.32, 1.33, 1.34, 1.35, 1.36 | None |+| 1.5 | 1.30 | ~Dec 2026 (expected) | 1.32, 1.33, 1.34, 1.35, 1.36, 1.37 | `Stable` |+| 1.6 | 1.31 | Not confirmed | 1.32, 1.33, 1.34, 1.35, 1.36, 1.37 | `Rapid` |++End-of-life dates marked *expected* are estimates of *earliest possible* retirement, not confirmed dates.++`SelfManaged` mode offers two or three supported Application Network versions at a time. An individual member might have fewer compatible versions available, depending on its AKS version and region.++`FullyManaged` mode uses the `Rapid` and `Stable` release channels instead of selecting a version directly. The channels offer two supported versions when they target different versions, or one when they target the same version. A channel is available to a member only if its target version is compatible with the member's AKS version.  Important -Application Network doesn't manage upgrades of your AKS cluster. To remain in support, select a compatible version during onboarding and upgrade it regularly to maintain compatibility with Application Network. Refer to the [AKS Kubernetes release calendar](../aks/supported-kubernetes-versions#aks-kubernetes-release-calendar-and-upcoming-versions) for AKS version end-of-life dates.+Application Network doesn't manage upgrades of your AKS cluster. To remain in support, select a compatible version during onboarding and upgrade it regularly to maintain compatibility with Application Network. Compatibility with Application Network doesn't extend the AKS version's support lifecycle. Refer to the [AKS Kubernetes release calendar](../aks/supported-kubernetes-versions#aks-kubernetes-release-calendar-and-upcoming-versions) for AKS version end-of-life dates.  ## List available versions in your region 

@@ -2,20 +2,24 @@  ## Summary -This article explains how to troubleshoot DNS resolution problems in Azure Kubernetes Service (AKS) by building a structured workflow to diagnose, isolate, and fix Domain Name System (DNS) failures. Follow this guide to learn how to identify root causes and resolve DNS issues in your AKS cluster.+This article explains how to troubleshoot Domain Name System (DNS) resolution problems in Azure Kubernetes Service (AKS) by using a structured workflow. Use it to diagnose, isolate, and fix DNS failures so that your cluster workloads can resolve names reliably.  ## Prerequisites +Ensure you have the following prerequisites before you start troubleshooting DNS resolution problems in AKS:+ - The Kubernetes [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) command-line tool -    **Note:** To install kubectl by using [Azure CLI](/en-us/cli/azure/install-azure-cli), run the [az aks install-cli](/en-us/cli/azure/aks#az-aks-install-cli) command.+    Note++    To install kubectl by using [Azure CLI](/en-us/cli/azure/install-azure-cli), run the [az aks install-cli](/en-us/cli/azure/aks#az-aks-install-cli) command. - The [BIND 9](https://www.isc.org/bind/) command-line tool for DNS lookup - The [grep](https://linux.die.net/man/1/grep) command-line tool for text search - The [Wireshark](https://www.wireshark.org) network packet analyzer  ## Troubleshooting checklist -Troubleshooting DNS problems in AKS is typically a complex process. You can easily get lost in the many different steps without ever seeing a clear path forward. To help make the process simpler and more effective, use the "scientific" method to organize the work:+Troubleshooting DNS problems in AKS is typically a complex process. You can easily get lost in the many different steps without ever seeing a clear path forward. To help make the process simpler and more effective, use the following method to organize the work:  - Step 1. Collect the facts. - Step 2. Develop a hypothesis.@@ -440,7 +444,7 @@ In Wireshark, open the capture file of the CoreDNS pods that you merged earlier  The dialog box indicates that there were a combined total of about 4.5 million (4,540,233) packets, of which 2.68 percent caused server failure. The difference in query and response packet percentages shows that 5.94 percent of the queries (52.97 percent minus 47.03 percent) didn't receive a response. The maximum response time was 12 seconds (12,000.532227 milliseconds). -If you apply the display filter for query responses that took five seconds or more (`dns.time >= 5`), most of the packets in the filter results will indicate that a server failure occurred. This is probably because of a client "timed out" error.+If you apply the display filter for query responses that took five seconds or more (`dns.time >= 5`), most of the packets in the filter results will indicate that a server failure occurred. This condition probably caused a client "timed out" error.  The following table is a summary of the capture findings. @@ -525,7 +529,7 @@ To develop your first hypothesis, map each of the results from the required inpu  - If test results show DNS resolution failures at the CoreDNS service, or contain "timed out" errors when trying to reach specific endpoints, then configuration or connectivity problems might exist. - Indications of compute resource starvation at CoreDNS pod or node levels might suggest performance problems.-- DNS captures that have a considerable mismatch between DNS queries and DNS responses can indicate that packets are being lost. This scenario suggests that there are connectivity or performance problems.+- DNS captures that have a considerable mismatch between DNS queries and DNS responses can indicate that packets are lost. This scenario suggests that there are connectivity or performance problems. - The presence of custom configurations at the virtual network level or Kubernetes level can contain setups that don't work with AKS and CoreDNS as expected.  ### Troubleshooting step 3: Create and implement an action plan

@@ -9,7 +9,7 @@ AKS preview features are available on a self-service, opt-in basis. Previews are  The managed KubeRay extension for Azure Kubernetes Service (AKS) is in preview. It installs and manages the [KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/getting-started.html) operator and the Ray custom resource definitions (CRDs) on your cluster: `RayCluster`, `RayJob`, `RayCronJob`, and `RayService`. -This article shows you how to install, verify, update, and remove the extension by using the Azure CLI. You manage your Ray workloads and their runtime images separately.+This article shows you how to install, verify, update, and remove the extension by using the Azure CLI. You also deploy a sample Ray workload to confirm that the operator manages Ray resources as expected.  ## Prerequisites @@ -178,6 +178,143 @@ kubectl get crds \  If provisioning or readiness checks fail, see [Troubleshoot the managed KubeRay extension](/en-us/troubleshoot/azure/azure-kubernetes/extensions/troubleshoot-kuberay-extension-errors). +## Run a sample Ray workload++Deploy a CPU-only `RayCluster` and submit a `RayJob` to confirm that the KubeRay operator can reconcile Ray resources and run distributed tasks. The sample uses one head pod and one worker pod.++Set the sample resource names and Ray image:++```bash+RAY_NAMESPACE="kuberay-sample"+RAY_CLUSTER_NAME="sample-raycluster"+RAY_JOB_NAME="sample-rayjob"+RAY_VERSION="2.58.0"+RAY_IMAGE="mcr.microsoft.com/aks/ai-runtime/ray@sha256:e9a329eef15e5a13d50c3186c2b8036cd3728c6e218da860de23e8494b99ddd3"+```++Create a namespace and deploy the `RayCluster`:++```bash+kubectl create namespace "$RAY_NAMESPACE"++kubectl apply -f - <<YAML+apiVersion: ray.io/v1+kind: RayCluster+metadata:+  name: ${RAY_CLUSTER_NAME}+  namespace: ${RAY_NAMESPACE}+spec:+  rayVersion: "${RAY_VERSION}"+  headGroupSpec:+    rayStartParams:+      num-cpus: "0"+      dashboard-host: "0.0.0.0"+    template:+      spec:+        containers:+        - name: ray-head+          image: ${RAY_IMAGE}+          resources:+            requests:+              cpu: 500m+              memory: 2Gi+            limits:+              cpu: "1"+              memory: 4Gi+  workerGroupSpecs:+  - groupName: cpu-workers+    replicas: 1+    minReplicas: 1+    maxReplicas: 1+    rayStartParams:+      num-cpus: "1"+    template:+      spec:+        containers:+        - name: ray-worker+          image: ${RAY_IMAGE}+          resources:+            requests:+              cpu: 500m+              memory: 1Gi+            limits:+              cpu: "1"+              memory: 2Gi+YAML+```++Wait for the `RayCluster` to report `ready`, and then confirm that the head and worker pods are running:++```bash+kubectl wait \+  --for=jsonpath='{.status.state}'=ready \+  raycluster/"$RAY_CLUSTER_NAME" \+  --namespace "$RAY_NAMESPACE" \+  --timeout=10m++kubectl get raycluster,pods --namespace "$RAY_NAMESPACE"+```++Submit an HTTP-mode `RayJob` to the existing cluster. The job runs four remote tasks on the worker and prints a success marker after validating their results:++```bash+kubectl apply -f - <<YAML+apiVersion: ray.io/v1+kind: RayJob+metadata:+  name: ${RAY_JOB_NAME}+  namespace: ${RAY_NAMESPACE}+spec:+  jobId: ${RAY_JOB_NAME}+  submissionMode: HTTPMode+  clusterSelector:+    ray.io/cluster: ${RAY_CLUSTER_NAME}+  entrypoint: >-+    python3 -c "import ray; ray.init(address='auto'); square=ray.remote(lambda x: x*x);+    values=ray.get([square.remote(i) for i in range(4)]);+    assert values == [0, 1, 4, 9], values;+    print('KUBERAY_SAMPLE_OK', values, flush=True)"+  shutdownAfterJobFinishes: false+YAML+```++Wait for the job to succeed:++```bash+kubectl wait \+  --for=jsonpath='{.status.jobStatus}'=SUCCEEDED \+  rayjob/"$RAY_JOB_NAME" \+  --namespace "$RAY_NAMESPACE" \+  --timeout=10m+```++Read the Ray driver log from the head pod:++```bash+HEAD_POD="$(kubectl get pods \+  --namespace "$RAY_NAMESPACE" \+  --selector="ray.io/cluster=$RAY_CLUSTER_NAME,ray.io/node-type=head" \+  --output jsonpath='{.items[0].metadata.name}')"++kubectl exec \+  --namespace "$RAY_NAMESPACE" \+  "$HEAD_POD" \+  --container ray-head \+  -- sh -c 'grep -h "KUBERAY_SAMPLE_OK" /tmp/ray/session_latest/logs/job-driver-*.log'+```++The expected output is:++```output+KUBERAY_SAMPLE_OK [0, 1, 4, 9]+```++This output confirms that the operator created the Ray cluster, submitted the job, and ran tasks on the worker. Delete the sample resources when you're finished:++```bash+kubectl delete namespace "$RAY_NAMESPACE"+```+ ## Update the managed KubeRay extension  Use the `az k8s-extension extension-types list-versions-by-cluster` command in Install the managed KubeRay extension to check which versions are available for your cluster.

@@ -38,7 +38,7 @@ az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_NAME  helm install alb-controller-servicemesh-extension oci://mcr.microsoft.com/application-lb/charts/alb-controller-servicemesh-extension \      --namespace $HELM_NAMESPACE \-     --version 1.12.4+     --version 1.13.0 ```  ### Verify the ALB Controller installation

@@ -112,7 +112,7 @@ To understand take over behavior, see [Taking over existing workloads with Azure Save the following content to the file `deploy-kuard-fleet.yaml`.  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata:     name: deploy-kuard-fleet@@ -151,7 +151,7 @@ When the placement finishes, Fleet Manager is now responsible for the workload o Modify the `deploy-kuard-fleet.yaml` file by adding `aks-member-2` to the `clusterNames` array.  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ClusterResourcePlacement metadata:     name: deploy-kuard-fleet

@@ -1,18 +1,27 @@ # Application Gateway for Containers - Configure the inference gateway | Microsoft Learn -This article shows how to expose a self-hosted vLLM model server through Application Gateway for Containers inference gateway by using the Kubernetes [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/). The configuration uses Gateway API resources, an `InferencePool`, and the Endpoint Picker extension (EPP) to route inference requests to model server pods.+This article shows how to expose a self-hosted vLLM model server through Application Gateway for Containers inference gateway by using the Kubernetes [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/). The configuration uses Gateway API resources, an `InferencePool`, and the [llm-d router](https://github.com/llm-d/llm-d-router) as the endpoint picker (EPP) that routes each inference request to a model server pod.  In this guide, you deploy:  - A GPU-backed vLLM model server that serves an OpenAI-compatible API-- Gateway API Inference Extension resources - A [Gateway](https://gateway-api.sigs.k8s.io/reference/api-types/gateway/)-- An EPP deployment, service, and `InferencePool`+- The llm-d router CRDs+- An `InferencePool` and an llm-d router EPP deployment - An `HTTPRoute` that sends `/v1` traffic to the `InferencePool`+- Optionally, an `InferenceModelRewrite` and `InferenceObjective` resources  ## How the components fit together -Application Gateway for Containers receives requests at the `Gateway` listener and matches them against an `HTTPRoute`. When the route's `backendRef` is an `InferencePool`, ALB Controller programs the data path to consult the Endpoint Picker (EPP) for each request. The EPP watches the pods that match the `InferencePool` selector, reads vLLM metrics (queue depth, KV cache utilization, served models), and tells the data path which specific pod to send the request to.+Application Gateway for Containers receives requests at the `Gateway` listener and matches them against an `HTTPRoute`. When the route's `backendRef` is an `InferencePool`, ALB Controller programs the data path to consult the pool's endpoint picker for each request. The EPP watches the pods that match the `InferencePool` selector, reads vLLM metrics such as queue depth and KV cache utilization, and tells the data path which pod to send the request to.++Starting with v1.6, the Gateway API Inference Extension defines the `InferencePool` API and the endpoint picker protocol, while the full-featured endpoint picker and the `InferenceObjective` and `InferenceModelRewrite` APIs are maintained by the llm-d project. This guide uses the llm-d router to provide provide load-aware, cache-aware, and priority-aware scheduling. For more information, see [Application Gateway for Containers - Inference gateway](inference-gateway#endpoint-picker-implementations).++| Resource | API group and version | Installed by |+| --- | --- | --- |+| `InferencePool` | `inference.networking.k8s.io/v1` | ALB Controller (CRD) and the llm-d router Helm chart (resource) |+| `InferenceModelRewrite` | `llm-d.ai/v1alpha2` | llm-d router CRDs |+| `InferenceObjective` | `llm-d.ai/v1` | llm-d router CRDs |  Important @@ -23,18 +32,18 @@ The Application Gateway for Containers inference gateway is currently in preview Before you begin, complete the following tasks:  1. Deploy Application Gateway for Containers and ALB Controller. For more information, see [Quickstart: Deploy Application Gateway for Containers ALB Controller](quickstart-deploy-application-gateway-for-containers-alb-controller).-2. Enable the inference gateway feature on ALB Controller. For new installations or upgrades of ALB Controller, include `--set albController.aiGateway=true` in the Helm command. When you enable this setting, ALB Controller also installs v1.3.1 of the Gateway API Inference Extension CRDs.+2. Enable the inference gateway feature on ALB Controller. For new installations or upgrades of ALB Controller, include `--set albController.aiGateway=true` in the Helm command. When you enable this setting, ALB Controller also installs the Gateway API Inference Extension CRDs. ALB Controller 1.13.0 and later installs v1.6.2 of the CRDs. -Note+    Note -Inference gateway isn't currently supported through the Application Gateway for Containers' AKS add-on. You must deploy the Application Gateway for Containers' ALB Controller by Helm chart to configure this feature.+    Inference gateway isn't currently supported through the Application Gateway for Containers' AKS add-on. You must deploy the Application Gateway for Containers' ALB Controller by Helm chart to configure this feature.+3. Prepare an AKS cluster with a GPU node pool that has at least one schedulable GPU and the NVIDIA device plugin installed. For instructions, see [Use GPUs for compute-intensive workloads on AKS](/en-us/azure/aks/gpu-cluster). The endpoint picker runs on regular CPU nodes and doesn't need a GPU.+4. Install the following tools on your workstation or use Azure Cloud Shell where available: -1. Prepare an AKS cluster with a GPU node pool that has at least one schedulable GPU and the NVIDIA device plugin installed. For instructions, see [Use GPUs for compute-intensive workloads on AKS](/en-us/azure/aks/gpu-cluster).-2. Install the following tools on your workstation or use Azure Cloud Shell where available:     - Azure CLI     - `kubectl`-    - `helm`-3. [Create or obtain a Hugging Face access token](https://huggingface.co/docs/hub/security-tokens) that can download the model used by your vLLM deployment.+    - `helm` version 3.8 or later, for OCI chart support+5. [Create or obtain a Hugging Face access token](https://huggingface.co/docs/hub/security-tokens) that can download the model used by your vLLM deployment.  ## Set environment variables @@ -46,8 +55,11 @@ INFERENCE_POOL_NAME='vllm-qwen2-5-0-5b' MODEL_NAME='Qwen/Qwen2.5-0.5B-Instruct' GATEWAY_NAME='ai-gateway' HF_TOKEN='<your Hugging Face access token>'+LLMD_ROUTER_VERSION='v1.0.0' ``` +`LLMD_ROUTER_VERSION` sets the llm-d router release used for the CRDs, the Helm chart, and the EPP image. llm-d router v1.0.0 is built against Gateway API Inference Extension v1.6.2.+ # [ALB managed deployment](#tab/alb-managed) If you're using the ALB managed deployment strategy, set the namespace and name of your `ApplicationLoadBalancer` resource. @@ -81,19 +93,47 @@ Create a namespace for the model server, Gateway, HTTPRoute, InferencePool, and kubectl create namespace $NAMESPACE --dry-run=client -o yaml | kubectl apply -f - ``` -## Verify Gateway API Inference Extension CRDs+## Verify the Gateway API Inference Extension CRD++Verify that ALB Controller installed the `InferencePool` CRD and that it serves the `v1` API.++```bash+kubectl get crd inferencepools.inference.networking.k8s.io \+  -o jsonpath='{.metadata.annotations.inference\.networking\.k8s\.io/bundle-version}{"  served: "}{range .spec.versions[?(@.served==true)]}{.name}{" "}{end}{"\n"}'+```++Expected output:++```output+v1.6.2  served: v1+```++Note++If you upgraded from an earlier ALB Controller version, the cluster might still have the `inferenceobjectives.inference.networking.x-k8s.io` CRD. Gateway API Inference Extension v1.6 removed that API, and the llm-d router doesn't use it. This guide uses the `InferenceObjective` API from the llm-d router (`llm-d.ai/v1`) instead.++## Install the llm-d router CRDs -Verify that ALB Controller installed the Gateway API Inference Extension CRDs.+Install the `InferenceObjective` and `InferenceModelRewrite` CRDs from the llm-d router release. The EPP checks for these CRDs when it starts, so install them before you deploy the EPP, even if you don't plan to create these resources.  ```bash-kubectl get crds | grep inference.networking+for crd in inferenceobjectives inferencemodelrewrites; do+  kubectl apply --server-side \+    -f "https://raw.githubusercontent.com/llm-d/llm-d-router/${LLMD_ROUTER_VERSION}/config/crd/bases/llm-d.ai_${crd}.yaml"+done++kubectl wait --for=condition=Established \+  crd/inferenceobjectives.llm-d.ai crd/inferencemodelrewrites.llm-d.ai \+  --timeout=60s ``` -Expected output (CRD names; creation timestamps differ):+Expected output:  ```output-inferenceobjectives.inference.networking.x-k8s.io   2026-05-06T22:07:44Z-inferencepools.inference.networking.k8s.io          2026-05-06T22:07:45Z+customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai serverside-applied+customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai serverside-applied+customresourcedefinition.apiextensions.k8s.io/inferenceobjectives.llm-d.ai condition met+customresourcedefinition.apiextensions.k8s.io/inferencemodelrewrites.llm-d.ai condition met ```  ## Deploy a vLLM model server@@ -109,7 +149,7 @@ kubectl create secret generic hf-token \  Deploy a vLLM model server. The example uses `Qwen/Qwen2.5-0.5B-Instruct`, but the Application Gateway for Containers and Gateway API Inference Extension configuration is model-agnostic. You can substitute any vLLM-compatible model that fits on your GPU SKU. -The vLLM pod requires one `nvidia.com/gpu` and tolerates a `sku=gpu:NoSchedule` taint, so it lands on a GPU node. Remove the toleration if your node pool isn't tainted, or apply the taint with `kubectl taint nodes -l agentpool=<gpu-pool> sku=gpu:NoSchedule`. The pod labels (`app: ${INFERENCE_POOL_NAME}`) are what the `InferencePool` selects on later, so don't change them without also updating the chart's `inferencePool.modelServers.matchLabels.app` value.+The vLLM pod requires one `nvidia.com/gpu` and tolerates a `sku=gpu:NoSchedule` taint, so it lands on a GPU node. Remove the toleration if your node pool isn't tainted, or apply the taint with `kubectl taint nodes -l agentpool=<gpu-pool> sku=gpu:NoSchedule`. The pod labels (`app: ${INFERENCE_POOL_NAME}`) are what the `InferencePool` selects on later, so don't change them without also updating the chart's `router.modelServers.matchLabels.app` value.  ```bash kubectl apply -n $NAMESPACE -f - <<EOF@@ -144,6 +184,7 @@ spec:         - "2048"         - "--gpu-memory-utilization"         - "0.8"+        - "--enable-prefix-caching"         env:         - name: HUGGING_FACE_HUB_TOKEN           valueFrom:@@ -269,15 +310,50 @@ You see the following message when it's ready:  `gateway.gateway.networking.k8s.io/ai-gateway condition met` -## Deploy the InferencePool and Endpoint Picker+## Deploy the InferencePool and the llm-d router++Deploy the EPP and `InferencePool` together by using the `llm-d-router-gateway` Helm chart published by the llm-d project at `oci://ghcr.io/llm-d/charts/llm-d-router-gateway`. The chart creates an EPP `Deployment` and `Service` (gRPC on port `9002`), the `ServiceAccount`, `Role`, and `RoleBinding` that let the EPP watch pods and llm-d resources, and the `InferencePool` resource itself, named after the Helm release. -Deploy the EPP and `InferencePool` together by using the [`inferencepool` Helm chart](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/release-1.5/config/charts/inferencepool) published by the Kubernetes Gateway API Inference Extension project. The chart creates an EPP `Deployment`, a `Service` on port `9002`, the `ServiceAccount`, `Role`, and `RoleBinding` that let the EPP watch pods, and the `InferencePool` resource itself (named after the Helm release).+Create a values file for the chart.  ```bash-IGW_CHART_VERSION='v1.3.1'-EPP_IMAGE_TAG='v1.3.1'+cat > llm-d-router-values.yaml <<EOF+router:+  modelServers:+    type: vllm+    protocol: http+    targetPorts:+    - number: 8000+    matchLabels:+      app: ${INFERENCE_POOL_NAME}+  inferencePool:+    create: true+    failureMode: FailClose+  epp:+    replicas: 1+    image:+      tag: ${LLMD_ROUTER_VERSION}+    # The chart's default requests are sized for production traffic.+    # Lower them for a test cluster, or remove this block to keep the defaults.+    resources:+      requests:+        cpu: 500m+        memory: 512Mi+      limits:+        memory: 2Gi+provider:+  name: none+httpRoute:+  create: false+EOF ``` +The values file sets these options:++- `router.modelServers` tells the `InferencePool` which pods to select, which port they serve on, and that they run vLLM, so the EPP reads the vLLM metric names.+- `router.inferencePool.failureMode` sets what Application Gateway for Containers does when the EPP is unavailable. `FailClose` rejects requests. `FailOpen` keeps routing to the pool with standard load balancing, without the EPP's scheduling.+- `provider.name: none` and `httpRoute.create: false` keep the chart from creating gateway-specific resources. You create the `HTTPRoute` in a later step.+ Install the chart.  Note@@ -286,17 +362,15 @@ The chart appends `-epp` to the Helm release name when naming the EPP `Deploymen  ```bash helm upgrade --install "${INFERENCE_POOL_NAME}" \+  oci://ghcr.io/llm-d/charts/llm-d-router-gateway \+  --version "${LLMD_ROUTER_VERSION}" \   --namespace "${NAMESPACE}" \-  --set "inferencePool.modelServers.matchLabels.app=${INFERENCE_POOL_NAME}" \-  --set "inferenceExtension.image.tag=${EPP_IMAGE_TAG}" \-  --set "provider.name=none" \-  --version "${IGW_CHART_VERSION}" \-  oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool+  -f llm-d-router-values.yaml ```  Warning -The `inferencepool` Helm chart maintained by the Kubernetes Gateway API Inference Extension project renders an `InferencePool` resource named after the Helm release. If an `InferencePool` with the same name already exists in the namespace and was created outside Helm, `helm upgrade --install` refuses to adopt it and the command fails with an `invalid ownership metadata` error. If the existing pool is owned by a different Helm release, the install overwrites that `InferencePool`'s spec, including resetting the EPP image, selector, and `endpointPickerRef` to the v1.3.1 chart defaults. Before running the command, either delete the existing pool (`kubectl delete inferencepool <name> -n $NAMESPACE`) or pick a different `INFERENCE_POOL_NAME` for this walkthrough.+The chart renders an `InferencePool` resource named after the Helm release. If an `InferencePool` with the same name already exists in the namespace and was created outside Helm, `helm upgrade --install` refuses to adopt it and the command fails with an `invalid ownership metadata` error. If the existing pool is owned by a different Helm release, such as one installed with the earlier Gateway API Inference Extension `inferencepool` chart, uninstall that release first. Otherwise, either delete the existing pool (`kubectl delete inferencepool <name> -n $NAMESPACE`) or pick a different `INFERENCE_POOL_NAME` for this walkthrough.  Wait for the EPP deployment to become available. @@ -316,7 +390,7 @@ The Helm chart created an `InferencePool` resource named `${INFERENCE_POOL_NAME} kubectl get inferencepool ${INFERENCE_POOL_NAME} -n $NAMESPACE -o yaml ``` -Look for `Accepted=True` and `ResolvedRefs=True` in the `InferencePool` status.+In the `spec`, confirm that `endpointPickerRef.name` is `${INFERENCE_POOL_NAME}-epp` and `endpointPickerRef.failureMode` matches the value you set. In the `status`, look for `Accepted=True` and `ResolvedRefs=True`.  ## Create the HTTPRoute @@ -344,6 +418,19 @@ spec: EOF ``` +To route by model name instead of by path alone, add a header match on `X-Gateway-Model-Name`. Application Gateway for Containers' managed body-based routing sets that header from the `model` field of the request body, so you don't deploy a separate body-based routing component.++```yaml+  - matches:+    - path:+        type: PathPrefix+        value: /v1+      headers:+      - type: Exact+        name: X-Gateway-Model-Name+        value: Qwen/Qwen2.5-0.5B-Instruct+```+ ## Verify the route status  Confirm that the HTTPRoute is accepted and references are resolved.@@ -358,22 +445,22 @@ Look for conditions similar to the following output.   status:     parents:     - conditions:-      - lastTransitionTime: "2026-05-07T03:54:14Z"+      - lastTransitionTime: "2026-11-02T03:54:14Z"         message: ""-        observedGeneration: 3+        observedGeneration: 1         reason: ResolvedRefs         status: "True"         type: ResolvedRefs-      - lastTransitionTime: "2026-05-07T03:54:14Z"+      - lastTransitionTime: "2026-11-02T03:54:14Z"         message: Route is Accepted-        observedGeneration: 3+        observedGeneration: 1         reason: Accepted         status: "True"         type: Accepted-      - lastTransitionTime: "2026-05-07T03:54:14Z"+      - lastTransitionTime: "2026-11-02T03:54:14Z"         message: Application Gateway for Containers resource has been successfully           updated.-        observedGeneration: 3+        observedGeneration: 1         reason: Programmed         status: "True"         type: Programmed@@ -437,35 +524,167 @@ curl -i http://${GATEWAY_ADDRESS}/v1/completions \   }" ``` -## Optional: create an InferenceObjective+## Optional: create an InferenceModelRewrite -Create an `InferenceObjective` when you want clients to signal serving priority to the EPP. The EPP consumes the objective; the `HTTPRoute` doesn't reference it directly. Higher `priority` values are served before lower ones when the EPP must shed load.+Create an `InferenceModelRewrite` when you want clients to request a stable model name while the EPP decides which model the pool actually serves. The EPP rewrites the `model` field before it sends the request to the model server, and returns the client's requested name in the response. Use this pattern to roll out a new model version or LoRA adapter on the same pool without changing clients. -Warning--The `InferenceObjective` resource is currently in the `inference.networking.x-k8s.io/v1alpha2` API group, while `InferencePool` has graduated to `v1`; both groups coexist while the `InferenceObjective` API stabilizes. Expect API changes to the resource.+The following example lets clients request `chat-default`, which the EPP maps to `${MODEL_NAME}`.  ```bash kubectl apply -n $NAMESPACE -f - <<EOF-apiVersion: inference.networking.x-k8s.io/v1alpha2-kind: InferenceObjective+apiVersion: llm-d.ai/v1alpha2+kind: InferenceModelRewrite metadata:-  name: critical-inference+  name: chat-default spec:   poolRef:     group: inference.networking.k8s.io     kind: InferencePool     name: ${INFERENCE_POOL_NAME}+  rules:+  - matches:+    - model:+        type: Exact+        value: chat-default+    targets:+    - modelRewrite: ${MODEL_NAME}+EOF+```++Send a request for `chat-default`.++```bash+curl -s http://${GATEWAY_ADDRESS}/v1/chat/completions \+  -H 'Content-Type: application/json' \+  -d '{+    "model": "chat-default",+    "messages": [{"role": "user", "content": "What is 2+2? Answer in one word."}],+    "max_tokens": 10+  }'+```++The response comes from the vLLM server, and its `model` field is `chat-default`.++Note++Application Gateway for Containers matches the `HTTPRoute` before the EPP rewrites the model name, so `X-Gateway-Model-Name` carries the name that the client requested. If your route matches on `X-Gateway-Model-Name`, add a match for each client-facing name, such as `chat-default`, that sends requests to this `InferencePool`. A route that matches on path alone, like the one in this article, needs no change.++To split traffic across models that the pool serves, such as two LoRA adapters, list more than one target with weights. For example, the following targets send 90 percent of `chat-default` requests to `adapter-v1` and 10 percent to `adapter-v2`:++```yaml+    targets:+    - modelRewrite: adapter-v1+      weight: 90+    - modelRewrite: adapter-v2+      weight: 10+```++Note++If the pool uses `FailOpen` and the EPP is unavailable, requests bypass the EPP and reach the model server with the original model name, such as `chat-default`, which the model server doesn't recognize. Use `FailClose` for pools that rely on model rewrites.++## Optional: prioritize traffic with InferenceObjective++Create `InferenceObjective` resources when different classes of traffic share a pool and you want latency-critical requests served first when the pool is saturated. Clients select an objective with the `x-llm-d-inference-objective` request header. The `HTTPRoute` doesn't reference objectives.++Priority takes effect only when flow control is enabled in the llm-d router. With flow control, the EPP holds requests in priority bands while the pool is saturated, dispatches higher-priority requests first, and sheds lower-priority requests when their band is full or their time-to-live expires. Flow control is a feature gate and is off by default.++### Enable flow control++Create a second values file that replaces the EPP's scheduling configuration with one that enables flow control.++```bash+cat > llm-d-router-flow-control.yaml <<'EOF'+router:+  epp:+    pluginsConfigFile: flow-control.yaml+    pluginsCustomConfig:+      flow-control.yaml: |+        apiVersion: llm-d.ai/v1+        kind: EndpointPickerConfig+        featureGates:+        - flowControl+        plugins:+        - type: prefix-cache-scorer+        - type: queue-scorer+        - type: kv-cache-utilization-scorer+        - type: active-request-scorer+        # Treats a pod as saturated when it has maxConcurrency requests in flight.+        # Set it to the number of concurrent requests you want each replica to run.+        - type: concurrency-detector+          parameters:+            concurrencyMode: requests+            maxConcurrency: 32+        schedulingProfiles:+        - name: default+          plugins:+          - pluginRef: prefix-cache-scorer+            weight: 3+          - pluginRef: active-request-scorer+            weight: 1+          - pluginRef: queue-scorer+            weight: 1+          - pluginRef: kv-cache-utilization-scorer+            weight: 1+        flowControl:+          saturationDetector:+            pluginRef: concurrency-detector+          defaultRequestTTL: 60s+          priorityBands:+          - priority: 10       # critical-inference+            maxRequests: 200+          - priority: 0        # requests without an objective+            maxRequests: 200+          - priority: -10      # batch-inference+            maxRequests: 50+EOF+```++Upgrade the release with both values files.++```bash+helm upgrade "${INFERENCE_POOL_NAME}" \+  oci://ghcr.io/llm-d/charts/llm-d-router-gateway \+  --version "${LLMD_ROUTER_VERSION}" \+  --namespace "${NAMESPACE}" \+  -f llm-d-router-values.yaml \+  -f llm-d-router-flow-control.yaml++kubectl rollout status deployment/${INFERENCE_POOL_NAME}-epp -n $NAMESPACE --timeout=180s+```++### Create the objectives++Create a high-priority objective for interactive traffic and a low-priority objective for batch traffic.++```bash+kubectl apply -n $NAMESPACE -f - <<EOF+apiVersion: llm-d.ai/v1+kind: InferenceObjective+metadata:+  name: critical-inference+spec:   priority: 10+  poolRefs:+  - name: ${INFERENCE_POOL_NAME}+---+apiVersion: llm-d.ai/v1+kind: InferenceObjective+metadata:+  name: batch-inference+spec:+  priority: -10+  poolRefs:+  - name: ${INFERENCE_POOL_NAME} EOF ``` -Send a request that uses the objective.+Send a request that uses the high-priority objective.  ```bash curl -i http://${GATEWAY_ADDRESS}/v1/chat/completions \   -H 'Content-Type: application/json' \-  -H 'x-gateway-inference-objective: critical-inference' \+  -H 'x-llm-d-inference-objective: critical-inference' \   -d "{     \"model\": \"${MODEL_NAME}\",     \"messages\": [{\"role\": \"user\", \"content\": \"Summarize what an inference gateway does.\"}],@@ -473,6 +692,8 @@ curl -i http://${GATEWAY_ADDRESS}/v1/chat/completions \   }" ``` +Priority has a visible effect only when the pool is saturated. To observe it, send a sustained burst of requests with `x-llm-d-inference-objective: batch-inference` that exceeds `maxConcurrency` across your replicas, then send requests with `critical-inference` and compare their latency with the batch requests.+ ## Observe endpoint picker behavior  Review EPP logs to confirm that requests flow through endpoint selection.@@ -481,6 +702,8 @@ Review EPP logs to confirm that requests flow through endpoint selection. kubectl logs deployment/${INFERENCE_POOL_NAME}-epp -n $NAMESPACE --tail=100 ``` +To see per-request scheduling decisions, raise the EPP log verbosity by adding `flags: {v: 2}` under `router.epp` in your values file, and then upgrade the release.+ Review vLLM metrics from the model server pod.  ```bash@@ -490,6 +713,8 @@ kubectl exec -n $NAMESPACE $POD -- curl -s http://localhost:8000/metrics \   | grep -E "^vllm:(num_requests_waiting|num_requests_running|kv_cache_usage_perc)" ``` +The EPP also exposes Prometheus metrics for its scheduling, flow control, and model rewrite decisions. By default, the metrics endpoint requires a bearer token. Use the chart's `router.monitoring` values to configure scraping. For details, see the [llm-d router documentation](https://github.com/llm-d/llm-d-router).+ ## Troubleshoot  If traffic doesn't reach the model server, use the following checks:@@ -509,9 +734,13 @@ If traffic doesn't reach the model server, use the following checks: | --- | --- | --- | | vLLM pod stuck in `Pending` | No allocatable GPU, taints don't match the pod's tolerations, or the `nvidia.com/gpu` resource isn't advertised by the device plugin. | Use `kubectl describe pod` to see the scheduler message. Verify the GPU node pool exists, the NVIDIA device plugin DaemonSet is `Ready`, and the toleration matches your taint. | | vLLM pod in `CrashLoopBackOff` with `401 Unauthorized` in logs | Hugging Face token is invalid, expired, or doesn't have access to the model. | Recreate the `hf-token` Secret with a token that has read access to the model repo, then run `kubectl rollout restart deployment/${INFERENCE_POOL_NAME}`. |+| EPP pod stuck in `Pending` | The EPP's CPU or memory requests don't fit on any node. The chart's default requests are sized for production. | Lower `router.epp.resources.requests` in your values file and upgrade the release, or add capacity to a CPU node pool. |+| EPP pod in `CrashLoopBackOff`, with logs that mention `InferenceObjective` or `InferenceModelRewrite` | The llm-d router CRDs aren't installed, or they come from a different llm-d router version than the EPP image. | Install the CRDs from the same `LLMD_ROUTER_VERSION` as the chart, then restart the EPP deployment. |+| `helm upgrade --install` fails with `invalid ownership metadata` | An `InferencePool` with the release name already exists and isn't owned by this release. | Uninstall the release that owns it, delete the pool, or choose a different `INFERENCE_POOL_NAME`. | | `HTTPRoute` status shows `ResolvedRefs=False` with `reason: BackendNotFound` and `message: InferencePool <namespace>/<name> not found` | The `HTTPRoute`'s `backendRef` name doesn't match an existing `InferencePool`, **or** the `InferencePool` exists but has no ready endpoints (for example, the vLLM pod is `Pending` because no GPU node could be scheduled or scaled up). ALB Controller reports a pool with zero ready endpoints as `BackendNotFound`. | Run `kubectl get inferencepool -n $NAMESPACE` to confirm the name matches the route's `backendRef`. Then run `kubectl get pods -n $NAMESPACE -l app=${INFERENCE_POOL_NAME}` and `kubectl describe pod <pod> -n $NAMESPACE`. If the pod is `Pending` with `FailedScheduling` events like `Insufficient nvidia.com/gpu` or `untolerated taint(s)`, scale up the GPU node pool, fix the toleration, or pick a SKU with available GPU capacity. |-| `curl` returns `503` from the Gateway | EPP can't reach the pool (no ready endpoints) or the EPP service name in `endpointPickerRef` doesn't resolve. | Check `kubectl get endpointslices -n $NAMESPACE -l kubernetes.io/service-name=${INFERENCE_POOL_NAME}-epp` and confirm the `InferencePool.spec.endpointPickerRef.name` matches the chart-installed service name. |-| `curl` returns `404` | Path doesn't match the `HTTPRoute`. | Confirm the request path begins with `/v1` and that the route's `parentRefs` references the same Gateway in the same namespace. |+| `curl` returns `503` from the Gateway | The pool has no ready endpoints, or the pool uses `FailClose` and the EPP isn't reachable (for example, the EPP pod isn't ready, or the service name in `endpointPickerRef` doesn't resolve). | Check `kubectl get endpointslices -n $NAMESPACE -l kubernetes.io/service-name=${INFERENCE_POOL_NAME}-epp` and confirm the `InferencePool.spec.endpointPickerRef.name` matches the chart-installed service name. |+| `curl` returns `404` | Path doesn't match the `HTTPRoute`, or the route matches on `X-Gateway-Model-Name` and the request's `model` value doesn't match. | Confirm the request path begins with `/v1`, the `model` value matches the route's header match exactly, and the route's `parentRefs` references the same Gateway in the same namespace. |+| Requests with `x-llm-d-inference-objective` aren't prioritized | Flow control isn't enabled, the objective's `poolRefs` doesn't name this pool, or the pool isn't saturated. | Confirm the EPP configuration includes the `flowControl` feature gate, check the objective with `kubectl get inferenceobjective -n $NAMESPACE -o yaml`, and test under enough load to exceed `maxConcurrency`. | | Requests hang for &gt;60 seconds, then error | vLLM is still loading the model into GPU memory. | Wait for the readiness probe to succeed; tail vLLM logs until you see `Application startup complete`. |  ## Clean up resources@@ -519,11 +748,19 @@ If traffic doesn't reach the model server, use the following checks: Delete the resources you created in this article.  ```bash-kubectl delete inferenceobjective critical-inference -n $NAMESPACE --ignore-not-found+kubectl delete inferenceobjective critical-inference batch-inference -n $NAMESPACE --ignore-not-found+kubectl delete inferencemodelrewrite chat-default -n $NAMESPACE --ignore-not-found kubectl delete httproute ${INFERENCE_POOL_NAME} -n $NAMESPACE --ignore-not-found helm uninstall ${INFERENCE_POOL_NAME} -n $NAMESPACE 2>/dev/null || true kubectl delete gateway ${GATEWAY_NAME} -n $NAMESPACE --ignore-not-found kubectl delete deployment ${INFERENCE_POOL_NAME} -n $NAMESPACE --ignore-not-found kubectl delete secret hf-token -n $NAMESPACE --ignore-not-found kubectl delete namespace $NAMESPACE --ignore-not-found+rm -f llm-d-router-values.yaml llm-d-router-flow-control.yaml+```++If no other workloads on the cluster use them, you can also remove the llm-d router CRDs. Deleting a CRD deletes every resource of that type in the cluster.++```bash+kubectl delete crd inferenceobjectives.llm-d.ai inferencemodelrewrites.llm-d.ai --ignore-not-found ```

@@ -371,6 +371,8 @@ On the selected Kubernetes clusters, you find the namespace, Deployment, and Ser  When you delete the resource placement on the Fleet Manager hub cluster, you remove the distributed resources from selected member clusters. +To remove distributed resources without deleting the placement, use `PickFixed` with an empty `clusterNames` array. For more information, see [PickFixed placement type](concepts-resource-placement#pickfixed-placement-type).+ ::: zone pivot="azure-cli"  ```bash

@@ -1,11 +1,13 @@-# Troubleshoot DNS resolution failures from inside the pod - Azure | Microsoft Learn+# Troubleshoot DNS resolution failures from inside the pod in AKS - Azure | Microsoft Learn  ## Summary -This article discusses how to troubleshoot Domain Name System (DNS) resolution failures that occur from inside the pod, but not from the worker node, when you try to establish an outbound connection from a Microsoft Azure Kubernetes Service (AKS) cluster.+This article discusses how to troubleshoot Domain Name System (DNS) resolution failures that occur from inside the pod, but not from the worker node, when you try to establish an outbound connection from an Azure Kubernetes Service (AKS) cluster.  ## Prerequisites +Ensure the following prerequisites are met before you start troubleshooting DNS resolution failures from inside the pod:+ - The Kubernetes [kubectl](https://kubernetes.io/docs/reference/kubectl/) tool, or a similar tool to connect to the cluster. To install kubectl by using [Azure CLI](/en-us/cli/azure/install-azure-cli), run the [az aks install-cli](/en-us/cli/azure/aks#az-aks-install-cli) command. - A test container image that the cluster can pull and that includes DNS lookup tools such as [host](https://linux.die.net/man/1/host), `dig`, and `nslookup`. This article uses `mcr.microsoft.com/aks/fundamental/base-ubuntu`. If you instead use a minimal image that doesn't include these tools, you also need a package manager (such as [apt-get](https://linux.die.net/man/8/apt-get)) to install them. If you prefer a different image, use a [glibc](https://www.gnu.org/software/libc/)-based distribution (such as Ubuntu, Debian, or Azure Linux) rather than a musl-based one (such as Alpine or BusyBox). The resolver behavior described in this article (for example, the *resolv.conf* search-domain and `ndots` handling and the three name server limit) follows the glibc implementation, and a musl-based image can resolve DNS differently. - The [systemctl](https://man7.org/linux/man-pages/man1/systemctl.1.html) command-line tool.@@ -16,7 +18,7 @@ For DNS resolution, the pods send requests to the CoreDNS pods in the `kube-syst  If the DNS query is for an internal component, such as a service name, the CoreDNS pod responds by itself. However, if the request is for an external domain, the CoreDNS pod sends the request to the upstream DNS server. -The upstream DNS servers are obtained based on the *resolv.conf* file of the worker node in which the pod is running. The *resolv.conf* file (*/run/systemd/resolve/resolv.conf*) is updated based on the DNS settings of the virtual network on which the worker node is running.+The upstream DNS servers are based on the *resolv.conf* file of the worker node where the pod runs. The DNS settings of the virtual network on which the worker node runs update the *resolv.conf* file (*/run/systemd/resolve/resolv.conf*).  ## Troubleshooting checklist 

@@ -1,12 +1,12 @@-# TCP 10250 I/O timeout errors when connecting to a node's Kubelet for log retrieval - Azure | Microsoft Learn+# Troubleshoot TCP 10250 I/O timeout errors when connecting to a node's Kubelet for log retrieval in AKS - Azure | Microsoft Learn  ## Summary -A TCP 10250 i/o timeout in kubectl logs is often caused by blocked internal traffic between cluster nodes. Use this article to verify whether [network security groups](/en-us/azure/aks/concepts-security#azure-network-security-groups) (NSGs) are blocking required node subnet traffic and restore log retrieval.+A TCP 10250 i/o timeout error when you run `kubectl logs` often occurs when internal traffic between cluster nodes is blocked. This article helps you check whether [network security groups](/en-us/azure/aks/concepts-security#azure-network-security-groups) (NSGs) block required node subnet traffic so that you can restore log retrieval.  ## Connect to the AKS cluster -First, connect to your Azure Kubernetes Service (AKS) cluster by running the following command:+First, connect to your Azure Kubernetes Service (AKS) cluster by running the following command.  ```bash export RESOURCE_GROUP=<your-resource-group>@@ -17,13 +17,13 @@ az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME  ## Symptoms -Tunnel functionalities, such as `kubectl logs` and code execution, work only for pods that are hosted on nodes on which tunnel service pods are deployed. Pods on other nodes that have no tunnel service pods cannot reach to the tunnel. When viewing the logs of these pods, you receive the following error message:+Tunnel functionalities, such as `kubectl logs` and code execution, work only for pods that are hosted on nodes where tunnel service pods are deployed. Pods on other nodes that don't have tunnel service pods can't reach the tunnel. When you view the logs of these pods, you receive the following error message.  ```bash kubectl logs $POD_NAME ``` -Results:+The following error messages appear.  ```output Error from server: Get "https://aks-agentpool-xxxxxxxxx-vmssxxxxxxxxx:10250/containerLogs/vsm-mba-prod/mba-api-app-xxxxxxxxxx/technosvc": dial tcp <IP-Address>:10250: i/o timeout@@ -31,4 +31,4 @@ Error from server: Get "https://aks-agentpool-xxxxxxxxx-vmssxxxxxxxxx:10250/cont  ## Solution -To resolve this issue, allow traffic on port 10250 as described in this [article](tunnel-connectivity-issues).+To resolve this issue, allow traffic on port 10250 as described in [Troubleshoot tunnel connectivity issues in Azure Kubernetes Service](tunnel-connectivity-issues).

@@ -140,7 +140,7 @@ AzureDiagnostics AzureDiagnostics | where _ResourceId == YOUR-RESOURCE-ID | where Category == "fleet-hub-agent"-| where log_s contains "v1beta1/member_controller.go"+| where log_s contains "v1/member_controller.go" | order by TimeGenerated desc | take 1000 ```@@ -218,7 +218,7 @@ AzureDiagnostics AzureDiagnostics | where _ResourceId == YOUR-RESOURCE-ID | where Category == "fleet-member-agent"-| where log_s contains "v1beta1/member_controller.go"+| where log_s contains "v1/member_controller.go" | order by TimeGenerated desc | take 1000 ```

@@ -2,12 +2,24 @@  This article provides an overview of AKS desktop: an application-focused developer portal for Azure Kubernetes Service (AKS) that simplifies application deployment and management without requiring deep Kubernetes expertise. AKS desktop is built on supported AKS features, best practices, and open-source [Headlamp](https://headlamp.dev/). -To install AKS desktop, see the [AKS desktop GitHub repository](https://github.com/Azure/aks-desktop/releases).- Important  AKS desktop abstracts Kubernetes complexity but doesn't remove access. You can still use kubectl, YAML, or other existing tools alongside AKS desktop. +## Install AKS desktop++If you have [Azure CLI](/en-us/cli/azure/install-azure-cli) version 2.91.0 or later, you can install AKS desktop by using the following command:++```bash+az aks install-desktop+```++You can also download and install it from the [AKS desktop GitHub repository](https://github.com/Azure/aks-desktop/releases).++Note++AKS desktop supports the following operating systems (OS): Windows (x64), Linux (x64 and arm64), and macOS (x64 and arm64).+ ## What is AKS desktop?  Kubernetes workflows often require writing and maintaining YAML, switching between tools (kubectl, dashboards, monitoring), and an understanding of low-level infrastructure concepts.

@@ -560,3 +560,11 @@ az network nic show --ids "$nicId" \  - **Use dynamic IP allocation** for the private endpoint if you expect to add geo-replicas or enable regional endpoints later. With dynamic allocation, the required data endpoint and regional endpoint members receive private IP addresses automatically. This approach is recommended. - **Create the private endpoint after all geo-replicas exist** if static IP allocation is required. Add every geo-replica and enable regional endpoints, if needed, first. Then create the static IP private endpoint so its IP configuration includes members for each region's data endpoint and, when enabled, regional endpoint. After a static IP private endpoint is created this way, you can't add further geo-replicas or enable regional endpoints without reconfiguring the private endpoint to add the new members.++### Geo-replica deletion or regional endpoints disablement fails on registries with a locked private DNS zone++Deleting a geo-replica or disabling regional endpoints on a private endpoint-enabled registry removes the corresponding private DNS records. If a **Delete** or **Read-only**[resource lock](/en-us/azure/azure-resource-manager/management/lock-resources) applies to the private DNS zone, either directly or inherited from its resource group or subscription, the record deletion fails with a `ScopeLocked` error and the operation can't complete.++**Solution:**++- Temporarily remove the lock before you delete a geo-replica or disable regional endpoints. Restore the lock after the registry and private endpoint provisioning operations complete successfully.

@@ -2,7 +2,7 @@  Running NVIDIA GPU workloads on Azure Kubernetes Service (AKS) traditionally requires you to install and maintain the NVIDIA GPU driver, Kubernetes device plugin, and a GPU metrics exporter on each GPU node. These components enable GPU scheduling, container-level GPU access, and telemetry, but installing them manually or through the [NVIDIA GPU Operator](nvidia-gpu-operator) adds operational overhead. -By using fully managed GPU nodes (preview), AKS installs and maintains the NVIDIA GPU driver, device plugin, and Data Center GPU Manager [(DCGM) metrics exporter](https://github.com/NVIDIA/dcgm-exporter/tree/main) for you. GPU node pool creation becomes a single step, and GPU capacity behaves like any other AKS node pool. You can use managed GPU nodes with manually created GPU node pools and with node auto-provisioning (NAP), including AKS Automatic clusters.+By using fully managed GPU nodes, AKS installs and maintains the NVIDIA GPU driver, device plugin, and Data Center GPU Manager [(DCGM) metrics exporter](https://github.com/NVIDIA/dcgm-exporter/tree/main) for you. GPU node pool creation becomes a single step, and GPU capacity behaves like any other AKS node pool. You can use managed GPU nodes with manually created GPU node pools and with node auto-provisioning (NAP), including AKS Automatic clusters.  You configure a managed GPU node pool through two fields under `gpuProfile.nvidia`: @@ -11,18 +11,10 @@ You configure a managed GPU node pool through two fields under `gpuProfile.nvidi  In this article, you provision a managed GPU node pool, optionally enable MIG, verify the stack, and run a sample GPU workload. -Important--AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:--- [AKS support policies](/en-us/azure/aks/support-policies)-- [Azure support FAQ](/en-us/azure/aks/faq)- ## Before you begin  - This article assumes you have an existing AKS cluster. If you don't have a cluster, create one using the [Azure CLI](learn/quick-kubernetes-deploy-cli), [Azure PowerShell](learn/quick-kubernetes-deploy-powershell), or the [Azure portal](learn/quick-kubernetes-deploy-portal). - You need the Azure CLI version 2.85.0 or later installed. To find the version, run `az --version`. If you need to install or upgrade, see [Install Azure CLI](/en-us/cli/azure/install-azure-cli).-- You need to install and upgrade to latest version of the `aks-preview` extension. Version 19.0.0b29 or later is required. - Get the credentials for your AKS cluster with [`az aks get-credentials`](/en-us/cli/azure/aks#az_aks_get_credentials) before running the `kubectl` examples in this article.  ## Managed GPU components@@ -57,28 +49,6 @@ Together, they produce three install profiles: - **Override**: `managementMode: Managed` requires the driver, so `--gpu-driver None` is ignored when `--enable-managed-gpu=true` and the driver is still installed. To skip the driver, set both `--enable-managed-gpu=false` and `--gpu-driver None`. - **Immutability**: `managementMode`, `migStrategy`, and `driver` are all fixed at creation time. To change profile, create a new node pool. -### Install the `aks-preview` CLI extension--Install the aks-preview CLI extension by using the [`az extension add`](/en-us/cli/azure/extension#az-extension-add) command.--```azurecli-az extension add --name aks-preview-```--Update the extension to ensure you have the latest version installed by using the [`az extension update`](/en-us/cli/azure/extension#az-extension-update) command.--```azurecli-az extension update --name aks-preview-```--### Register the `ManagedGPUExperiencePreview` feature flag--Register the `ManagedGPUExperiencePreview` feature flag in your subscription using the [`az feature register`](/en-us/cli/azure/feature#az-feature-register) command.--```azurecli-az feature register --namespace Microsoft.ContainerService --name ManagedGPUExperiencePreview-```- ## Limitations  - This feature currently supports [NVIDIA GPU-enabled virtual machine (VM) sizes](/en-us/azure/virtual-machines/sizes-gpu) only.@@ -92,7 +62,7 @@ Note  GPU-enabled VMs contain specialized hardware subject to higher pricing and region availability. For more information, see the [pricing](https://azure.microsoft.com/pricing/) tool and [region availability](https://azure.microsoft.com/global-infrastructure/services/). -## Create an AKS-managed GPU node pool (preview)+## Create an AKS-managed GPU node pool  Add a managed GPU node pool to an existing AKS cluster by passing `--enable-managed-gpu=true` to [`az aks nodepool add`](/en-us/cli/azure/aks/nodepool#az_aks_nodepool_add). AKS sets `gpuProfile.nvidia.managementMode` to `Managed` and installs the GPU driver, device plugin, and DCGM metrics exporter automatically. @@ -178,9 +148,15 @@ To use Azure Linux, you specify the OS SKU by setting `--os-sku` to `AzureLinux`     ```  # [Node auto-provisioning node pool](#tab/nap-managed-node)-To use managed GPU nodes with node auto-provisioning (NAP), add the `EnableManagedGPUExperience: "true"` tag to your `AKSNodeClass` custom resource definition (CRD). Then create a `NodePool` that selects NVIDIA GPU-enabled VM sizes. This configuration applies to AKS Automatic clusters, where NAP is enabled by default, and to AKS Standard clusters with NAP enabled.+To use managed GPU nodes with node auto-provisioning (NAP), configure `spec.gpu` on an `AKSNodeClass` resource, then create a `NodePool` that references it and selects Linux NVIDIA GPU-enabled VM sizes. This configuration applies to AKS Automatic clusters, where NAP is enabled by default, and to AKS Standard clusters with NAP enabled. -To learn more about configurable Azure-specific settings, see [Configure AKSNodeClass resources for NAP](node-auto-provisioning-aksnodeclass).+Important++Managed GPU with NAP requires the AKS Machine API provision mode.++Set `spec.gpu.mode` to `Driver` and `spec.gpu.nvidia.managementMode` to `Managed` to have AKS install and maintain the GPU driver, device plugin, and DCGM metrics exporter. The default NVIDIA management mode is `Unmanaged`. Don't set `mode` to `None` for managed GPU nodes, because it skips driver installation.++For more information, see [GPU settings in AKSNodeClass](node-auto-provisioning-aksnodeclass#gpu-settings).  1. Create an `AKSNodeClass` CRD and set `mode: Driver` and `nvidia.managementMode: Managed` in the `spec.gpu` field. @@ -189,64 +165,61 @@ To learn more about configurable Azure-specific settings, see [Configure AKSNode     kind: AKSNodeClass     metadata:       name: managed-gpu-class+    spec:+      gpu:+        mode: Driver+        nvidia:+          managementMode: Managed     ```+2. Create a `NodePool` CRD that allows NAP to deploy NVIDIA GPU SKUs. -spec: gpu: mode: Driver nvidia: managementMode: Managed--```--1. Create a `NodePool` CRD that allows NAP to deploy NVIDIA GPU SKUs.--```yaml-apiVersion: karpenter.sh/v1-kind: NodePool-metadata:-  name: managed-gpu-np-spec:-  disruption:-    consolidationPolicy: WhenEmptyOrUnderutilized-    consolidateAfter: 30s-  template:+    ```yaml+    apiVersion: karpenter.sh/v1+    kind: NodePool+    metadata:+      name: managed-gpu-np     spec:-      nodeClassRef:-        group: karpenter.azure.com-        kind: AKSNodeClass-        name: managed-gpu-class-      requirements:-        - key: kubernetes.io/arch-          operator: In-          values: ["amd64"]-        - key: kubernetes.io/os-          operator: In-          values: ["linux"]-        - key: karpenter.azure.com/sku-family-          operator: In-          values: ["N"]-        - key: karpenter.azure.com/sku-gpu-manufacturer-          operator: In-          values: ["nvidia"]-        - key: karpenter.sh/capacity-type-          operator: In-          values: ["spot", "on-demand"]-```--1. Deploy the `AKSNodeClass` and `NodePool` CRDs.+      disruption:+        consolidationPolicy: WhenEmptyOrUnderutilized+        consolidateAfter: 30s+      template:+        spec:+          nodeClassRef:+            group: karpenter.azure.com+            kind: AKSNodeClass+            name: managed-gpu-class+          requirements:+            - key: kubernetes.io/arch+              operator: In+              values: ["amd64"]+            - key: kubernetes.io/os+              operator: In+              values: ["linux"]+            - key: karpenter.azure.com/sku-family+              operator: In+              values: ["N"]+            - key: karpenter.azure.com/sku-gpu-manufacturer+              operator: In+              values: ["nvidia"]+            - key: karpenter.sh/capacity-type+              operator: In+              values: ["spot", "on-demand"]+    ```+3. Deploy the `AKSNodeClass` and `NodePool` CRDs.      ```bash     kubectl apply -f managed-gpu-class.yaml     kubectl apply -f managed-gpu-np.yaml     ```-2. Deploy a sample workload that requests GPU resources. NAP provisions a matching GPU node when the pod is pending.-3. Confirm that the `AKSNodeClass` and `NodePool` resources are deployed and the managed GPU label is applied to the NAP-provisioned node.+4. Deploy a sample workload that requests GPU resources. NAP provisions a matching GPU node when the pod is pending.+5. Confirm that the `AKSNodeClass` and `NodePool` resources are deployed and the managed GPU label is applied to the NAP-provisioned node.      ```bash     kubectl get aksnodeclasses.karpenter.azure.com,nodepools.karpenter.sh     kubectl get nodes -L kubernetes.azure.com/dcgm-exporter     ``` ------## Create a managed Multi-Instance GPU (MIG) node pool (preview)+## Create a managed Multi-Instance GPU (MIG) node pool  For GPU SKUs that support Multi-Instance GPU (such as A100 and H100), configure a MIG strategy at node pool creation with the `--gpu-mig-strategy` flag. The strategy controls how MIG partitions are exposed to Kubernetes: @@ -290,9 +263,7 @@ az aks nodepool add \  With this configuration, pods request specific MIG partitions, for example `nvidia.com/mig-1g.10gb: 1`. ------## Verify the managed GPU node pool (preview)+## Verify the managed GPU node pool  After the node pool is ready, run the following checks to confirm that the full managed stack is installed and healthy. @@ -352,7 +323,7 @@ After the node pool is ready, run the following checks to confirm that the full     kubectl logs managed-gpu-test     ``` -## Scale a managed GPU node pool (preview)+## Scale a managed GPU node pool  Scale a managed GPU node pool manually with [`az aks nodepool scale`](/en-us/cli/azure/aks/nodepool#az_aks_nodepool_scale). New nodes install the full managed GPU stack. @@ -368,7 +339,7 @@ az aks nodepool scale \  If the **Full managed stack** profile isn't the right fit, AKS supports two alternative profiles on GPU node pools. -# [Driver only (preview)](#tab/driver-only)+# [Driver only](#tab/driver-only) Use this profile when you want AKS to install and maintain the NVIDIA GPU driver, but you plan to deploy the device plugin and metrics exporter yourself (for example, with the [NVIDIA GPU Operator](nvidia-gpu-operator)). Set `--enable-managed-gpu=false`:  ```azurecli

@@ -16,12 +16,18 @@ Azure Kubernetes Application Network traffic management uses Istio [ambient mode  This article provides step-by-step instructions on how to implement these traffic management use cases using Azure Kubernetes Application Network across multiple Azure Kubernetes Service (AKS) clusters. +## Watch the demo++This optional video demonstrates Application Network traffic policies across AKS clusters.++For a separate viewing option, [open the demo](https://aka.ms/aks/appnetdemo).+ ## Prerequisites  Before you begin, ensure you have the following prerequisites in place:  - An [existing Azure Kubernetes Application Network resource](get-started#create-an-azure-kubernetes-application-network-resource).-- Two [existing AKS clusters](get-started#create-an-aks-cluster) with network reachability between their east-west gateways. You can achieve this using [Azure virtual network (VNet) peering](/en-us/azure/virtual-network/virtual-network-peering-overview), [VNet to VNet VPN connection](/en-us/azure/vpn-gateway/vpn-gateway-howto-vnet-vnet-resource-manager-portal), or another supported connectivity model.+- Two [existing AKS clusters](get-started#create-an-aks-cluster) configured for [multi-cluster connectivity](multi-cluster-connectivity). Members in different mesh networks need to reach remote east-west gateways; members in the same network need direct pod-IP reachability.      - Both clusters must be [joined as members to your Azure Kubernetes Application Network](get-started#join-an-aks-cluster-as-a-member-of-azure-kubernetes-application-network). - The Istio CLI, [istioctl](https://istio.io/latest/docs/setup/install/istioctl/), installed. To select the correct version of istioctl, refer to the [supported Istio version](supported-versions) for your Application Network. You can install istioctl using the following command:@@ -90,7 +96,7 @@ Before you begin, ensure you have the following prerequisites in place:  This architecture extends the [Bookinfo sample application](https://istio.io/latest/docs/ambient/getting-started/deploy-sample-app/) to run across multiple AKS clusters. -![Screenshot of an architecture diagram showing the Bookinfo sample application architecture running across two AKS clusters.](media/traffic-management-use-cases/traffic-management-sample-architecture.png)+[![Diagram of the Bookinfo sample across two AKS clusters, showing the productpage, details, reviews, and ratings services and the global reviews service endpoints.](media/traffic-management-use-cases/traffic-management-sample-architecture.svg)](media/traffic-management-use-cases/traffic-management-sample-architecture.svg#lightbox)  - **Cluster 1**hosts:     - `productpage-v1` (Python)@@ -103,7 +109,7 @@ This architecture extends the [Bookinfo sample application](https://istio.io/lat  The `productpage` service sends requests to `reviews.default.svc.cluster.local`. Since the `reviews` service is available in both AKS clusters with global service scope, requests can be routed to any available backends (`reviews-v1`, `reviews-v2`, or `reviews-v3`) across clusters. The `reviews-v2` and `reviews-v3` services also call the `ratings` service to retrieve rating data before returning the response to `productpage`. -This architecture demonstrates how a distributed application can span across multiple clusters while continuing to use a consistent service name and communication pattern, enabling seamless service interaction across cluster boundaries.+The arrows show logical service calls and candidate endpoints. The actual cross-cluster path depends on the members' network names and available reachability. For details, see [Configure multi-cluster connectivity](multi-cluster-connectivity).  ### Deploy the application services 

@@ -1,21 +1,21 @@-# Troubleshoot Kubernetes resource viewer access in Azure portal - Azure | Microsoft Learn+# Troubleshoot Kubernetes resource viewer access in Azure portal for AKS - Azure | Microsoft Learn  ## Summary -This article discusses how to resolve an issue that prevents you from using the Azure portal to view resources in the Kubernetes resource viewer.+This article discusses how to resolve an issue that prevents you from using the Azure portal to view resources in the Kubernetes resource viewer for Azure Kubernetes Service (AKS).  ## Symptoms -When you try to view resources in the [Kubernetes resource viewer](/en-us/azure/aks/kubernetes-portal) in the [Azure portal](https://portal.azure.com), you receive the following error message:+When you try to view resources in the [Kubernetes resource viewer](/en-us/azure/aks/kubernetes-portal) in the [Azure portal](https://portal.azure.com), you receive the following error message.  >-> `Unable to reach the API server '<server-url>' or API is too busy to response. Check your network setting`.+> `Unable to reach the API server '<server-url>' or API is too busy to respond. Check your network settings`. -[![Screenshot of the Kubernetes resource viewer in Azure portal showing an API server network connectivity error message.](media/cannot-view-resources-kubernetes-resource-viewer-portal/network-error.png)](media/cannot-view-resources-kubernetes-resource-viewer-portal/network-error.png#lightbox)+[![Screenshot of the Kubernetes resource viewer in the Azure portal showing an API server network connectivity error message.](media/cannot-view-resources-kubernetes-resource-viewer-portal/network-error.png)](media/cannot-view-resources-kubernetes-resource-viewer-portal/network-error.png#lightbox)  ## Cause 1: You configured authorized IP address ranges -You configured your Microsoft Azure Kubernetes Service (AKS) cluster to access the cluster API server by using authorized IP address ranges that your computer can't access.+You configured your AKS cluster to access the cluster API server by using authorized IP address ranges that your computer can't access.  ### Solution @@ -23,12 +23,12 @@ When you run the [az aks create](/en-us/cli/azure/aks#az-aks-create) or [az aks  ## Cause 2: Your AKS cluster is private -Your AKS is created as a private cluster, and you're accessing the Azure portal from a network that can't communicate with the subnet to which your AKS is connected.+You created your AKS cluster as a private cluster, and you're accessing the Azure portal from a network that can't communicate with the subnet to which your AKS is connected.  ### Solution  This behavior is expected. To view the AKS resources in the Azure portal, you must access the Azure portal from a client that's located on a network that has connectivity to the AKS subnet. Then, you can view all AKS-related resources in the Azure portal. -## More information+## References  - [How to find my IP to include in --api-server-authorized-ip-ranges?](/en-us/azure/aks/api-server-authorized-ip-ranges#how-to-find-my-ip-to-include-in---api-server-authorized-ip-ranges)

@@ -1,8 +1,8 @@-# Error from server - error dialing backend - dial tcp message - Azure | Microsoft Learn+# Error from server - error dialing backend - dial tcp message in AKS - Azure | Microsoft Learn  ## Summary -This article explains how to troubleshoot the "Error from server: error dialing backend: dial tcp" error message that you might receive when you use `kubectl` commands or other tools to connect to the API server. This error message indicates that the API server is having trouble connecting to an upstream component, like a Kubernetes service or kubelet, which is required for certain operations.+This article helps you troubleshoot the "Error from server: error dialing backend: dial tcp" error in Azure Kubernetes Service (AKS) so you can restore `kubectl logs`, `exec`, and `top` access. The error indicates that the API server can't connect to an upstream component, like a Kubernetes service or kubelet, that certain operations require.  ## Symptoms @@ -11,37 +11,36 @@ You receive an "Error from server: error dialing backend: dial tcp" error messag - Use any of the `kubectl``logs`, `exec`, `attach`, `top`, or `port-forward` commands. - Use third-party Kubernetes client tools to achieve the same functionality as the commands in the previous list item. -### Why the error occurs+### Why the dial tcp error occurs -The Kubernetes API server has to forward API requests to an upstream component for several use cases. This error occurs if the API server can't establish a Transmission Control Protocol (TCP) connection to the upstream component. Example of such upstream components includes kubernetes services that are inside the cluster and kubelet.+The Kubernetes API server has to forward API requests to an upstream component for several use cases. This error occurs if the API server can't establish a Transmission Control Protocol (TCP) connection to the upstream component. Examples of such upstream components include Kubernetes services that are inside the cluster and kubelet. -If the issue persists, a network blockage is likely the cause. To identify the responsible network configuration, first determine the scope of the problem.+If the issue persists, a network blockage is likely the cause. To identify the responsible network configuration, determine the scope of the problem. -### Narrowing down: Are all `kubectl` subcommands failing?+### Determine whether all `kubectl` subcommands fail -Try to run at least the `kubectl exec`, `kubectl logs <podname>`, and `kubectl top pods` commands.+Try to run at least the `kubectl exec`, `kubectl logs <podname>`, and `kubectl top pods` commands. Use the following guidance to interpret the results. -If only `kubectl logs <podname>` or `kubectl exec` fails, check whether the issue occurs by having pods on different nodes.+- If only `kubectl logs <podname>` or `kubectl exec` fails, check whether the issue occurs by having pods on different nodes.+- If only `kubectl top pods` fails, check whether the issue occurs for pods on all nodes or only for pods on one node. -If only `kubectl top pods` fails, check whether the issue occurs for pods on all nodes or only for pods on one node.+## Cause 1: Kubelet port 10250 is blocked -## Cause 1: kubelet port (node:10250) blocked+Pod-specific access issues, such as those that occur when you run `kubectl logs` and `kubectl exec`, happen if the API server can't reach the node on port 10250 to access the Kubelet API. A network security group (NSG) or firewall might block the connection. -Pod-specific access issues, such as those that are experienced by running `kubectl logs` and `kubectl exec`, occur if the API server can't reach the node on port 10250 to access the Kubelet API. These issues can be caused by a connection that's blocked by a Network Security Group (NSG) or firewall.+To resolve the issue, check whether the NSG on the node subnet includes an inbound rule that blocks TCP port 10250. -To resolve the issue, check whether the NSG on the node subnet includes an inbound rule that might block TCP port 10250.+## Cause 2: Specific service failure -## Cause 2: specific service failure+Kubernetes accesses `svc/metrics-server` under the `kube-system` namespace to run `kubectl top` commands. In other scenarios, such as [admission webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/), the API server can reach other services. The error message can vary depending on the service failure pattern. -Kubernetes accesses `svc/metrics-server` under the `kube-system` namespace for running kubectl top commands. There are other scenarios, such as [admission webhooks](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/), in which the API server can also reach other services. It's important to note that, depending on the service failure pattern, the error message can vary.--To troubleshoot the issue, check the error message to identify which service is affected and review the status of the related pods, services, and endpoints.+To troubleshoot the issue, check the error message to identify which service is affected. Review the status of the related pods, services, and endpoints.  ## Cause 3: Konnectivity or tunnel failure -When [API Server VNet Integration](/en-us/azure/aks/api-server-vnet-integration) isn't enabled, AKS deploys a tunnel solution that proxies API server requests to in-cluster networking locations. Most AKS clusters use the Konnectivity solution. Konnectivity doesn't require that you open special ports on the API server. For more information, see [AKS required network rules](/en-us/azure/aks/outbound-rules-control-egress#azure-global-required-network-rules).+When you don't enable [API Server VNet Integration](/en-us/azure/aks/api-server-vnet-integration), AKS deploys a tunnel solution that proxies API server requests to in-cluster networking locations. Most AKS clusters use the Konnectivity solution. Konnectivity doesn't require you to open special ports on the API server. For more information, see [AKS required network rules](/en-us/azure/aks/outbound-rules-control-egress#azure-global-required-network-rules). -To resolve the issue, check whether the `konnectivity-agent` in the `kube-system` namespace is running and its containers are in a ready state. Try to delete the pods to see whether the connection recovers after the new pods are ready.+To resolve the issue, check whether the `konnectivity-agent` in the `kube-system` namespace is running and its containers are in a ready state. Try deleting the pods to see whether the connection recovers after the new pods are ready.  **Third-party information disclaimer** 

@@ -1,17 +1,19 @@-# Fix intermittent time-outs or server issues during app access - Azure | Microsoft Learn+# Troubleshoot intermittent time-outs or server issues during app access on AKS - Azure | Microsoft Learn  ## Summary -This article describes how to troubleshoot intermittent connectivity issues that affect your applications that are hosted on an Azure Kubernetes Service (AKS) cluster.+This article helps you troubleshoot intermittent time-outs or server issues that affect applications hosted on an Azure Kubernetes Service (AKS) cluster. Learn how to identify the cause, such as container memory limits, and restore reliable application access.  ## Prerequisites +Ensure you have the following prerequisites installed and configured:+ - The Client URL ([cURL](https://www.tecmint.com/install-curl-in-linux/)) tool, or a similar command-line tool. - The Kubernetes [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) tool, or a similar tool to connect to the cluster. To install kubectl by using [Azure CLI](/en-us/cli/azure/install-azure-cli), run the [az aks install-cli](/en-us/cli/azure/aks#az-aks-install-cli) command.  ## Symptoms -When you run a cURL command, you occasionally receive a "Timed out" error message. The output might resemble the following text:+When you run a cURL command, you occasionally receive a "Timed out" error message. The output might resemble the following text.  ```console $ # One connection is successful, which results in a HTTP 200 response.@@ -43,9 +45,9 @@ HTTP/1.1 200 OK  ## Cause -Intermittent time-outs suggest component performance issues, as opposed to networking problems.+Intermittent timeouts suggest component performance issues, rather than networking problems. -In this scenario, it's important to check the usage and health of the components. You can use the *inside-out* technique to check the status of the pods. Run the [kubectl top](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#top) and [kubectl get](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get) commands, as follows:+In this scenario, check the usage and health of the components. Use the *inside-out* technique to check the status of the pods. Run the [kubectl top](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#top) and [kubectl get](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get) commands as follows.  ```console $ kubectl top pods  # Check the health of the pods and the nodes.@@ -65,7 +67,7 @@ The output shows that the current usage of the pods and nodes appears to be acce  Although the pod is in the `Running` state, one restart occurs after the first 108 seconds of the pod running. This occurrence might indicate that some issues affect the pods or containers that run in the pod. -If the issue persists, the status of the pod changes after some time:+If the issue persists, the status of the pod changes after some time. The following example shows this scenario.  ```console $ kubectl get pods@@ -77,7 +79,7 @@ This example shows that the `Ready` state is changed, and there are several rest  This situation occurs because the container fails after starting, and then Kubernetes tries to restart the container to force it to start working. However, if the issue persists, the application continues to fail after it runs for some time. Kubernetes eventually changes the status to `CrashLoopBackOff`. -To check the logs for the pod, run the following [kubectl logs](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs) commands:+To check the logs for the pod, run the following [kubectl logs](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs) commands.  ```console $ kubectl logs my-deployment-fc94b7f98-m9z2l@@ -115,7 +117,7 @@ Started increasing memory  Log entries were made the previous time that the container was run. The existence of these entries suggests that the application did start, but it closed because of some issues. -Check the service associated with the deployment and try to curl the cluster IP of the service from inside the cluster to identify the issue:+Check the service associated with the deployment and try to curl the cluster IP of the service from inside the cluster to identify the issue.  ```console $ kubectl get svc # Check the service associated with deployment@@ -124,7 +126,7 @@ kubernetes              ClusterIP      10.0.0.1      <none>         443/TCP my-deployment-service   LoadBalancer   10.0.136.71   20.62.x.x      80:30790/TCP   21m ``` -The next step is to check the events of the pod by running the [kubectl describe](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe) command:+The next step is to check the events of the pod by running the [kubectl describe](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe) command.  ```console $ kubectl describe pod my-deployment-fc94b7f98-m9z2l@@ -164,7 +166,7 @@ Events:   Warning  BackOff  4m10s (x902 over 4h2m)  kubelet  Back-off restarting failed container ``` -Observations:+The following are potential observations:  - The exit code is 137. For more information about exit codes, see the [Docker run reference](https://docs.docker.com/engine/reference/run/#exit-status) and [Exit codes with special meanings](https://tldp.org/LDP/abs/html/exitcodes.html). - The termination reason is `OOMKilled`.

@@ -6,6 +6,11 @@ When any of the Arc-enabled Kubernetes agents are updated, all of the agents in  We generally recommend using the most recent versions of the agents. The [version support policy](agent-upgrade#version-support-policy) covers the most recent version and the two previous versions (N-2). +## Version 1.39.3 (November 2026)++- Security updates include fixes for multiple CVEs, removal of the compromised eMudhra certificate from the root CA certificate bundle, and agents built using the Go 1.26.10 toolchain for additional security hardening.+- General bug fixes and reliability improvements+ ## Version 1.38.0 (October 2026)  - Security vulnerability fixes

@@ -280,4 +280,4 @@ You can retrieve logs and status updates from Karpenter to help diagnose and deb  ## Node auto-provisioning metrics -You can enable [control plane metrics (Preview)](control-plane-metrics-monitor) to see specific Karpenter metrics and operations from [node auto-provisioning](control-plane-metrics-default-list#minimal-ingestion-for-default-off-targets) with the [Azure Monitor managed service for Prometheus add-on](/en-us/azure/azure-monitor/essentials/prometheus-metrics-overview).+You can enable [control plane metrics](control-plane-metrics-monitor) to see specific Karpenter metrics and operations from [node auto-provisioning](control-plane-metrics-default-list#minimal-ingestion-for-default-off-targets) with the [Azure Monitor managed service for Prometheus add-on](/en-us/azure/azure-monitor/essentials/prometheus-metrics-overview).

@@ -202,7 +202,7 @@ spec: #### ResourcePlacement example  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ResourcePlacement metadata:   name: rp-demo@@ -264,7 +264,7 @@ spec: #### ResourcePlacement example  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ResourcePlacement metadata:   name: rp-demo@@ -329,7 +329,7 @@ spec: #### ResourcePlacement example  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ResourcePlacement metadata:   name: rp-demo

@@ -10,9 +10,11 @@ This article shows you how to migrate AKS Automatic clusters between supported [ | --- | --- | | AKS Automatic with managed system node pools to AKS Standard (`base` SKU) | Supported | | AKS Automatic without managed system node pools to AKS Automatic with managed system node pools | Supported in preview in regions where managed system node pools are generally available |-| AKS Standard (`base` SKU) to AKS Automatic with managed system node pools | Coming soon |+| AKS Standard (`base` SKU) to AKS Automatic with managed system node pools | Supported in preview in regions where managed system node pools are generally available | | AKS Automatic with managed system node pools to AKS Automatic without managed system node pools | Not supported | +By using managed system node pools, AKS provisions, scales, and upgrades the system node pools and system components for you, and isolates them from your workloads. AKS Automatic clusters are also backed by a [pod readiness SLA and an uptime SLA](../intro-aks-automatic#service-level-agreement). For more information, see [Overview of AKS Automatic clusters with managed system node pools](aks-automatic-managed-system-node-pools-about).+ ## Migrate from an AKS Automatic cluster with managed system node pools to an AKS Standard cluster  ### How the migration works@@ -173,3 +175,304 @@ The following example output shows the `hostedSystemProfile.enabled` property se   "systemNodeSubnetId": "<system-node-subnet-resource-id>" } ```++## Migrate from an AKS Standard cluster to an AKS Automatic cluster with managed system node pools (preview)++AKS supports preview migration from AKS Standard (`base` SKU) clusters to AKS Automatic clusters with managed system node pools in regions where managed system node pools are generally available.++To migrate, you first configure your AKS Standard cluster to match the AKS Automatic configuration, and then update the cluster SKU. Most of the work is in preparing the cluster. Complete the following sections in order, because several settings depend on others being configured first.++### Before you begin++Important++AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:++- [AKS support policies](/en-us/azure/aks/support-policies)+- [Azure support FAQ](/en-us/azure/aks/faq)++- An existing AKS Standard (`base` SKU) cluster. For general AKS Automatic requirements and limitations, see [Quickstart: Create an Azure Kubernetes Service (AKS) Automatic cluster](quick-automatic-managed-network) and [Overview of AKS Automatic clusters with managed system node pools](aks-automatic-managed-system-node-pools-about).+- Azure CLI version 2.86.0 or later and the latest version of the `aks-preview` extension.+- The `Microsoft.ContainerService/NonHoboToHoboConversionPreview` feature flag registered in your subscription. To register the feature flag and install the extension, follow the steps in Prerequisites.++Set environment variables for your cluster:++```azurecli+export RESOURCE_GROUP="<resource-group>"+export RESOURCE_NAME="<cluster-name>"+export NODE_RESOURCE_GROUP=$(az aks show --resource-group "${RESOURCE_GROUP}" --name "${RESOURCE_NAME}" --query nodeResourceGroup --output tsv)+```++### Migration requirements++Before you can update the SKU, your cluster must meet the following requirements. Some requirements differ depending on whether your cluster uses an AKS-managed virtual network or a custom virtual network. You can configure these settings by using the Azure portal, Azure CLI, an ARM template, or other tools. The rest of this article shows the Azure CLI steps in the order that you should apply them.++| Requirement | Required setting | Learn more |+| --- | --- | --- |+| Cluster tier | Standard | [AKS pricing tiers](../free-standard-pricing-tiers) |+| Load balancer SKU | Standard | [Customize cluster egress with outbound types](../egress-outboundtype) |+| Outbound type | AKS-managed virtual network: managed NAT gateway (`StandardV2`). The `loadBalancer` outbound type isn't supported.Custom virtual network: any outbound type. | [Update outboundType after cluster creation](../egress-outboundtype#update-outboundtype-after-cluster-creation) |+| Cluster identity | AKS-managed virtual network: system-assigned managed identity.Custom virtual network: user-assigned managed identity with the *Network Contributor* role on the virtual network. | [Use a user-assigned managed identity](../user-assigned-managed-identity) |+| API Server VNet Integration | Enabled, with a dedicated API server subnet in the same virtual network as your nodes | [Convert an existing AKS Standard cluster to API Server VNet Integration](../api-server-vnet-integration#convert-an-existing-aks-standard-cluster-to-api-server-vnet-integration) |+| Network plugin | Azure CNI Overlay (`azure` plugin in `overlay` mode) | [Update the IPAM mode to Azure CNI Overlay](../update-azure-cni#update-the-ipam-mode-to-azure-cni-overlay) |+| Network dataplane and network policy | Cilium | [Azure CNI Powered by Cilium](../azure-cni-powered-by-cilium) |+| Node provisioning mode | Auto | [Node auto-provisioning](../node-auto-provisioning) |+| System node pool OS SKU | Azure Linux | [Use the Azure Linux Container Host](../use-azure-linux) |+| System node pool SSH access | Disabled | [Manage SSH access on AKS cluster nodes](../manage-ssh-node-access) |+| System node pool OS disk | Ephemeral | [Ephemeral OS disks in AKS](../concepts-storage#ephemeral-os-disks-in-aks) |+| Azure Policy add-on | Enabled | [Use Azure Policy](../use-azure-policy) |+| Azure Key Vault Secrets Provider add-on | Enabled, with secret rotation turned on | [Use the Secrets Store CSI Driver](../csi-secrets-store-driver) |+| Auto-upgrade channels | Cluster: `stable`Node OS: `NodeImage` | [Automatically upgrade an AKS cluster](../auto-upgrade-cluster) |+| Workload identity and OIDC issuer | Enabled | [Use Microsoft Entra Workload ID](../workload-identity-overview) |+| Microsoft Entra integration and Azure RBAC | Enabled | [Microsoft Entra integration](../access-control-managed-azure-ad) |+| Local accounts | Disabled | [Manage local accounts with Microsoft Entra integration](../local-accounts) |+| Image Cleaner | Enabled | [Image Cleaner](../image-cleaner) |+| Vertical Pod Autoscaler | Enabled | [Vertical Pod Autoscaling](../vertical-pod-autoscaler) |+| KEDA | Enabled | [KEDA](../keda-about) |+| Node resource group lockdown | `ReadOnly` | [Node resource group lockdown](../node-resource-group-lockdown) |++### Plan your migration++1. Determine whether your cluster uses an AKS-managed virtual network or a custom virtual network. If the following command returns a virtual network name, your cluster uses an AKS-managed virtual network. If it returns nothing, your cluster uses a custom virtual network.++    ```azurecli+    az network vnet list --resource-group "${NODE_RESOURCE_GROUP}" --query "[].name" --output tsv+    ```++    Several steps in this article differ depending on your virtual network type.+2. Check your cluster's current network plugin, outbound type, load balancer SKU, and node resource group restriction level:++    ```azurecli+    az aks show \+        --resource-group "${RESOURCE_GROUP}" \+        --name "${RESOURCE_NAME}" \+        --query "{networkPlugin:networkProfile.networkPlugin, networkPluginMode:networkProfile.networkPluginMode, outboundType:networkProfile.outboundType, loadBalancerSku:networkProfile.loadBalancerSku, nrgRestrictionLevel:nodeResourceGroupProfile.restrictionLevel}" \+        --output table+    ```++    - **Network plugin**: Must be `azure` with plugin mode `overlay`. If your cluster uses kubenet or Azure CNI without Overlay, update it to Azure CNI Overlay before you continue. For more information, see [Update the IPAM mode to Azure CNI Overlay](../update-azure-cni#update-the-ipam-mode-to-azure-cni-overlay).+    - **Load balancer SKU**: Must be `standard`. If your cluster still uses `basic`, update it with `az aks update --load-balancer-sku standard`.+    - **Node resource group restriction level**: For an AKS-managed virtual network, you need to create a subnet in the node resource group. If the restriction level is `ReadOnly`, set it to `Unrestricted` before you continue. You lock it down again in a later step.+3. Review the changes that disrupt running workloads or access, and plan a maintenance window:++    - Changing the outbound type changes your cluster's egress IP addresses. Update any firewall rules that allow traffic from the cluster.+    - Enabling API Server VNet Integration changes the API server IP address and requires a cluster restart. This change is permanent.+    - Changing the network dataplane, OS SKU, and SSH access settings reimages your nodes.+    - Disabling local accounts removes static kubeconfig credentials. Ensure you have access through Microsoft Entra ID and an Azure RBAC role, such as *Azure Kubernetes Service RBAC Cluster Admin*, before you disable them.+    - The migration enables Deployment Safeguards in `Enforce` mode with baseline Pod Security Standards. Deployments that don't comply are rejected. You can exclude namespaces, but you can't switch the cluster-wide level to `Warn`. Review your workloads before you migrate. For more information, see [Use Deployment Safeguards](../deployment-safeguards#excluding-namespaces).++### Prepare cluster networking++1. **Outbound type**: For an AKS-managed virtual network, the outbound type must be a managed NAT gateway. The `loadBalancer` outbound type isn't supported. If needed, update the outbound type:++    ```azurecli+    az aks update \+        --resource-group "${RESOURCE_GROUP}" \+        --name "${RESOURCE_NAME}" \+        --outbound-type managedNATGateway \+        --outbound-type-sku StandardV2+    ```++    For a custom virtual network, any outbound type is supported and no change is required. For more information, see [Update outboundType after cluster creation](../egress-outboundtype#update-outboundtype-after-cluster-creation).+2. **Cluster identity**: For an AKS-managed virtual network, keep the cluster's system-assigned managed identity. For a custom virtual network, the cluster must use a user-assigned managed identity that has the *Network Contributor* role on the virtual network. If needed, assign the identity:++    ```azurecli+    az aks update \+        --resource-group "${RESOURCE_GROUP}" \+        --name "${RESOURCE_NAME}" \+        --enable-managed-identity \+        --assign-identity "${IDENTITY_RESOURCE_ID}" \+        --yes+    ```++    Replace `IDENTITY_RESOURCE_ID` with the resource ID of your user-assigned managed identity. For more information, see [Use a user-assigned managed identity](../user-assigned-managed-identity).+3. **API server subnet**: Create a dedicated subnet for the API server in the same virtual network as your nodes. The subnet must be separate from your node subnet, be at least `/28`, and be delegated to `Microsoft.ContainerService/managedClusters`.++    For an AKS-managed virtual network, create the subnet in the virtual network in the node resource group:++    ```azurecli+    export VNET_NAME=$(az network vnet list --resource-group "${NODE_RESOURCE_GROUP}" --query "[0].name" --output tsv)++    export APISERVER_SUBNET_ID=$(az network vnet subnet create \+        --resource-group "${NODE_RESOURCE_GROUP}" \+        --vnet-name "${VNET_NAME}" \+        --name apiserver-subnet \+        --address-prefixes "<unused-address-range>" \+        --delegations Microsoft.ContainerService/managedClusters \+        --query id --output tsv)+    ```++    For a custom virtual network, create the subnet in your own virtual network instead, and set `APISERVER_SUBNET_ID` to its resource ID.+4. **API Server VNet Integration**: Enable API Server VNet Integration with the subnet you created, and then restart the cluster:++    ```azurecli+    az aks update \+        --resource-group "${RESOURCE_GROUP}" \+        --name "${RESOURCE_NAME}" \+        --enable-apiserver-vnet-integration \+        --apiserver-subnet-id "${APISERVER_SUBNET_ID}"++    az aks stop --resource-group "${RESOURCE_GROUP}" --name "${RESOURCE_NAME}"+    az aks start --resource-group "${RESOURCE_GROUP}" --name "${RESOURCE_NAME}"+    ```++    For more information, see [Convert an existing AKS Standard cluster to API Server VNet Integration](../api-server-vnet-integration#convert-an-existing-aks-standard-cluster-to-api-server-vnet-integration).+5. **Network dataplane**: Update the network dataplane to Cilium. This command also sets the network policy to Cilium.++    ```azurecli+    az aks update \+        --resource-group "${RESOURCE_GROUP}" \+        --name "${RESOURCE_NAME}" \+        --network-dataplane cilium+    ```++    For more information, see [Azure CNI Powered by Cilium](../azure-cni-powered-by-cilium).++Important++Update the network dataplane before you enable Auto node provisioning mode in the next section. After Auto mode is enabled, you can't change the network dataplane.++### Prepare the system node pool and node provisioning++1. **Ephemeral OS disk**: The system node pool must use an ephemeral OS disk. Check the current OS disk type:++    ```azurecli+    az aks nodepool list \+        --resource-group "${RESOURCE_GROUP}" \+        --cluster-name "${RESOURCE_NAME}" \+        --query "[?mode=='System'].{name:name, osDiskType:osDiskType}" \+        --output table+    ```++    You can't change the OS disk type of an existing node pool. If your system node pool uses a `Managed` OS disk, [add a new system node pool](../create-node-pools) with `--mode System --node-osdisk-type Ephemeral --os-sku AzureLinux --ssh-access disabled`, move your system workloads to it, and then [delete the old node pool](../delete-node-pool). You can then skip the next step.+2. **OS SKU and SSH access**: Update the system node pool to use Azure Linux and disable SSH access:++    ```azurecli+    az aks nodepool update \+        --resource-group "${RESOURCE_GROUP}" \+        --cluster-name "${RESOURCE_NAME}" \+        --name "<system-node-pool-name>" \+        --os-sku AzureLinux \+        --ssh-access disabled \+        --yes+    ```++    This command reimages the node pool. Wait for it to finish before you make other changes to the node pool. For more information, see [Use the Azure Linux Container Host](../use-azure-linux) and [Manage SSH access on AKS cluster nodes](../manage-ssh-node-access).+3. **Node provisioning mode**: Enable Auto node provisioning mode:++    ```azurecli+    az aks update \+        --resource-group "${RESOURCE_GROUP}" \+        --name "${RESOURCE_NAME}" \+        --node-provisioning-mode Auto+    ```++    After you enable Auto mode, you can't revert to Manual mode without first removing all autoprovisioned nodes. For more information, see [Node auto-provisioning](../node-auto-provisioning).++### Enable required features and add-ons++1. Update the cluster tier to Standard, and enable the Azure Policy and Azure Key Vault Secrets Provider add-ons with secret rotation:++    ```azurecli+    az aks update --resource-group "${RESOURCE_GROUP}" --name "${RESOURCE_NAME}" --tier standard++    az aks enable-addons \+        --resource-group "${RESOURCE_GROUP}" \+        --name "${RESOURCE_NAME}" \+        --addons azure-policy,azure-keyvault-secrets-provider \+        --enable-secret-rotation+    ```+2. Enable the remaining required features:++    ```azurecli+    az aks update \+        --resource-group "${RESOURCE_GROUP}" \+        --name "${RESOURCE_NAME}" \+        --auto-upgrade-channel stable \+        --node-os-upgrade-channel NodeImage \+        --enable-workload-identity \+        --enable-oidc-issuer \+        --enable-aad \+        --enable-azure-rbac \+        --disable-local-accounts \+        --enable-image-cleaner \+        --enable-vpa \+        --enable-keda \+        --yes+    ```++### Lock down the node resource group++Set the node resource group restriction level to `ReadOnly`. Do this step last, because lockdown blocks changes to resources in the node resource group, including the API server subnet you created earlier.++```azurecli+az aks update \+    --resource-group "${RESOURCE_GROUP}" \+    --name "${RESOURCE_NAME}" \+    --nrg-lockdown-restriction-level ReadOnly+```++For more information, see [Node resource group lockdown](../node-resource-group-lockdown).++### Migrate the cluster to AKS Automatic++If any requirement is still unmet, the migration command fails and lists the unmet requirements it detected. Some checks run only after earlier ones pass, so resolve the listed items and run the command again until it succeeds.++#### AKS-managed virtual network++```azurecli+az aks update \+    --resource-group "${RESOURCE_GROUP}" \+    --name "${RESOURCE_NAME}" \+    --sku automatic \+    --enable-hosted-system+```++#### Custom virtual network++Provide a subnet for the managed system node pool by using the `--system-node-subnet-id` parameter. For subnet requirements, see [Migrate an existing AKS Automatic cluster to managed system node pools](aks-automatic-managed-system-node-pools-about#migrate-an-existing-aks-automatic-cluster-to-managed-system-node-pools-preview). The `--node-subnet-id` parameter is optional. If you don't provide it, AKS uses the existing system node pool subnet.++```azurecli+az aks update \+    --resource-group "${RESOURCE_GROUP}" \+    --name "${RESOURCE_NAME}" \+    --sku automatic \+    --enable-hosted-system \+    --system-node-subnet-id "${SYSTEM_NODE_SUBNET_ID}" \+    --node-subnet-id "${NODE_SUBNET_ID}"+```++### Verify the migration++1. Confirm that the cluster uses the `automatic` SKU and `Standard` tier, and that the update finished successfully:++    ```azurecli+    az aks show \+        --resource-group "${RESOURCE_GROUP}" \+        --name "${RESOURCE_NAME}" \+        --query "{sku:sku, provisioningState:provisioningState}" \+        --output json+    ```++    Confirm `sku.name` is `Automatic`, `sku.tier` is `Standard`, and `provisioningState` is `Succeeded`.+2. Confirm the cluster uses managed system node pools:++    ```azurecli+    az aks show \+        --resource-group "${RESOURCE_GROUP}" \+        --name "${RESOURCE_NAME}" \+        --query hostedSystemProfile \+        --output json+    ```++    Confirm `hostedSystemProfile.enabled` is `true`. For a custom virtual network, `systemNodeSubnetId` and `nodeSubnetId` show your subnet resource IDs. For an AKS-managed virtual network, these values are `null`.+3. Get credentials for the cluster and confirm all nodes are in the `Ready` state:++    ```azurecli+    az aks get-credentials --resource-group "${RESOURCE_GROUP}" --name "${RESOURCE_NAME}"+    kubectl get nodes -o wide+    ```++### Failure behavior++If migration can't provision managed system pool infrastructure (for example, because of regional capacity constraints), AKS preserves existing system and user node pools in their pre-migration state.

@@ -6,7 +6,7 @@ This article walks you through how to create a multi-instance GPU node pool usin  Tip -For a fully managed MIG experience where AKS installs and maintains the NVIDIA driver, device plugin, and DCGM metrics exporter, see [Create a managed Multi-Instance GPU (MIG) node pool (preview)](aks-managed-gpu-nodes#create-a-managed-multi-instance-gpu-mig-node-pool-preview).+For a fully managed MIG experience where AKS installs and maintains the NVIDIA driver, device plugin, and DCGM metrics exporter, see [Create a managed Multi-Instance GPU (MIG) node pool](aks-managed-gpu-nodes#create-a-managed-multi-instance-gpu-mig-node-pool).  ## Prerequisites and limitations 

@@ -28,7 +28,7 @@ Application Network currently allows both encrypted and unencrypted traffic. Thi mTLS in Azure Kubernetes Application Network helps you:  - **Protect data in transit**: All service-to-service traffic can be encrypted automatically (FIPS-compliant).-- **Verify service identity**: Each workload has a unique Application Network-managed root and intermediate certificate.+- **Verify service identity**: Each Application Network has a unique root CA, and each member has a dedicated intermediate CA chained to that root. Workloads prove their identities with short-lived workload certificates signed by their member's intermediate CA. - **Simplify operations**: Application Network handles certificate issuance, rotation, and revocation for users.  ## Certificate management@@ -37,13 +37,13 @@ Azure Kubernetes Application Network provides certificate management backed by A  Application Network automatically manages the entire certificate lifecycle for root and intermediate certificates including provisioning, renewal, and rotation. This ensures that workload identities remain valid, and communication stays secure over time. You don't need to create, store, or rotate certificates manually. -The following diagram illustrates the certificate hierarchy in Application Network, showing how the root CA, intermediate CAs, and workload certificates are structured to establish trust across connected clusters:+The following diagram shows how the AppNet root CA signs a separate intermediate CA for each member. Each intermediate CA issues workload certificates with namespace and service account identities: -[![Screenshot of a diagram that illustrates the certificate hierarchy in Azure Kubernetes Application Network, showing how the root CA, intermediate CAs, and workload certificates are structured to establish trust across connected clusters.](media/security/certificate-management.png)](media/security/certificate-management.png#lightbox)+[![Diagram of the Application Network certificate hierarchy, with one Key Vault-backed root CA, a separate intermediate CA for each member, and workload certificates that carry namespace and service account identities.](media/security/certificate-management.svg)](media/security/certificate-management.svg#lightbox)  ### Certificate hierarchy and lifecycle -When an Application Network resource is created, a **root CA** is provisioned to serve as the trust anchor for the Application Network. When an AKS cluster joins the Application Network as a member, Application Network issues an **intermediate certificate** for that member signed by the Application Network root CA. Each connected AKS cluster uses its Application Network-managed intermediate certificate to issue short-lived **workload certificates** to workloads in that cluster.+When an Application Network resource is created, a unique **root CA** is provisioned as its trust anchor. Each Application Network member receives a dedicated **intermediate CA** signed by that root CA. The member's intermediate CA issues short-lived **workload certificates** to workloads in its connected AKS cluster.  All workload certificates inherit trust from the Application Network root CA, maintaining a unified trust boundary across the resource. The rotation of root and intermediate certificates is handled transparently for users without causing any downtime or traffic disruptions. This managed hierarchy ensures consistent, verifiable trust across all clusters in an Application Network while automatically maintaining the validity of each certificate. @@ -54,7 +54,7 @@ From any member cluster, you can get the root CA certificate from config map `is | **Certificate type** | **Scope** | **Validity period** | **Rotation period** | **Purpose** | | --- | --- | --- | --- | --- | | **Root CA** | Azure Kubernetes Application Network | 12 months | 6 months | Establishes the trust boundary for all clusters connected to the Azure Kubernetes Application Network |-| **Intermediate CA** | Cluster | 90 days | 45 days | Issues workload certificates for workloads within that cluster |+| **Intermediate CA** | Application Network member | 90 days | 45 days | Issues workload certificates for workloads associated with that member | | **Workload certificate** | Workload | 24 hours | 12 hours | Authenticates workloads and secures service-to-service communication |  ## Workload identity and authorization@@ -90,7 +90,7 @@ spec:   - from:     - source:         principals:-        - cluster.local/ns/default/sa/myApp-gateway // see SPIFFE format above+        - cluster.local/ns/default/sa/myapp-gateway  # See SPIFFE format above. ```  ### Policy design guidance

@@ -12,13 +12,15 @@ Azure Kubernetes Application Network provides two types of logging that you can - **Data plane logs**: Include logs from application workloads, `applink-system` namespace components, and Envoy proxy access logs. - **Control plane logs**: Include Istiod logs from the Azure Kubernetes Application Network control plane components. -Azure Kubernetes Application Network logging is disabled by default. This article describes how to enable both types of logging and view them in Azure Monitor.+Configure data plane log collection and control plane diagnostic settings separately. If your AKS cluster already has Container Insights enabled, its existing collection settings apply. This article describes how to configure both log paths and view them in Azure Monitor.  ## Set environment variables  - Set environment variables for your selected workflow using the following commands:      ```bash+    export SUBSCRIPTION=<subscription-id>+     # Set if specifying an existing or creating a new Log Analytics workspace     export LAW_NAME=<log-analytics-workspace-name> @@ -68,36 +70,67 @@ When enabling data plane logs, you have the option to specify an existing Log An      ```kusto     ContainerLogV2+    | where TimeGenerated > ago(45m)     | where PodNamespace == "applink-system"     ```  ### Envoy access logs -In some cases, you might need to examine Envoy access logs when traffic is generated for your applications/workloads. To enable Envoy access logs, use the Telemetry API.+Use the [Telemetry API](https://istio.io/latest/docs/reference/config/telemetry/) as described in the following steps to enable Envoy access-log generation. Container Insights collects the proxy's standard output and sends it to your Log Analytics workspace, so collection must include the namespace where the proxy runs.++For HTTP request-level logs, traffic must pass through an Envoy waypoint or gateway. Traffic that only passes through ztunnel doesn't produce Envoy HTTP access logs. For waypoint deployment and enrollment steps, see [Traffic management use cases](traffic-management-use-cases#install-waypoint-and-mark-services-as-global).++Note++A namespace can have only one Telemetry default without a workload selector or `targetRefs`. If a default already exists, merge the access-logging configuration into that resource instead of creating another default. Namespace-specific or targeted access-logging settings override the corresponding mesh-wide settings. -1. In the YAML below, specify the namespace where you want to enable Envoy access logging. For mesh-wide Envoy access logs, remove the namespace from the metadata.+1. To enable Envoy access logging for a namespace, apply the following example. For an application namespace other than `default`, replace `default` in both the command and `metadata.namespace`.      ```bash     cat <<EOF | kubectl apply -n default -f -     apiVersion: telemetry.istio.io/v1     kind: Telemetry     metadata:-      name: mesh-logging-default+      name: namespace-access-logging       namespace: default     spec:       accessLogging:       - providers:         - name: envoy+        disabled: false+    EOF+    ```+2. To set the mesh-wide default for Envoy access logging, apply the configuration to the mesh root namespace, `applink-system`, in each member cluster where you want the default. Set the namespace in both the command and `metadata.namespace`.++    Note++    Removing `metadata.namespace` doesn't create a mesh-wide policy.++    ```bash+    cat <<EOF | kubectl apply -n applink-system -f -+    apiVersion: telemetry.istio.io/v1+    kind: Telemetry+    metadata:+      name: mesh-access-logging+      namespace: applink-system+    spec:+      accessLogging:+      - providers:+        - name: envoy+        disabled: false     EOF     ``` -    Envoy access logs will be available in the Log Analytics workspace where the Container Insights add-on was enabled.-2. View the Envoy access logs by navigating to the Log Analytics workspace and running the following query:+    The namespace and mesh-wide examples configure defaults at different scopes. A namespace-specific setting takes precedence over the mesh-wide default.+3. Generate application traffic through the waypoint or gateway, then query its container output in the Log Analytics workspace used by Container Insights. Replace `default` with the namespace where the proxy runs. This query includes proxy component logs as well as access logs:      ```kusto     ContainerLogV2     | where TimeGenerated > ago(45m)-    | where PodName contains "gateway"+    | where PodNamespace == "default"+    | where ContainerName == "istio-proxy"+    | project TimeGenerated, PodNamespace, PodName, LogMessage+    | order by TimeGenerated desc     | take 10     ``` @@ -123,12 +156,13 @@ To enable control plane logs, you create a diagnostic setting and send logs to t      Note -    `$appnet_member_resourceId` should be in the format: `/subscriptions/$subscription/resourceGroups/$appnet_rg/providers/microsoft.applink/applinks/$appnet_name/applinkmembers/$appnet_member_name`.+    Replace the placeholders in the `appnetMemberResourceId` variable with your member's values. Log Analytics doesn't expand Bash environment variables. Resource ID filters use the case-insensitive `=~` operator.      ```kusto+    let appnetMemberResourceId = "/subscriptions/<subscription-id>/resourceGroups/<appnet-resource-group>/providers/Microsoft.AppLink/appLinks/<appnet-name>/appLinkMembers/<appnet-member-name>";     MeshControlPlane     | where TimeGenerated > ago(1h)-    | where _ResourceId == $appnet_member_resourceId // appnet member resourceId+    | where _ResourceId =~ appnetMemberResourceId     ```  ## Log tables and schema@@ -163,23 +197,25 @@ To find the `ContainerLogV2` schema, see [ContainerLogV2 columns](/en-us/azure/a  ### Filter logs by AKS cluster resource ID -- Filter logs based on your AKS managed cluster resource ID in the ContainerLogV2 table using the following query:+- Filter logs based on your AKS managed cluster resource ID in the `ContainerLogV2` table by using the following query. Replace the placeholders in the `aksClusterResourceId` variable with your cluster's values:      ```kusto+    let aksClusterResourceId = "/subscriptions/<subscription-id>/resourceGroups/<aks-resource-group>/providers/Microsoft.ContainerService/managedClusters/<aks-cluster-name>";     ContainerLogV2     | where TimeGenerated > ago(45m)-    | where _ResourceId == $aks_cluster_resource_id // managed cluster resourceId+    | where _ResourceId =~ aksClusterResourceId     ```  ### Filter logs by Azure Kubernetes Application Network member resource ID -- Filter logs based on your Azure Kubernetes Application Network member resource ID in the MeshControlPlane table for control plane logs using the following query:+- Filter logs based on your Azure Kubernetes Application Network member resource ID in the `MeshControlPlane` table for control plane logs by using the following query. Replace the placeholders in the `appnetMemberResourceId` variable with your member's values:      ```kusto+    let appnetMemberResourceId = "/subscriptions/<subscription-id>/resourceGroups/<appnet-resource-group>/providers/Microsoft.AppLink/appLinks/<appnet-name>/appLinkMembers/<appnet-member-name>";     MeshControlPlane     | where TimeGenerated > ago(45m)-    | where Level == "ERROR"-    | where _ResourceId == $appnet_member_resourceId // appnet member resourceId+    | where Level =~ "ERROR"+    | where _ResourceId =~ appnetMemberResourceId     ```      The following screenshot shows an example of the Application Network Istiod errors in Azure Monitor Logs:

@@ -292,12 +292,12 @@ The following table lists the metrics available for the Microsoft.Compute/virtua | **Disk Read Operations/Sec**Disk Read IOPS | `Disk Read Operations/Sec` | No | CountPerSecond | Average | <none> | PT1M | Yes | | **Disk Write Bytes**Bytes written to disk during monitoring period | `Disk Write Bytes` | No | Bytes | Total (Sum) | <none> | PT1M | Yes | | **Disk Write Operations/Sec**Disk Write IOPS | `Disk Write Operations/Sec` | No | CountPerSecond | Average | <none> | PT1M | Yes |-| **Inbound Flows**Inbound Flows are number of current flows in the inbound direction (traffic going into the VM) | `Inbound Flows` | No | Count | Average | <none> | PT1M | Yes |-| **Inbound Flows Maximum Creation Rate**The maximum creation rate of inbound flows (traffic going into the VM) | `Inbound Flows Maximum Creation Rate` | No | CountPerSecond | Average | <none> | PT1M | Yes |+| **Inbound Flows**Inbound Flows are number of current flows in the inbound direction (traffic going into the VM) | `Inbound Flows` | No | Count | Average | <none> | PT1M | No |+| **Inbound Flows Maximum Creation Rate**The maximum creation rate of inbound flows (traffic going into the VM) | `Inbound Flows Maximum Creation Rate` | No | CountPerSecond | Average | <none> | PT1M | No | | **Network In Billable (Deprecated)**The number of billable bytes received on all network interfaces by the Virtual Machine(s) (Incoming Traffic) (Deprecated) | `Network In` | No | Bytes | Total (Sum) | <none> | PT1M | Yes |-| **Network In Total**The number of bytes received on all network interfaces by the Virtual Machine(s) (Incoming Traffic) | `Network In Total` | No | Bytes | Total (Sum) | <none> | PT1M | Yes |+| **Network In Total**The number of bytes received on all network interfaces by the Virtual Machine(s) (Incoming Traffic) | `Network In Total` | No | Bytes | Total (Sum) | <none> | PT1M | No | | **Network Out Billable (Deprecated)**The number of billable bytes out on all network interfaces by the Virtual Machine(s) (Outgoing Traffic) (Deprecated) | `Network Out` | No | Bytes | Total (Sum) | <none> | PT1M | Yes |-| **Network Out Total**The number of bytes out on all network interfaces by the Virtual Machine(s) (Outgoing Traffic) | `Network Out Total` | No | Bytes | Total (Sum) | <none> | PT1M | Yes |+| **Network Out Total**The number of bytes out on all network interfaces by the Virtual Machine(s) (Outgoing Traffic) | `Network Out Total` | No | Bytes | Total (Sum) | <none> | PT1M | No | | **OS Disk Bandwidth Consumed Percentage**Percentage of operating system disk bandwidth consumed per minute. Only available on VM series that support premium storage. | `OS Disk Bandwidth Consumed Percentage` | No | Percent | Average | `LUN` | PT1M | Yes | | **OS Disk IOPS Consumed Percentage**Percentage of operating system disk I/Os consumed per minute. Only available on VM series that support premium storage. | `OS Disk IOPS Consumed Percentage` | No | Percent | Average | `LUN` | PT1M | Yes | | **OS Disk Latency**Average time to complete each IO during monitoring period for OS Disk. Values are in milliseconds. | `OS Disk Latency` | No | Milliseconds | Average | <none> | PT1M | Yes |@@ -312,8 +312,8 @@ The following table lists the metrics available for the Microsoft.Compute/virtua | **OS Disk Used Burst IO Credits Percentage**Percentage of OS Disk burst I/O credits used so far | `OS Disk Used Burst IO Credits Percentage` | No | Percent | Average | `LUN` | PT1M | Yes | | **OS Disk Write Bytes/Sec**Bytes/Sec written to a single disk during monitoring period for OS disk | `OS Disk Write Bytes/sec` | No | BytesPerSecond | Average | <none> | PT1M | Yes | | **OS Disk Write Operations/Sec**Write IOPS from a single disk during monitoring period for OS disk | `OS Disk Write Operations/Sec` | No | CountPerSecond | Average | <none> | PT1M | Yes |-| **Outbound Flows**Outbound Flows are number of current flows in the outbound direction (traffic going out of the VM) | `Outbound Flows` | No | Count | Average | <none> | PT1M | Yes |-| **Outbound Flows Maximum Creation Rate**The maximum creation rate of outbound flows (traffic going out of the VM) | `Outbound Flows Maximum Creation Rate` | No | CountPerSecond | Average | <none> | PT1M | Yes |+| **Outbound Flows**Outbound Flows are number of current flows in the outbound direction (traffic going out of the VM) | `Outbound Flows` | No | Count | Average | <none> | PT1M | No |+| **Outbound Flows Maximum Creation Rate**The maximum creation rate of outbound flows (traffic going out of the VM) | `Outbound Flows Maximum Creation Rate` | No | CountPerSecond | Average | <none> | PT1M | No | | **Percentage CPU**The percentage of allocated compute units that are currently in use by the Virtual Machine(s) | `Percentage CPU` | No | Percent | Average | <none> | PT1M | Yes | | **Premium Data Disk Cache Read Hit**Premium Data Disk Cache Read Hit | `Premium Data Disk Cache Read Hit` | No | Percent | Average | `LUN` | PT1M | Yes | | **Premium Data Disk Cache Read Miss**Premium Data Disk Cache Read Miss | `Premium Data Disk Cache Read Miss` | No | Percent | Average | `LUN` | PT1M | Yes |@@ -334,7 +334,6 @@ The following table lists the metrics available for the Microsoft.Compute/virtua | **VM Uncached Bandwidth Consumed Percentage**Percentage of uncached disk bandwidth consumed by the VM. Only available on VM series that support premium storage. | `VM Uncached Bandwidth Consumed Percentage` | No | Percent | Average | <none> | PT1M | Yes | | **VM Uncached IOPS Consumed Percentage**Percentage of uncached disk IOPS consumed by the VM. Only available on VM series that support premium storage. | `VM Uncached IOPS Consumed Percentage` | No | Percent | Average | <none> | PT1M | Yes | | **VM Availability Metric**Measure of availability of virtual machines over time. | `VmAvailabilityMetric` | No | Count | Average, Minimum, Maximum | `Context` | PT1M | Yes |-| **VM Time Offset**Time offset (drift) of the VM's clock compared to the Azure time source, in nanoseconds. | `VMTimeOffset` | No | Count | Average, Minimum, Maximum | `CompliantFlag` | PT1M | Yes |  ### Supported metrics for Microsoft.Compute/virtualmachineScaleSets @@ -378,12 +377,12 @@ The following table lists the metrics available for the Microsoft.Compute/virtua | **Disk Read Operations/Sec**Disk Read IOPS | `Disk Read Operations/Sec` | No | CountPerSecond | Average | `VMName` | PT1M | Yes | | **Disk Write Bytes**Bytes written to disk during monitoring period | `Disk Write Bytes` | No | Bytes | Total (Sum) | `VMName` | PT1M | Yes | | **Disk Write Operations/Sec**Disk Write IOPS | `Disk Write Operations/Sec` | No | CountPerSecond | Average | `VMName` | PT1M | Yes |-| **Inbound Flows**Inbound Flows are number of current flows in the inbound direction (traffic going into the VM) | `Inbound Flows` | No | Count | Average | `VMName` | PT1M | Yes |-| **Inbound Flows Maximum Creation Rate**The maximum creation rate of inbound flows (traffic going into the VM) | `Inbound Flows Maximum Creation Rate` | No | CountPerSecond | Average | `VMName` | PT1M | Yes |+| **Inbound Flows**Inbound Flows are number of current flows in the inbound direction (traffic going into the VM) | `Inbound Flows` | No | Count | Average | `VMName` | PT1M | No |+| **Inbound Flows Maximum Creation Rate**The maximum creation rate of inbound flows (traffic going into the VM) | `Inbound Flows Maximum Creation Rate` | No | CountPerSecond | Average | `VMName` | PT1M | No | | **Network In Billable (Deprecated)**The number of billable bytes received on all network interfaces by the Virtual Machine(s) (Incoming Traffic) (Deprecated) | `Network In` | No | Bytes | Total (Sum) | `VMName` | PT1M | Yes |-| **Network In Total**The number of bytes received on all network interfaces by the Virtual Machine(s) (Incoming Traffic) | `Network In Total` | No | Bytes | Total (Sum) | `VMName` | PT1M | Yes |+| **Network In Total**The number of bytes received on all network interfaces by the Virtual Machine(s) (Incoming Traffic) | `Network In Total` | No | Bytes | Total (Sum) | `VMName` | PT1M | No | | **Network Out Billable (Deprecated)**The number of billable bytes out on all network interfaces by the Virtual Machine(s) (Outgoing Traffic) (Deprecated) | `Network Out` | No | Bytes | Total (Sum) | `VMName` | PT1M | Yes |-| **Network Out Total**The number of bytes out on all network interfaces by the Virtual Machine(s) (Outgoing Traffic) | `Network Out Total` | No | Bytes | Total (Sum) | `VMName` | PT1M | Yes |+| **Network Out Total**The number of bytes out on all network interfaces by the Virtual Machine(s) (Outgoing Traffic) | `Network Out Total` | No | Bytes | Total (Sum) | `VMName` | PT1M | No | | **OS Disk Bandwidth Consumed Percentage**Percentage of operating system disk bandwidth consumed per minute. Only available on VM series that support premium storage. | `OS Disk Bandwidth Consumed Percentage` | No | Percent | Average | `LUN`, `VMName` | PT1M | Yes | | **OS Disk IOPS Consumed Percentage**Percentage of operating system disk I/Os consumed per minute. Only available on VM series that support premium storage. | `OS Disk IOPS Consumed Percentage` | No | Percent | Average | `LUN`, `VMName` | PT1M | Yes | | **OS Disk Latency**Average time to complete each IO during monitoring period for OS Disk. Values are in milliseconds. | `OS Disk Latency` | No | Milliseconds | Average | `VMName` | PT1M | Yes |@@ -398,8 +397,8 @@ The following table lists the metrics available for the Microsoft.Compute/virtua | **OS Disk Used Burst IO Credits Percentage**Percentage of OS Disk burst I/O credits used so far | `OS Disk Used Burst IO Credits Percentage` | No | Percent | Average | `LUN`, `VMName` | PT1M | Yes | | **OS Disk Write Bytes/Sec**Bytes/Sec written to a single disk during monitoring period for OS disk | `OS Disk Write Bytes/sec` | No | BytesPerSecond | Average | `VMName` | PT1M | Yes | | **OS Disk Write Operations/Sec**Write IOPS from a single disk during monitoring period for OS disk | `OS Disk Write Operations/Sec` | No | CountPerSecond | Average | `VMName` | PT1M | Yes |-| **Outbound Flows**Outbound Flows are number of current flows in the outbound direction (traffic going out of the VM) | `Outbound Flows` | No | Count | Average | `VMName` | PT1M | Yes |-| **Outbound Flows Maximum Creation Rate**The maximum creation rate of outbound flows (traffic going out of the VM) | `Outbound Flows Maximum Creation Rate` | No | CountPerSecond | Average | `VMName` | PT1M | Yes |+| **Outbound Flows**Outbound Flows are number of current flows in the outbound direction (traffic going out of the VM) | `Outbound Flows` | No | Count | Average | `VMName` | PT1M | No |+| **Outbound Flows Maximum Creation Rate**The maximum creation rate of outbound flows (traffic going out of the VM) | `Outbound Flows Maximum Creation Rate` | No | CountPerSecond | Average | `VMName` | PT1M | No | | **Percentage CPU**The percentage of allocated compute units that are currently in use by the Virtual Machine(s) | `Percentage CPU` | No | Percent | Average | `VMName` | PT1M | Yes | | **Premium Data Disk Cache Read Hit**Premium Data Disk Cache Read Hit | `Premium Data Disk Cache Read Hit` | No | Percent | Average | `LUN`, `VMName` | PT1M | Yes | | **Premium Data Disk Cache Read Miss**Premium Data Disk Cache Read Miss | `Premium Data Disk Cache Read Miss` | No | Percent | Average | `LUN`, `VMName` | PT1M | Yes |@@ -463,12 +462,12 @@ The following table lists the metrics available for the Microsoft.Compute/virtua | **Disk Read Operations/Sec**Disk Read IOPS | `Disk Read Operations/Sec` | No | CountPerSecond | Average | <none> | PT1M | Yes | | **Disk Write Bytes**Bytes written to disk during monitoring period | `Disk Write Bytes` | No | Bytes | Total (Sum) | <none> | PT1M | Yes | | **Disk Write Operations/Sec**Disk Write IOPS | `Disk Write Operations/Sec` | No | CountPerSecond | Average | <none> | PT1M | Yes |-| **Inbound Flows**Inbound Flows are number of current flows in the inbound direction (traffic going into the VM) | `Inbound Flows` | No | Count | Average | <none> | PT1M | Yes |-| **Inbound Flows Maximum Creation Rate**The maximum creation rate of inbound flows (traffic going into the VM) | `Inbound Flows Maximum Creation Rate` | No | CountPerSecond | Average | <none> | PT1M | Yes |+| **Inbound Flows**Inbound Flows are number of current flows in the inbound direction (traffic going into the VM) | `Inbound Flows` | No | Count | Average | <none> | PT1M | No |+| **Inbound Flows Maximum Creation Rate**The maximum creation rate of inbound flows (traffic going into the VM) | `Inbound Flows Maximum Creation Rate` | No | CountPerSecond | Average | <none> | PT1M | No | | **Network In Billable (Deprecated)**The number of billable bytes received on all network interfaces by the Virtual Machine(s) (Incoming Traffic) (Deprecated) | `Network In` | No | Bytes | Total (Sum) | <none> | PT1M | Yes |-| **Network In Total**The number of bytes received on all network interfaces by the Virtual Machine(s) (Incoming Traffic) | `Network In Total` | No | Bytes | Total (Sum) | <none> | PT1M | Yes |+| **Network In Total**The number of bytes received on all network interfaces by the Virtual Machine(s) (Incoming Traffic) | `Network In Total` | No | Bytes | Total (Sum) | <none> | PT1M | No | | **Network Out Billable (Deprecated)**The number of billable bytes out on all network interfaces by the Virtual Machine(s) (Outgoing Traffic) (Deprecated) | `Network Out` | No | Bytes | Total (Sum) | <none> | PT1M | Yes |-| **Network Out Total**The number of bytes out on all network interfaces by the Virtual Machine(s) (Outgoing Traffic) | `Network Out Total` | No | Bytes | Total (Sum) | <none> | PT1M | Yes |+| **Network Out Total**The number of bytes out on all network interfaces by the Virtual Machine(s) (Outgoing Traffic) | `Network Out Total` | No | Bytes | Total (Sum) | <none> | PT1M | No | | **OS Disk Bandwidth Consumed Percentage**Percentage of operating system disk bandwidth consumed per minute. Only available on VM series that support premium storage. | `OS Disk Bandwidth Consumed Percentage` | No | Percent | Average | `LUN` | PT1M | Yes | | **OS Disk IOPS Consumed Percentage**Percentage of operating system disk I/Os consumed per minute. Only available on VM series that support premium storage. | `OS Disk IOPS Consumed Percentage` | No | Percent | Average | `LUN` | PT1M | Yes | | **OS Disk Latency**Average time to complete each IO during monitoring period for OS Disk. Values are in milliseconds. | `OS Disk Latency` | No | Milliseconds | Average | <none> | PT1M | Yes |@@ -483,8 +482,8 @@ The following table lists the metrics available for the Microsoft.Compute/virtua | **OS Disk Used Burst IO Credits Percentage**Percentage of OS Disk burst I/O credits used so far | `OS Disk Used Burst IO Credits Percentage` | No | Percent | Average | `LUN` | PT1M | Yes | | **OS Disk Write Bytes/Sec**Bytes/Sec written to a single disk during monitoring period for OS disk | `OS Disk Write Bytes/sec` | No | BytesPerSecond | Average | <none> | PT1M | Yes | | **OS Disk Write Operations/Sec**Write IOPS from a single disk during monitoring period for OS disk | `OS Disk Write Operations/Sec` | No | CountPerSecond | Average | <none> | PT1M | Yes |-| **Outbound Flows**Outbound Flows are number of current flows in the outbound direction (traffic going out of the VM) | `Outbound Flows` | No | Count | Average | <none> | PT1M | Yes |-| **Outbound Flows Maximum Creation Rate**The maximum creation rate of outbound flows (traffic going out of the VM) | `Outbound Flows Maximum Creation Rate` | No | CountPerSecond | Average | <none> | PT1M | Yes |+| **Outbound Flows**Outbound Flows are number of current flows in the outbound direction (traffic going out of the VM) | `Outbound Flows` | No | Count | Average | <none> | PT1M | No |+| **Outbound Flows Maximum Creation Rate**The maximum creation rate of outbound flows (traffic going out of the VM) | `Outbound Flows Maximum Creation Rate` | No | CountPerSecond | Average | <none> | PT1M | No | | **Percentage CPU**The percentage of allocated compute units that are currently in use by the Virtual Machine(s) | `Percentage CPU` | No | Percent | Average | <none> | PT1M | Yes | | **Premium Data Disk Cache Read Hit**Premium Data Disk Cache Read Hit | `Premium Data Disk Cache Read Hit` | No | Percent | Average | `LUN` | PT1M | Yes | | **Premium Data Disk Cache Read Miss**Premium Data Disk Cache Read Miss | `Premium Data Disk Cache Read Miss` | No | Percent | Average | `LUN` | PT1M | Yes |

@@ -15,7 +15,9 @@ Control plane metrics aren't currently supported. However, data plane metrics ar  ## Data plane metrics -Data plane metrics include metrics from your workloads/applications and the `appnet-system` namespace. Azure Kubernetes Application Network currently only provides support for **data plane metrics**.+Data plane metrics include metrics from your workloads/applications and the `applink-system` namespace. Azure Kubernetes Application Network currently only provides support for **data plane metrics**.++To collect request and response metrics, route application traffic through a waypoint. Enrolling workloads in ambient mode alone doesn't produce L7 metrics for traffic that only passes through ztunnel.  ## Configure data plane metrics 

@@ -9,6 +9,35 @@ Use this guidance for Azure Kubernetes Service (AKS) on bare metal on Azure Linu 3. Confirm that the host can reach the required outbound endpoints. 4. Open the resource group **Deployments** page and review the failed deployment operation. +## NodeRelay terminal failure++Cluster creation fails with the following error:++```text+Error: NodeRelay terminal failure (reason=NodeRelayUpdate_TerminalFailure): terminal error: hybrid connectivity v1 is no longer supported (Code: CapeActivityFailAtSetupNodeRelay)+```++**Cause:** The error message is misleading. The subscription isn't registered for the `Microsoft.HybridConnectivity/hiddenPreviewAccess` preview feature, so Azure Resource Manager rejects the Hybrid Connectivity calls with `NoRegisteredProviderFound`. This error isn't an API version problem.++**Mitigation:**++1. Register the preview feature. Replace `<subscription-id>` with your subscription ID.++    ```azurecli+    az feature register --namespace Microsoft.HybridConnectivity --name hiddenPreviewAccess --subscription <subscription-id>+    ```+2. Check the feature registration state. Wait until the command returns `Registered` before you continue.++    ```azurecli+    az feature show --namespace Microsoft.HybridConnectivity --name hiddenPreviewAccess --subscription <subscription-id> --query properties.state -o tsv+    ```+3. Register the resource provider.++    ```azurecli+    az provider register --namespace Microsoft.HybridConnectivity --subscription <subscription-id>+    ```+4. Retry the cluster creation.+ ## Upgrade remains in progress  Wait up to 30 minutes, then confirm that the host remains connected to Azure Arc. Because the preview uses a single-node cluster, workloads are unavailable while the node is cordoned, drained, and upgraded.

@@ -2,7 +2,7 @@  ## Summary -This article discusses how to troubleshoot issues that occur after you restrict egress traffic for cluster nodes in Microsoft Azure Kubernetes Service (AKS).+This article helps you troubleshoot errors that occur after you restrict egress traffic for cluster nodes in Azure Kubernetes Service (AKS). Use it to identify conflicting outbound network and FQDN rules and restore kubectl commands, cluster creation, and node pool scaling.  ## Symptoms @@ -10,24 +10,24 @@ Certain commands of the [kubectl](https://kubernetes.io/docs/reference/kubectl/)  ## Cause -When you restrict egress traffic from an AKS cluster, your settings must comply with required Outbound network and FQDN (fully qualified domain names) rules for AKS clusters. If your settings are in conflict with any of these rules, the egress traffic restriction issues occur.+When you restrict egress traffic from an AKS cluster, your settings must comply with required outbound network and fully qualified domain name (FQDN) rules for AKS clusters. If your settings conflict with any of these rules, the egress traffic restriction causes problems.  ## Solution -Verify that your configuration doesn't conflict with any of the [required Outbound network and FQDN (fully qualified domain names) rules for AKS clusters](/en-us/azure/aks/outbound-rules-control-egress) for the following items:+Verify that your configuration doesn't conflict with any of the [required Outbound network and FQDN rules for AKS clusters](/en-us/azure/aks/outbound-rules-control-egress) for the following items:  - Outbound ports - Network rules - FQDNs - Application rules -Check for conflicts with the rules that might occur in the NSG (network security group), firewall, or appliance that AKS traffic passes through according to the configuration.+Check for conflicts with the rules that might occur in the network security group (NSG), firewall, or appliance that AKS traffic passes through according to the configuration.  Note -The AKS outbound dependencies are almost entirely defined by using FQDNs. These FQDNs don't have static addresses behind them. The lack of static addresses means that you can't use NSGs to restrict outbound traffic from an AKS cluster. Additionally, scenarios that allow only IPs that are obtained from required FQDNs after all deny in NSG are not enough to restrict outbound traffic. Because the IPs are not static, issues might occur later.+The AKS outbound dependencies are almost entirely defined by using FQDNs. These FQDNs don't have static addresses behind them. The lack of static addresses means that you can't use NSGs to restrict outbound traffic from an AKS cluster. Additionally, scenarios that allow only IPs that are obtained from required FQDNs after all deny in NSG aren't enough to restrict outbound traffic. Because the IPs aren't static, issues might occur later. -## More information+## References  - [Limit network traffic with Azure Firewall in AKS clusters](/en-us/azure/aks/limit-egress-traffic) 

@@ -7,47 +7,53 @@ AKS preview features are available on a self-service, opt-in basis. Previews are - [AKS support policies](/en-us/azure/aks/support-policies) - [Azure support FAQ](/en-us/azure/aks/faq) -Azure Kubernetes Application Network provides a fully managed, [ambient-based](https://istio.io/latest/docs/ambient/) service network solution for Azure Kubernetes Service (AKS). It enables secure, policy-driven communication between services without requiring sidecars or changes to your applications. Azure manages both the control plane and data plane, so you can focus on your workloads instead of operating and maintaining service-network infrastructure.+Azure Kubernetes Application Network is a fully managed, [ambient-based](https://istio.io/latest/docs/ambient/) service network for Azure Kubernetes Service (AKS). It enables secure, policy-driven service-to-service communication without sidecars or application changes. Azure manages the control plane and data plane so you can focus on your workloads. -When you create an Application Network resource, a dedicated, fully managed control and data plane is provisioned for service-to-service communication. You can then connect one or more AKS clusters to the Application Network resource to onboard them to the service network. Once connected, Application Network manages configuration, service identity, and version upgrades, providing consistent security and traffic policies across workloads with minimal configuration and operational overhead.+## Watch an overview -Important--Azure Kubernetes Application Network manages only service network-related components and policies. It doesn't manage or upgrade your AKS cluster, Kubernetes version, or node images.--Important+Watch the AKS Me Anything conversation with Jorge and Jack for an introduction to Application Network. -Azure Kubernetes Application Network doesn't store any customer data.+You can also [watch the interview on YouTube](https://www.youtube.com/watch?v=Rg21m1YTdBA). To try it yourself, follow the [get-started guide](get-started) and check the [currently supported versions](supported-versions).  ## Core capabilities -Application Network delivers a streamlined approach to zero-trust networking in AKS. Key capabilities include:+Application Network provides the following security and lifecycle capabilities: -- **Ambient architecture**: Provides mesh functionality without sidecars, reducing resource usage and simplifying operations. - **Automatic mutual TLS (mTLS)**: Secures service-to-service traffic by default, supporting both encrypted and unencrypted connections automatically. - **Managed certificate authority**: Issues and rotates SPIFFE compliant workload certificates automatically.-- **Version and upgrade control**: Select minor versions manually or use release channels for automated upgrades. Rollout and patching are always managed by Application Network.-- **Integrated lifecycle management**: Operates as part of the AKS platform with control plane operations fully managed by Azure.+- **Version and upgrade control**: Select minor versions manually or use release channels for automated upgrades. Application Network always manages rollout and patching.  ## Architecture overview  Application Network uses an **ambient mesh model** composed of lightweight node-level and optional L7 proxies: -- **Node-level proxies** secure all L4 service-to-service traffic in the cluster through mTLS, eliminating the need for sidecars.+- **Node-level proxies** secure all L4 service-to-service traffic in the cluster through mTLS. - **Optional L7 proxies** provide advanced routing and policy enforcement at the service level. -This design separates the data plane from individual workloads, improving efficiency, and simplifying adoption. For more details, see [Azure Kubernetes Application Network architecture](architecture).+Separating the data plane from individual workloads reduces resource usage and simplifies operations. For more details, see [Azure Kubernetes Application Network architecture](architecture).  ## Supported scenarios -Application Network supports single-cluster AKS deployments running **Linux-based node pools**. It's ideal for applications that require secure, in-cluster communication and simplified mesh management without managing sidecars or control plane components.+Create an Application Network resource to provision a dedicated control plane and data plane, then connect one or more AKS clusters as members. Application Network manages configuration and service identity for communication within and across member clusters.++For multi-cluster deployments, choose a topology that matches your network reachability. Members in the same mesh network require direct pod-IP connectivity; members in different networks communicate through east-west gateways. Application Network doesn't create the underlying connectivity for either path. For configuration steps, see [Configure multi-cluster connectivity](multi-cluster-connectivity).++Joining a cluster doesn't automatically enroll all application workloads or export every service. Enroll the intended workloads in ambient mode and configure global services as described in the [traffic management walkthrough](traffic-management-use-cases).  ## Limitations +Important++Azure Kubernetes Application Network manages only service network-related components and policies. It doesn't manage or upgrade your AKS cluster, Kubernetes version, or node images.++Important++Azure Kubernetes Application Network doesn't store any customer data.+ Application Network currently has the following limitations:  - Application Network requires AKS version 1.30 or higher. Refer to [Application Network supported versions](supported-versions) for each AKS version's specific compatibility information.-- AKS [private clusters](../aks/private-clusters) aren't currently supported.+- AKS [private clusters](../aks/private-clusters) require service enablement and `Microsoft.AppLink/PrivatePath` registration. To check eligibility and subnet requirements, see [Connect private AKS clusters](private-connectivity). Public AKS clusters can't use the private-connect path. Ordinary public-cluster onboarding stays the same. - Linux only. Windows node pools aren't supported. - Switching between upgrade modes `SelfManaged` and `FullyManaged` isn't supported. - Currently supported only in the following regions:

@@ -13,7 +13,7 @@ The set of plugins in the catalog can change as publishers add, update, deprecat ## Prerequisites  - Confirm that your organization allows third-party plugins and external integrations.-- [Install or update to AKS desktop v0.10.0 or later](https://github.com/Azure/aks-desktop#installation).+- [Install or update to AKS desktop v0.10.0 or later](aks-desktop-overview#install-aks-desktop). - [Connect AKS desktop to the Kubernetes cluster](aks-desktop-install-cluster-setup) where you want to use the plugin. - Review the plugin notes, required cluster components, supported versions, network requirements, and publisher before installation. - Confirm that your Kubernetes identity has only the permissions required for the plugin workflow. For more information, see [Set up permissions and role-based access control (RBAC) in AKS desktop](aks-desktop-permissions).

@@ -1,14 +1,16 @@-# InsufficientSubnetSize error code - Azure | Microsoft Learn+# Troubleshoot InsufficientSubnetSize error code in AKS - Azure | Microsoft Learn  ## Summary -Use this article to fix the `InsufficientSubnetSize` error when deploying an Azure Kubernetes Service (AKS) cluster with advanced networking. You learn why it happens and how to restore enough subnet IP capacity for successful cluster and node pool operations across Kubenet, Azure CNI, and Azure CNI Overlay.+Use this article to troubleshoot the `InsufficientSubnetSize` error when you deploy an Azure Kubernetes Service (AKS) cluster with advanced networking. Learn why the error occurs and how to restore subnet IP capacity for cluster and node pool operations in Kubenet, Azure CNI, and Azure CNI Overlay.  ## Symptoms  The `InsufficientSubnetSize` error occurs during any of the following operations. You also encounter this error in [AKS Diagnostics](/en-us/azure/aks/concepts-diagnostics), which proactively discovers issues such as an insufficient subnet size. -#### Operation 1: Scaling an AKS cluster or an AKS node pool+### Operation 1: Scale an AKS cluster or node pool++The following table summarizes the symptoms for different cluster types when the `InsufficientSubnetSize` error occurs.  | Cluster type | Symptom:The number of free IP addresses in the subnet is*less than*... | | --- | --- |@@ -16,7 +18,9 @@ The `InsufficientSubnetSize` error occurs during any of the following operations | Azure CNI | The number of new nodes that you request **times** the node pool value in the `--max-pod` parameter. | | Azure CNI Overlay | The number of new nodes that you request.(In the node pools that use the autoscaler, the number of nodes is the value in the `--max-count` parameter.) | -#### Operation 2: Upgrading an AKS cluster or an AKS node pool+### Operation 2: Upgrade an AKS cluster or node pool++The following table summarizes the symptoms for different cluster types when the `InsufficientSubnetSize` error occurs during an upgrade.  | Cluster type | Symptom:The number of free IP addresses in the subnet is*less than*... | | --- | --- |@@ -26,7 +30,9 @@ The `InsufficientSubnetSize` error occurs during any of the following operations  By default, an AKS cluster sets a maximum surge (upgrade buffer) value of one (1). However, you can customize this upgrade behavior by setting the maximum surge value of a node pool. This action increases the number of available IP addresses that are necessary to complete an upgrade. -#### Operation 3: Creating an AKS cluster or adding an AKS node pool+### Operation 3: Create an AKS cluster or add a node pool++The following table summarizes the symptoms for different cluster types when the `InsufficientSubnetSize` error occurs during cluster creation or node pool addition.  | Cluster type | Symptom:The number of free IP addresses in the subnet is*less than*... | | --- | --- |@@ -36,7 +42,7 @@ By default, an AKS cluster sets a maximum surge (upgrade buffer) value of one (1  ## Cause -A subnet that's in use for a cluster no longer has available IP addresses within its Classless Inter-Domain Routing (CIDR) address space for successful resource assignment.+A subnet that a cluster uses no longer has available IP addresses within its Classless Inter-Domain Routing (CIDR) address space for resource assignment.  | Cluster type | Requirement | | --- | --- |@@ -44,11 +50,11 @@ A subnet that's in use for a cluster no longer has available IP addresses within | Azure CNI | Sufficient IP space for each *node and pod* in the cluster | | Azure CNI Overlay | Sufficient IP space for each *node* in the cluster | -Read more about the [design of Azure CNI to assign IP addresses to pods](/en-us/azure/aks/configure-azure-cni#plan-ip-addressing-for-your-cluster).+To learn more, see [design of Azure CNI to assign IP addresses to pods](/en-us/azure/aks/configure-azure-cni#plan-ip-addressing-for-your-cluster). -AKS considers all node pools in the cluster and their respective subnet. If you upgrade a node pool, AKS considers the IP requirements of all node pools in the cluster. For example, if you upgrade node pool "pool-A" in subnet "subnet-A", and there's another subnet "subnet-B" linked to another node pool "pool-B" facing IP exhaustion, then the following error might happen:+AKS considers all node pools in the cluster and their respective subnets. If you upgrade a node pool, AKS considers the IP requirements of all node pools in the cluster. For example, if you upgrade node pool `pool-A` in subnet `subnet-A`, and another subnet `subnet-B` links to another node pool `pool-B` that's facing IP exhaustion, the following error might occur. -`Code: InsufficientSubnetSize; Message: Pre-allocated IPs <number> exceeds IPs available <number> in Subnet Cidr <CIDR>, Subnet Name <subnet-name>. If Autoscaler is enabled, the max-count from each nodepool is counted towards this total, which means that pre-allocated IPs count represents a theoretical max value, not the actual number of IPs requested. See http://aka.ms/aks/insufficientsubnetsize for troubleshooting steps. Surge nodes would also consume the subnet IP space, consider use smaller maxSurge or use maxUnavailable (aka.ms/aks/maxUnavailable).`+`Code: InsufficientSubnetSize; Message: Pre-allocated IPs <number> exceeds IPs available <number> in Subnet Cidr <CIDR>, Subnet Name <subnet-name>. If Autoscaler is enabled, the max-count from each nodepool is counted towards this total, which means that pre-allocated IPs count represents a theoretical max value, not the actual number of IPs requested. See http://aka.ms/aks/insufficientsubnetsize for troubleshooting steps. Surge nodes also consume the subnet IP space. Consider using smaller maxSurge or use maxUnavailable (aka.ms/aks/maxUnavailable).`  ## Solution 

@@ -2,45 +2,323 @@  ## Summary -This article discusses how to identify and resolve the `K8SAPIServerConnFailVMExtensionError` error (also known as error code ERR\_K8S\_API\_SERVER\_CONN\_FAIL, error number 51) that occurs when you try to start or create and deploy a Microsoft Azure Kubernetes Service (AKS) cluster.+This article describes how to identify and resolve the `K8SAPIServerConnFailVMExtensionError` error (also known as error code `ERR_K8S_API_SERVER_CONN_FAIL`, error number 51). This error might occur when you create, start, scale, or upgrade a Microsoft Azure Kubernetes Service (AKS) cluster or node pool.++When AKS provisions a Linux node, the node bootstrap script (delivered through the `vmssCSE` Custom Script Extension) checks whether the node can reach the cluster's Kubernetes API server on TCP port 443 before it starts the kubelet. If the node can resolve the API server address but can't establish a connection to it, the script exits with code 51, the node isn't registered in the cluster, and the operation fails.++In most cases, this error is caused by a network configuration in the customer's environment (such as a firewall, network security group, route table, proxy, or authorized IP range) that blocks or drops traffic from the node subnet to the API server.++Note++- This article applies to Linux nodes. If the node can't *resolve* the API server name, you receive a different error. For more information, see [Troubleshoot the K8SAPIServerDNSLookupFailVMExtensionError error code (52)](error-code-k8sapiserverdnslookupfailvmextensionerror). If the node can't reach required outbound endpoints such as `mcr.microsoft.com`, see [Troubleshoot the OutboundConnFailVMExtensionError error code (50)](error-code-outboundconnfailvmextensionerror).+- If your cluster uses [API Server VNet Integration](/en-us/azure/aks/api-server-vnet-integration), a blocked connection to the API server usually causes a different error. See Clusters that use API Server VNet Integration.  ## Prerequisites -- The [Netcat](https://linuxcommandlibrary.com/man/netcat) (nc) command-line tool+- [Azure CLI](/en-us/cli/azure/install-azure-cli), signed in to the subscription that contains the AKS cluster.+- Permissions to read the AKS cluster and to run commands on the virtual machine scale sets (VMSS) or virtual machines (VMs) in the cluster's node resource group (for example, the [Virtual Machine Contributor](/en-us/azure/role-based-access-control/built-in-roles/compute#virtual-machine-contributor) role).++The connectivity tests in this article run on the AKS nodes by using [Azure Run Command](/en-us/azure/virtual-machines/linux/run-command). They use the [curl](https://curl.se/docs/manpage.html) command-line tool (or [Netcat](https://linuxcommandlibrary.com/man/netcat) (`nc`) for clusters that use API Server VNet Integration). These tools are already included in AKS Linux node images, so you don't have to install them on your local computer or on the nodes.  ## Symptoms -When you try to start or create an AKS cluster, you receive the following error message:-->-> Unable to establish connection from agents to Kubernetes API server, please see https://aka.ms/aks-required-ports-and-addresses for more information.->-> Details: Code="VMExtensionProvisioningError"->-> Message="VM has reported a failure when processing extension 'vmssCSE'.->-> Error message: "**Enable failed: failed to execute command: command terminated with exit status=51**\n[stdout]\n{->-> "ExitCode": "51",->-> "Output": "Thu Oct 14 18:07:37 UTC 2021,aks-nodepool1-18315663-vmss000000\nConnection to+When you try to create, start, scale, or upgrade an AKS cluster or node pool, the operation fails and you receive an error message that resembles the following example (identifiers are replaced with placeholders, and repeated progress output is removed):++```output+Code: K8SAPIServerConnFailVMExtensionError+Message: Create or update VMSS <vmss-resource-id> failed. Multiple VM extensions failed to be provisioned on the VM. The Extensions failed due to the errors:++[Extension Name: '<vmss-name>-AKSLinuxBilling'+Error Message: The handler for VM extension type 'Microsoft.AKS.Compute.AKS.Linux.Billing' has reported terminal failure for VM extension '<vmss-name>-AKSLinuxBilling' with error message: 'Dependent Extension Microsoft.Azure.Extensions.CustomScript did not succeed. Status was error'.]++[Extension Name: 'vmssCSE'+Error Message: VM has reported a user failure when processing extension 'vmssCSE'. Please correct the error and try again. (publisher 'Microsoft.Azure.Extensions' and type 'CustomScript'). Error code: '2'. Error message: 'failed to execute command: command terminated with exit status=51+[stdout]+{ "ExitCode": "51", "Output": "...+* connect to <api-server-ip> port 443 from <node-ip> port <port> failed: Connection timed out+* Failed to connect to <api-server-fqdn> port 443 after <n> ms: Couldn't connect to server+curl: (28) Failed to connect to <api-server-fqdn> port 443 after <n> ms: Couldn't connect to server+...+API server connection check code: 51++ exit 51", ... }+```++Key indicators of this error:++- The `vmssCSE` extension fails with `exit status=51` and the output contains `API server connection check code: 51`.+- The output shows a failed connection to the API server on port 443 (for example, `Connection timed out`, `Connection refused`, or `curl: (28) Failed to connect`).+- Other AKS extensions on the same VM (such as `AKSLinuxBilling`) might also fail with `Dependent Extension Microsoft.Azure.Extensions.CustomScript did not succeed`. This error is a side effect of the `vmssCSE` failure, not a separate problem.++In some API versions or tools, the message might begin with `Unable to establish connection from agents to Kubernetes API server, please see https://aka.ms/aks-required-ports-and-addresses for more information.` followed by `Code="VMExtensionProvisioningError"`.  ## Cause -Your cluster nodes can't connect to your cluster API server pod.+During node provisioning, the bootstrap script validates connectivity from the node to the API server before it starts the kubelet:++| Cluster configuration | Address that the node uses | Check performed by the bootstrap script |+| --- | --- | --- |+| Public cluster | Public API server FQDN (`<dns-prefix>-<hash>.hcp.<region>.azmk8s.io`) | DNS lookup, then `curl https://<api-server-fqdn>:443` (with retries) |+| Private cluster | Private API server FQDN (for example, `<dns-prefix>-<hash>.<guid>.privatelink.<region>.azmk8s.io`) that resolves to the private endpoint IP | DNS lookup, then `curl https://<api-server-fqdn>:443` (with retries) |++If the DNS lookup fails, the script exits with code 52 or 53. If the DNS lookup succeeds but the TCP/TLS connection to port 443 can't be established within the retry period, the script exits with code 51.++The node must be able to reach the API server for AKS to work. This requirement is documented in [Outbound network and FQDN rules for AKS clusters](/en-us/azure/aks/outbound-rules-control-egress#azure-global-required-fqdn--application-rules): `*.hcp.<location>.azmk8s.io` on HTTPS port 443 is required for node-to-API server communication.++Common reasons why the connection fails include:++- **Egress filtering blocks the API server.** A firewall, network virtual appliance (NVA), or proxy on the egress path doesn't allow TCP 443 to the API server FQDN (`*.hcp.<region>.azmk8s.io`) or to the API server public IP address.+- **TLS inspection or a proxy interferes with the connection.** A device that intercepts, decrypts, or rewrites TLS traffic (including the Application-Layer Protocol Negotiation (ALPN) extension) to the API server breaks the connection.+- **Network security group (NSG) rules deny the traffic.** An outbound rule on the node subnet or NIC blocks TCP 443 from the nodes to the API server.+- **User-defined routes (UDRs) send traffic to the wrong next hop.** A route table on the node subnet sends API server traffic to an NVA that drops it, routes it asymmetrically, or has no path to the destination.+- **API server authorized IP ranges don't include the cluster's egress IP.** If [authorized IP ranges](/en-us/azure/aks/api-server-authorized-ip-ranges) are enabled and the nodes egress through an IP address that isn't in the list (for example, the public IP of an Azure Firewall, NAT gateway, or other NVA), the API server rejects the connection.+- **(Private clusters) The private FQDN resolves to the wrong IP address.** Custom DNS servers or a manually managed private DNS zone return an outdated or incorrect IP address, so the node connects to an address where the API server isn't listening.+- **(Private clusters) The node can't reach the private endpoint.** For example, NSG rules or UDRs block traffic to the private endpoint IP, or the private endpoint is in a peered VNet that isn't reachable from the node subnet.  ## Solution -Run a Netcat command to verify that your nodes can resolve the cluster's fully qualified domain name (FQDN):+### Step 1: Gather cluster information++1. Get the API server endpoints and the network configuration of the cluster by running the [az aks show](/en-us/cli/azure/aks#az-aks-show) command:++    ```azurecli+    az aks show \+        --resource-group <resource-group> \+        --name <cluster-name> \+        --query "{fqdn:fqdn, privateFqdn:privateFqdn, nodeResourceGroup:nodeResourceGroup, outboundType:networkProfile.outboundType, privateCluster:apiServerAccessProfile.enablePrivateCluster, vnetIntegration:apiServerAccessProfile.enableVnetIntegration, authorizedIpRanges:apiServerAccessProfile.authorizedIpRanges}" \+        --output json+    ```++    Use the output to determine the API server FQDN that the nodes use:++    - **Public cluster**: Use the `fqdn` value.+    - **Private cluster**: Use the `privateFqdn` value. If `privateFqdn` is empty (for example, when the private DNS zone is set to `none`), use the `fqdn` value.++    If `vnetIntegration` is `true`, see Clusters that use API Server VNet Integration instead.++    The error message usually contains the API server FQDN that the node tried to reach (for example, `Failed to connect to <api-server-fqdn> port 443`).+2. Identify the node pool instances that failed provisioning.++    - For node pools that use **Virtual Machine Scale Sets** (default):++        ```azurecli+        az vmss list \+            --resource-group <node-resource-group> \+            --query "[].{name:name, nodePool:tags.\"aks-managed-poolName\", provisioningState:provisioningState}" \+            --output table++        az vmss list-instances \+            --resource-group <node-resource-group> \+            --name <vmss-name> \+            --query "[].{instanceId:instanceId, name:name, provisioningState:provisioningState}" \+            --output table+        ```+    - For [Virtual Machines node pools](/en-us/azure/aks/virtual-machines-node-pools):++        ```azurecli+        az vm list \+            --resource-group <node-resource-group> \+            --query "[?tags.\"aks-managed-poolName\"=='<node-pool-name>'].{name:name, provisioningState:provisioningState}" \+            --output table+        ```++### Step 2: Test connectivity from the node to the API server++Because the node didn't register in the cluster, you can't use `kubectl debug` to access it. Instead, use [Azure Run Command](/en-us/azure/virtual-machines/linux/run-command) to run the same check that the bootstrap script runs.++Note++- The VM or VMSS instance must still exist and be running. If AKS already removed the failed instance, run the tests from a test VM that you deploy in the same node subnet (with the same NSG and route table).+- Run Command returns only the last 4,096 bytes of output, and only one Run Command can run on a VM at a time.+- The `--query "value[0].message" --output tsv` parameters in the following commands print the command output as readable text instead of JSON.++The following command is the same check that the bootstrap script runs. It sends an HTTPS request to the API server FQDN by using the cluster certificate authority (`/etc/kubernetes/certs/ca.crt`) that's already on the node. The verbose (`-v`) output shows the IP address that the FQDN resolves to and where the connection fails, so a separate DNS lookup isn't required. The `--max-time 30` parameter stops the test after 30 seconds if the API server doesn't respond.++- For a **Virtual Machine Scale Sets** node pool, run the [az vmss run-command invoke](/en-us/cli/azure/vmss/run-command#az-vmss-run-command-invoke) command:++    ```azurecli+    az vmss run-command invoke \+        --resource-group <node-resource-group> \+        --name <vmss-name> \+        --instance-id <instance-id> \+        --command-id RunShellScript \+        --scripts "curl -v --max-time 30 --cacert /etc/kubernetes/certs/ca.crt https://<api-server-fqdn>:443" \+        --query "value[0].message" \+        --output tsv+    ```+- For a **Virtual Machines** node pool, run the [az vm run-command invoke](/en-us/cli/azure/vm/run-command#az-vm-run-command-invoke) command:++    ```azurecli+    az vm run-command invoke \+        --resource-group <node-resource-group> \+        --name <vm-name> \+        --command-id RunShellScript \+        --scripts "curl -v --max-time 30 --cacert /etc/kubernetes/certs/ca.crt https://<api-server-fqdn>:443" \+        --query "value[0].message" \+        --output tsv+    ```++If the node can reach the API server, the `[stdout]` section contains an `Unauthorized` response from the API server, and the `[stderr]` section shows a successful connection. The `401` response is expected because the request is anonymous. The following example is abbreviated: -```shell-nc -vz <cluster-fqdn> 443+```output+Enable succeeded:+[stdout]+{+  "kind": "Status",+  "apiVersion": "v1",+  "metadata": {},+  "status": "Failure",+  "message": "Unauthorized",+  "reason": "Unauthorized",+  "code": 401+}+[stderr]+* Host <api-server-fqdn>:443 was resolved.+* IPv4: <api-server-ip>+*   Trying <api-server-ip>:443...+* Connected to <api-server-fqdn> (<api-server-ip>) port 443+...+*  SSL certificate verify ok.+...+< HTTP/2 401+... ``` -If you're using egress filtering through a firewall, make sure that traffic is allowed to your cluster FQDN.+If the node can't reach the API server, the `[stdout]` section is empty and curl reports the error at the end of the `[stderr]` section. For example:++```output+Enable succeeded:+[stdout]++[stderr]+* Host <api-server-fqdn>:443 was resolved.+* IPv4: <api-server-ip>+*   Trying <api-server-ip>:443...+* Connection timed out after 30001 milliseconds+* Closing connection+curl: (28) Connection timed out after 30001 milliseconds+```++Note++`Enable succeeded` only means that Run Command ran the script. It doesn't mean that the connectivity test passed.++#### Review the node provisioning log (optional)++To see the exact command and output of the failed check, review the end of the node provisioning log:++```azurecli+az vmss run-command invoke \+    --resource-group <node-resource-group> \+    --name <vmss-name> \+    --instance-id <instance-id> \+    --command-id RunShellScript \+    --scripts "grep -E 'curl -v --cacert|API server connection check code' /var/log/azure/cluster-provision.log | tail -n 20" \+    --query "value[0].message" \+    --output tsv+```++For a Virtual Machines node pool, use `az vm run-command invoke --resource-group <node-resource-group> --name <vm-name>` with the same `--command-id`, `--scripts`, `--query`, and `--output` values.++### Step 3: Interpret the results++| Result | Meaning | Next step |+| --- | --- | --- |+| curl returns an HTTP response (for example, `HTTP/2 401` with an `Unauthorized` status) | The node can now reach the API server. The issue might have been transient or already fixed. | Retry the operation (see Step 5). |+| `curl: (6) Could not resolve host` | The node can't resolve the API server name. | See [Troubleshoot the K8SAPIServerDNSLookupFailVMExtensionError error code (52)](error-code-k8sapiserverdnslookupfailvmextensionerror). |+| The IP address in the `IPv4:` or `Trying` line isn't the expected API server IP address (private clusters) | DNS returns an incorrect or outdated record. | Verify that the private DNS zone record matches the private endpoint IP and that custom DNS servers forward the zone to Azure DNS (`168.63.129.16`). See [Private AKS clusters](/en-us/azure/aks/private-clusters). |+| `curl: (28)` (`Connection timed out` or `Operation timed out`) | Packets are dropped on the path (NSG, firewall, NVA, UDR, or authorized IP ranges). | Go to Step 4. |+| `curl: (7)` (`Connection refused`), `curl: (35)` or `curl: (56)` (`Connection reset by peer`) | A device on the path actively rejects or resets the connection, or the node connects to the wrong IP address. | Check firewalls, proxies, and NVAs on the path, and verify the resolved IP address. |+| A TLS error, such as `curl: (60) SSL certificate problem` or `curl: (35) ... unexpected eof while reading` | A proxy or TLS inspection device intercepts the connection. | Exclude the API server FQDN from TLS inspection, and don't modify the ALPN extension. |++### Step 4: Fix the network configuration++Based on the cluster configuration, verify the following items.++Tip++To check Azure networking resources (NSGs, route tables, Azure Firewall, NAT gateway, load balancer, and VNet peering) along the path from a node pool to a destination, you can also run a connectivity analysis by using [Azure Virtual Network Verifier](/en-us/azure/virtual-network-manager/concept-virtual-network-verifier). This tool doesn't evaluate third-party or on-premises devices.++#### Public clusters++- If you restrict egress traffic through a firewall or NVA, allow TCP 443 to `*.hcp.<region>.azmk8s.io` (application rule) or to the API server public IP address (network rule). For the complete list of requirements, see [Outbound network and FQDN rules for AKS clusters](/en-us/azure/aks/outbound-rules-control-egress) and [Limit network traffic with Azure Firewall in AKS](/en-us/azure/aks/limit-egress-traffic).+- Make sure that NSGs on the node subnet and node NICs don't deny outbound TCP 443 to the API server.+- Make sure that TLS inspection and proxies don't intercept or alter traffic to the API server FQDN.+- If API server authorized IP ranges are enabled, make sure that the public IP address that the nodes use for egress (for example, the Azure Firewall, NAT gateway, or NVA public IP) is included. For more information, see [Secure access to the API server using authorized IP address ranges in AKS](/en-us/azure/aks/api-server-authorized-ip-ranges).++#### Private clusters++- Verify that the API server FQDN resolves to the private endpoint IP address from the node subnet. If you use custom DNS servers, make sure that they forward the private DNS zone to Azure DNS (`168.63.129.16`) or otherwise return the correct record.+- Make sure that NSGs and UDRs on the node subnet allow traffic to the private endpoint IP address on TCP 443, and that the traffic isn't routed through an NVA that drops it.+- For more information, see [Create a private AKS cluster](/en-us/azure/aks/private-clusters).++### Step 5: Retry the operation++After you fix the network configuration, run the connectivity test from Step 2 again to confirm that the node can reach the API server. Then, retry the operation that failed (create, start, scale, or upgrade). If the cluster or node pool remains in a failed state, you can reconcile the cluster by running the [az aks update](/en-us/cli/azure/aks#az-aks-update) command without other parameters:++```azurecli+az aks update --resource-group <resource-group> --name <cluster-name>+```++## Clusters that use API Server VNet Integration++In clusters that use [API Server VNet Integration](/en-us/azure/aks/api-server-vnet-integration), the nodes connect directly to the private IP address of the API server internal load balancer in the delegated API server subnet, without a DNS lookup. The bootstrap script tests this connection by using `nc -vz <api-server-ip> 443` instead of curl, and it retries the test for longer than the time limit of the Custom Script Extension.++Because of this, if the connection is blocked, node provisioning usually fails with a Custom Script Extension timeout instead of the `K8SAPIServerConnFailVMExtensionError` error (exit code 51). For example, the error contains `CSETimeoutVMExtensionError`, `exit status=124`, `nc -vz <api-server-ip> 443`, and `CSE timeout approaching, exiting early`.++To check whether the connection to the API server is blocked, follow these steps:++1. Get the private IP address of the API server. The IP address also appears in the error message (`nc -vz <api-server-ip> 443`).++    ```azurecli+    az network lb show \+        --resource-group <node-resource-group> \+        --name kube-apiserver \+        --query "frontendIPConfigurations[].privateIPAddress" \+        --output tsv+    ```+2. Test the connection from an affected node by running the same Netcat check that the bootstrap script runs. Use the [az vmss run-command invoke](/en-us/cli/azure/vmss/run-command#az-vmss-run-command-invoke) command for a Virtual Machine Scale Sets node pool, or the [az vm run-command invoke](/en-us/cli/azure/vm/run-command#az-vm-run-command-invoke) command for a Virtual Machines node pool:++    ```azurecli+    az vmss run-command invoke \+        --resource-group <node-resource-group> \+        --name <vmss-name> \+        --instance-id <instance-id> \+        --command-id RunShellScript \+        --scripts "nc -vz -w 10 <api-server-ip> 443" \+        --query "value[0].message" \+        --output tsv+    ```++    ```azurecli+    az vm run-command invoke \+        --resource-group <node-resource-group> \+        --name <vm-name> \+        --command-id RunShellScript \+        --scripts "nc -vz -w 10 <api-server-ip> 443" \+        --query "value[0].message" \+        --output tsv+    ```++    Netcat writes its result to the `[stderr]` section. A successful test shows `Connection to <api-server-ip> 443 port [tcp/https] succeeded!`. A failed test shows an error such as `nc: connect to <api-server-ip> port 443 (tcp) timed out` or `nc: connect to <api-server-ip> port 443 (tcp) failed: Connection refused`.+3. If the test fails, check what blocks the traffic between the node subnet and the API server subnet:++    - Make sure that NSG rules on the node subnet, the node NICs, and the API server subnet allow the following communication:++        | Destination | Source | Protocol | Port |+        | --- | --- | --- | --- |+        | API server subnet CIDR | Cluster (node) subnet | TCP | 443 and 4443 |+        | API server subnet CIDR | Azure Load Balancer | TCP | 9988 |+    - Make sure that UDRs on the node subnet don't send traffic for the API server subnet through an NVA or firewall that drops it or routes it asymmetrically. -In rare cases, the firewall's outbound IP address can be blocked if you've authorized IP addresses that are enabled on your cluster. In this scenario, you must add the outbound IP address of your firewall to the list of authorized IP ranges for the cluster. For more information, see [Secure access to the API server using authorized IP address ranges in AKS](/en-us/azure/aks/api-server-authorized-ip-ranges).+    For more information, see [API Server VNet Integration - NSG security rules](/en-us/azure/aks/api-server-vnet-integration#network-security-group-nsg-security-rules).+4. After you fix the network configuration, run the Netcat test again, and then retry the operation.  ## More information  - [General troubleshooting of AKS cluster creation issues](troubleshoot-aks-cluster-creation-issues)+- [Troubleshoot the OutboundConnFailVMExtensionError error code (50)](error-code-outboundconnfailvmextensionerror)+- [Troubleshoot the K8SAPIServerDNSLookupFailVMExtensionError error code (52)](error-code-k8sapiserverdnslookupfailvmextensionerror)+- [Outbound network and FQDN rules for AKS clusters](/en-us/azure/aks/outbound-rules-control-egress)+- [API Server VNet Integration in AKS](/en-us/azure/aks/api-server-vnet-integration)+- [Create a private AKS cluster](/en-us/azure/aks/private-clusters)+- [Run scripts in a Linux VM by using action Run Commands](/en-us/azure/virtual-machines/linux/run-command)

@@ -19,21 +19,21 @@ The architectural layers manage the following responsibilities: - **Control plane**: Manages mesh configuration, certificate lifecycle, and service discovery for each member cluster. - **Data plane**: Secures service-to-service traffic directly in the member cluster using ambient mode, which requires no sidecars or changes to your applications. -For multi-cluster deployments, Application Network synchronizes service discovery information across member clusters so that services can communicate transparently across cluster boundaries.+For multi-cluster deployments, Application Network synchronizes discovery for participating global services across member clusters. You configure workload enrollment, service participation, and the underlying connectivity required by your topology. -The following diagram illustrates the architectural layers of Azure Kubernetes Application Network and how they interact with each other and with your AKS clusters:+The following diagram shows the management plane, managed control plane, and data plane, with their separate API, control, and application traffic paths: -[![Screenshot of a diagram illustrating the architectural layers of Azure Kubernetes Application Network and how they interact with each other and with AKS clusters.](media/architecture/application-network-architecture.png)](media/architecture/application-network-architecture.png#lightbox)+[![Diagram of the Application Network management plane, managed control plane, and AKS member data plane, with separate control and application traffic paths.](media/architecture/application-network-architecture.svg)](media/architecture/application-network-architecture.svg#lightbox)  ## Management plane  The management plane is the Azure resource provider that handles all Application Network resource operations. When you create an Application Network resource or join a member cluster, the management plane validates the request, provisions supporting Azure resources such as Azure Key Vault for certificate storage, and orchestrates the creation of a regional control plane for the member. -You interact with the management plane through the Azure CLI ([`az appnet`](/en-us/cli/azure/appnet)) or the ARM API. The management plane coordinates all downstream provisioning so that control plane and data plane components are deployed without manual intervention.+You interact with the management plane through the Azure CLI ([`az appnet`](/en-us/cli/azure/appnet)) or the ARM API. The management plane coordinates all downstream provisioning so that control plane and data plane components deploy without manual intervention.  ## Control plane -The control plane is fully managed infrastructure that runs outside your AKS cluster. When a member cluster joins an Application Network, a dedicated control plane is provisioned in the same Azure region as the member cluster, keeping latency between control and data plane components low.+The control plane is fully managed infrastructure that runs outside your AKS cluster. When a member cluster joins an Application Network, the service provisions a dedicated control plane in the same Azure region as the member cluster, keeping latency between control and data plane components low.  Note @@ -52,25 +52,25 @@ The data plane consists of components deployed into the member cluster's `applin The data plane includes the following components:  - **Ztunnel**: A node-level L4 proxy deployed as a DaemonSet. Ztunnel intercepts service-to-service traffic on the node, establishes mTLS connections transparently, and enforces L4 authorization policies.-- **Waypoint proxies**: Optional per-namespace L7 proxies that provide HTTP routing, traffic shifting, fault injection, and L7 authorization. Waypoint proxies are deployed only when L7 policies are configured for a namespace.-- **East-west gateway**: Handles cross-cluster traffic in multi-cluster deployments. You're responsible for providing network connectivity between east-west gateways in each member cluster. For more information, see Multi-cluster service discovery.+- **Waypoint proxies**: Optional per-namespace L7 proxies that provide HTTP routing, traffic shifting, fault injection, and L7 authorization. For deployment and enrollment steps, see the [traffic management walkthrough](traffic-management-use-cases#install-waypoint-and-mark-services-as-global).+- **East-west gateway**: Handles traffic between member clusters in different mesh networks. Members in the same network use direct pod-IP connectivity instead. You're responsible for providing the reachability required by either path. For more information, see Multi-cluster service discovery. - **Istio CNI**: A DaemonSet that configures pod networking for ambient mesh traffic interception. - **CRDs**: Kubernetes custom resource definitions for configuring traffic management and security policies. -When a request is made between services, ztunnel intercepts the traffic, establishes an mTLS connection with the destination's ztunnel, and delivers the request to the destination workload. If L7 policies are configured, traffic routes through a waypoint proxy before reaching its destination. For cross-cluster traffic, ztunnel routes through the east-west gateway to reach workloads in remote member clusters.+When a request is made between services, ztunnel intercepts the traffic, establishes an mTLS connection with the destination's ztunnel, and delivers the request to the destination workload. If the service is enrolled with a waypoint, traffic routes through that proxy before reaching its destination. For cross-cluster traffic, the network identities and available reachability determine whether traffic uses direct pod addresses or east-west gateways.  ## Multi-cluster service discovery -Application Network supports connecting multiple AKS clusters into a unified service mesh. Services in one member cluster can communicate with services in other member clusters, with mTLS enforced end-to-end across cluster boundaries.+Application Network supports connecting multiple AKS clusters into a unified service mesh. Enrolled workloads can communicate with participating services in other member clusters, with mTLS across cluster boundaries. Joining a cluster registers it as a member; it doesn't automatically enroll every workload or export every service. -In open-source Istio multi-cluster deployments, control planes typically connect to the Kubernetes API servers of other clusters to discover services and configuration changes. Application Network follows a similar service discovery model, where control planes obtain service information from member clusters so that services can be discovered across cluster boundaries. This approach enables workloads in one member cluster to resolve and communicate with services running in other clusters without requiring application changes. Adding a new member cluster expands the mesh membership so that services in the new cluster become discoverable by other members.+Configure workload namespaces for ambient mode and mark the services that should participate in cross-cluster discovery with `istio.io/global="true"`. For L7 routing, also configure the required waypoints and their cross-cluster service participation. See the [traffic management walkthrough](traffic-management-use-cases#install-waypoint-and-mark-services-as-global). -Cross-cluster traffic flows through east-west gateways deployed in each member cluster. You're responsible for providing network reachability between the east-west gateways of your member clusters — for example, through VNet peering, VPN, or other connectivity solutions. All cross-cluster traffic is encrypted with mTLS.+Each member has a mesh network identity. Members with the same network name require direct, bidirectional reachability between their pod IP addresses. Members in different networks use east-west gateways, for which you provide reachability through VNet peering, VPN, or another connectivity solution. If a custom name isn't provided, Application Network uses a distinct default identity for each member. A shared name describes the direct traffic path; it doesn't create connectivity or disable gateway provisioning. See [Configure multi-cluster connectivity](multi-cluster-connectivity) for topology selection and CLI examples.  ## How components interact -The control plane connects to the member cluster's Kubernetes API server, watches for service and configuration changes, and pushes xDS updates to ztunnel and waypoint proxies. When multiple members exist, each control plane also exchanges service information through managed messaging so that every member has a consistent view of all services across the mesh.+The control plane connects to the member cluster's Kubernetes API server, watches for service and configuration changes, and pushes xDS updates to ztunnel and waypoint proxies. When multiple members exist, Application Network synchronizes discovery for participating global services so the control planes can configure cross-cluster communication. -Certificate management provisions a root CA when the Application Network resource is created and issues intermediate CAs for each member cluster. Workload certificates are issued with a 24-hour validity period and rotated automatically every 12 hours. The entire certificate lifecycle, from root CA through intermediate and workload certificates, is managed without manual intervention.+Certificate management provisions a unique root CA when the Application Network resource is created and issues a dedicated intermediate CA for each Application Network member. Workload certificates are issued with a 24-hour validity period and rotated automatically every 12 hours. The entire certificate lifecycle, from root CA through intermediate and workload certificates, is managed without manual intervention.  For the full certificate hierarchy and rotation schedule, see the [Overview of Azure Kubernetes Application Network security](security).

@@ -15,8 +15,10 @@ This article provides detailed descriptions and requirements for components of A - An Application Gateway for Containers frontend defines the entry point client traffic uses for a given Application Gateway for Containers.     - You can't associate a frontend with more than one Application Gateway for Containers.     - You can create a CNAME record by using the unique FQDN that each frontend provides.-    - Private IP addresses aren't currently supported.+    - Both public and private frontends can exist on the same gateway. - A single Application Gateway for Containers can support more than one frontend.+- [Private frontends](private-ingress)allow private endpoints to be connected for private connectivity.+    - You can connect up to 200 private endpoints to a private frontend.  ### Application Gateway for Containers associations @@ -97,6 +99,10 @@ Some of the benefits of using the managed add-on over a helm based deployment ar  - A private IP address isn't explicitly defined as an Azure Resource Manager resource. A private IP address refers to a specific host address within a given virtual network's subnet. +### Private endpoint++- A [private endpoint](/en-us/azure/private-link/private-endpoint-overview) is a network interface that uses a private IP address from your virtual network. This network interface connects you privately and securely to a private frontend in Application Gateway for Containers.+ ### Subnet delegation  - Microsoft.ServiceNetworking/trafficControllers is the namespace adopted by Application Gateway for Containers and you can delegate it to a virtual network's subnet.

@@ -7,7 +7,7 @@ AKS preview features are available on a self-service, opt-in basis. Previews are - [AKS support policies](/en-us/azure/aks/support-policies) - [Azure support FAQ](/en-us/azure/aks/faq) -Azure Kubernetes Application Network leverages [Azure Monitor](/en-us/azure/azure-monitor/fundamentals/overview) for comprehensive observability of member clusters. This article provides an overview of the observability features available for Azure Kubernetes Application Network, including supported metrics, logs, and tracing capabilities.+Azure Kubernetes Application Network uses [Azure Monitor](/en-us/azure/azure-monitor/fundamentals/overview) to provide comprehensive observability for member clusters. This article gives an overview of the observability features available for Azure Kubernetes Application Network, including supported metrics, logs, and tracing capabilities.  ## Observability features @@ -16,12 +16,15 @@ Azure Kubernetes Application Network member observability infrastructure include | Feature | Workspace type | Description | | --- | --- | --- | | [**Managed Prometheus**](/en-us/azure/azure-monitor/metrics/prometheus-metrics-overview) | [Azure Monitor workspace](/en-us/azure/azure-monitor/metrics/azure-monitor-workspace-overview) | Stores Prometheus metrics scraped from your cluster. |-| [**Container logging**](/en-us/azure/azure-monitor/containers/container-insights-log-query) | [Log Analytics workspace](/en-us/azure/azure-monitor/logs/log-analytics-workspace-overview) | Stores container logs and control plane logs. |+| [**Data plane logs**](logs#enable-data-plane-logs) | [Log Analytics workspace](/en-us/azure/azure-monitor/logs/log-analytics-workspace-overview) | Stores application and in-cluster component logs collected through Container Insights. |+| [**Control plane logs**](logs#enable-control-plane-logs) | [Log Analytics workspace](/en-us/azure/azure-monitor/logs/log-analytics-workspace-overview) | Stores Istiod logs collected through a diagnostic setting on the Application Network member resource. | | [**Dashboards with Grafana**](/en-us/azure/azure-monitor/visualize/visualize-grafana-overview) | [Azure Monitor workspace](/en-us/azure/azure-monitor/metrics/azure-monitor-workspace-overview) | Links to your Azure Monitor workspace to visualize Prometheus metrics in Grafana dashboards. |  ## Workspace options for Azure Kubernetes Application Network observability -You can create new workspaces or use existing ones. If you don't specify a workspace ID during onboarding, a default workspace will be created in your resource group. You can onboard workspaces for [metrics](metrics) or [logs](logs). For more information about workspace options, see [Create workspaces](/en-us/azure/azure-monitor/containers/kubernetes-monitoring-enable?tabs=cli#create-workspaces).+You can create new workspaces or use existing ones. Configure the Azure Monitor workspace when you enable [metrics collection](metrics), and configure the Log Analytics workspace when you enable Container Insights or create a member diagnostic setting for [logs](logs). For more information about workspace options, see [Create workspaces](/en-us/azure/azure-monitor/containers/kubernetes-monitoring-enable?tabs=cli#create-workspaces).++Configure metrics collection, data plane log collection, and control plane diagnostic logs separately. A workspace alone doesn't enable collection. Envoy access logs also require a Telemetry configuration and traffic through the relevant proxy.  ## Observability limitations 

@@ -44,6 +44,12 @@ Note that the `ImageUpdateAutomation` commit template should use the fields `.Ch  Once the manifests are updated in the sources, Flux will reconcile the new API versions. +## October 2026 - `microsoft.flux` version 1.26.4++Changes in this version include:++- Addressed security vulnerabilities in `fluxconfig-agent`, `fluxconfig-controller`, `fluent-bit-mdm`, `source-controller`, `kustomize-controller`, `helm-controller`, `notification-controller`, `image-automation-controller`, and `image-reflector-controller` by updating the Go packages and base images. Azure Flux controllers are pinned to go1.26.+ ## October 2026 - `microsoft.flux` version 1.26.2  Flux version: [Release v2.9.6](https://github.com/fluxcd/flux2/releases/tag/v2.9.6)

@@ -2,16 +2,20 @@  ## Summary -This article helps you find and troubleshoot Azure Kubernetes Service (AKS) nodes that experience Source Network Address Translation (SNAT) port exhaustion.+This article helps you troubleshoot SNAT port exhaustion on Azure Kubernetes Service (AKS) nodes. Learn how to find the affected node, identify pods with high outbound connections, and reduce outbound connectivity issues caused by Source Network Address Translation (SNAT) port exhaustion.  Note  - To troubleshoot SNAT port exhaustion on AKS nodes in an AKS cluster running [Kubernetes jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/), perform the following steps only when the jobs are actively running on the AKS nodes. - To learn more about SNAT ports and their allocation per virtual machine, see [What SNAT ports are](/en-us/azure/load-balancer/load-balancer-outbound-connections#what-are-snat-ports). -## Step 1: Locate the node with SNAT port exhaustion+## Troubleshoot SNAT port exhaustion -1. Get the IP address of the AKS node that experiences active SNAT port exhaustion from the Azure portal.+Follow these steps to identify and troubleshoot SNAT port exhaustion on AKS nodes.++### Step 1: Locate the node with SNAT port exhaustion++1. Get the IP address of the AKS node that experiences active SNAT port exhaustion from the [Azure portal](https://portal.azure.com).      1. Locate the default Kubernetes load balancer by navigating to your AKS cluster's resource group. @@ -19,28 +23,28 @@ Note     2. Locate the AKS node that's experiencing SNAT port exhaustion by [checking SNAT port usage and allocation](/en-us/azure/load-balancer/load-balancer-standard-diagnostics#how-do-i-check-my-snat-port-usage-and-allocation) on the load balancer metrics page. The **Values** drop-down list shows node IP addresses.          [![Screenshot of load balancer metrics showing an AKS node IP address to identify SNAT port exhaustion.](media/snat-port-exhaustion/aks-node-ip-address.png)](media/snat-port-exhaustion/aks-node-ip-address.png#lightbox)-2. Connect to your AKS cluster and use the node IP address to get the node name by running the following `kubectl` command:+2. Connect to your AKS cluster and use the node IP address to get the node name by running the following `kubectl` command.      ```console     kubectl get nodes -o wide | grep <node IP>     ``` -## Step 2: Locate the Linux pod with high outbound connections+### Step 2: Locate the Linux pod with high outbound connections  Note -- [Tcptracer](https://github.com/iovisor/bcc/blob/master/tools/tcptracer.py) is one of the [BPF Compiler Collection (BCC) tools](https://github.com/iovisor/bcc#contents) that are pre-installed on Linux nodes. It allows you to trace TCP established connections (`connect()`, `accept()`, and `close()`). You can use it to find high outbound connections from the source IP address and network namespace (netns) of a pod.-- To access the BCC tools, use [kubectl node-shell](https://github.com/kvaps/kubectl-node-shell)*only*.-- All the following commands in this section are run as the root user on a Linux node that has the BCC tools installed.+- [Tcptracer](https://github.com/iovisor/bcc/blob/master/tools/tcptracer.py) is one of the [BPF Compiler Collection (BCC) tools](https://github.com/iovisor/bcc#contents) that are preinstalled on Linux nodes. It allows you to trace TCP established connections (`connect()`, `accept()`, and `close()`). You can use it to find high outbound connections from the source IP address and network namespace (netns) of a pod.+- Use *only*[kubectl node-shell](https://github.com/kvaps/kubectl-node-shell) to access the BCC tools.+- Run all the following commands in this section as the root user on a Linux node that has the BCC tools installed. -1. On your running terminal, install [kubectl node-shell](https://github.com/kvaps/kubectl-node-shell):+1. On your running terminal, install [kubectl node-shell](https://github.com/kvaps/kubectl-node-shell).      ```bash     curl -LO https://github.com/kvaps/kubectl-node-shell/raw/master/kubectl-node_shell     chmod +x ./kubectl-node_shell     sudo mv ./kubectl-node_shell /usr/local/bin/kubectl-node_shell     ```-2. Use kube node-shell plugin to connect to the node that experiences SNAT port exhaustion and use `tcptracer` to trace TCP established connections:+2. Use the kube node-shell plugin to connect to the node that experiences SNAT port exhaustion. Use `tcptracer` to trace TCP established connections.      ```bash     kubectl node-shell <node name>@@ -48,7 +52,7 @@ Note     /usr/share/bcc/tools# python3 tcptracer -t4v     ``` -    Here's a command output example:+    Here's a command output example.      ```output     Tracing TCP established connections. Ctrl-C to end.@@ -62,9 +66,9 @@ Note     7xxxx3      connect      18633  curl             4  1.2.3.4           13.14.15.16      35690  80     4026532785     9xxxx7      close        18633  curl             4  1.2.3.4           13.14.15.16      35690  80     4026532785     ```-3. Write the previous command output to a log file, and then sort the output to review a list of high connections.+3. Write the previous command output to a log file. Sort the output to review a list of high connections. -    Because SNAT port exhaustion is caused only by outbound connections to *public* endpoints, filter the trace on the destination address (`DADDR`, the `$7` column) to keep only public IPv4 destinations before aggregating. This removes noise from traffic that doesn't consume SNAT ports, such as private (RFC 1918) addresses, localhost, the Azure Instance Metadata Service (IMDS), and Azure WireServer:+    Because SNAT port exhaustion is caused only by outbound connections to *public* endpoints, filter the trace on the destination address (`DADDR`, the `$7` column`) to keep only public IPv4 destinations before aggregating. This filter removes noise from traffic that doesn't consume SNAT ports, such as private (RFC 1918) addresses, localhost, the Azure Instance Metadata Service (IMDS), and Azure WireServer.      ```bash     python3 tcptracer -t4v > log@@ -81,7 +85,7 @@ Note     - Link-local addresses, including the IMDS endpoint `169.254.169.254`: `169.254.0.0/16`     - Azure WireServer: `168.63.129.16` -    Here's a command output example:+    Here's a command output example.      ```output     Count  SADDR         DADDR          NETNS@@ -89,9 +93,9 @@ Note     6      10.244.1.190  13.89.174.133  4026532293     ``` -    The remaining sources (`SADDR`) are the pods that establish connections to public destinations, which are the ones actually consuming SNAT ports.+    The remaining sources (`SADDR`) are the pods that establish connections to public destinations. These pods consume SNAT ports. 4. Map the source IP address (`SADDR`) with the most connections from the previous output to a pod. If it doesn't work, you can continue.-5. Note the `SADDR` or `NETNS` value with the most connections from the previous output, and then run the following [lsns](https://man7.org/linux/man-pages/man8/lsns.8.html) command to map it to a PID. Lsns is a Linux tool that lists namespaces and maps namespaces to PIDs in the Linux process tree.+5. Note the `SADDR` or `NETNS` value with the most connections from the previous output. Run the following [lsns](https://man7.org/linux/man-pages/man8/lsns.8.html) command to map it to a PID. Lsns is a Linux tool that lists namespaces and maps namespaces to PIDs in the Linux process tree.      ```bash     lsns -t net@@ -116,14 +120,14 @@ Note       `-containerd-shim,20946 -namespace k8s.io -id 2xxxf...     ``` -    Note the first five characters of the containerd `-id` in the command output. It will be used in step 7.+    Note the first five characters of the containerd `-id` in the command output. It will be used in the next step. 7. Map the previous containerd `-id` value to a POD ID by using [crictl](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md). Crictl provides a CLI for CRI-compatible container runtimes.      ```bash     crictl ps -a     ``` -    Here's a command output example:+    Here's a command output example.      ```output     CONTAINER   IMAGE     CREATED      STATE     NAME    ATTEMPT    POD ID        POD@@ -131,16 +135,16 @@ Note     ```      Use the first five characters of the previous containerd `-id` value to match a POD ID.-8. Get all pods running on the node and use the previous POD ID to match the pod that has high outbound connections from the command output:+8. Get all pods running on the node. Use the previous POD ID to match the pod that has high outbound connections from the command output.      ```bash     kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<nodename>     ``` -## Step 3: Find all outbound network connections made by the application+### Step 3: Find all outbound network connections made by the application  # [For a Linux pod](#tab/for-a-linux-pod)-1. Execute into the pod that's identified as having high outbound connections in Step 2: Locate the Linux pod with high outbound connections by using one of the following commands:+1. Use one of the following commands to connect to the pod that has high outbound connections, as identified in Step 2: Locate the Linux pod with high outbound connections:      - ```bash         kubectl exec -it <pod name> -n <namespace> /bin/bash@@ -148,7 +152,7 @@ Note     - ```bash         kubectl exec -it <pod name> -n <namespace> /bin/sh         ```-2. Install the netstat command-line tool in the pod by running the following command. Netstat is a network troubleshooting tool for admins only.+2. Install the netstat command-line tool in the pod by running the appropriate command for your Linux distribution. Netstat is a network troubleshooting tool for admins only.      - On Debian, Ubuntu, or Linux Mint @@ -182,31 +186,31 @@ Note         ```bash         zypper install net-tools         ```-3. Once netstat is installed in the pod, run the following command:+3. After installing netstat in the pod, run the following command.      ```bash     netstat -ptn | grep -i established     ``` -    Here's a command output example:+    Here's a command output example.      ```output     tcp        0      0 10.x.x.x:xxxx        20.x.x.x:443       ESTABLISHED xxxxx3/telnet     ```  # [For a Windows pod](#tab/for-a-windows-pod)-1. Execute into a Windows pod by running the following command:+1. Connect to a Windows pod by running the following command.      ```console     kubectl exec -it <Windows pod name> -- cmd.exe     ```-2. Run the following netstat command to find established connections from the Windows application running in the pod:+2. Run the following netstat command to find established connections from the Windows application running in the pod.      ```console     netstat -aon | find /i "established"     ``` -    Here's a command output example:+    Here's a command output example.      ```output     C:\inetpub\wwwroot>netstat -aon | find /i "established"@@ -218,6 +222,6 @@ Note  --- -In the command output, the local address is the pod's IP address, and the foreign address is the IP to which the application connects. The public IP connections in the `ESTABLISHED` state are the connections that utilize SNAT. Ensure that you count only the connections in the `ESTABLISHED` state to public IP addresses and ignore any connections in the `ESTABLISHED` state to private IP addresses.+In the command output, the local address is the pod's IP address, and the foreign address is the IP to which the application connects. The public IP connections in the `ESTABLISHED` state are the connections that use SNAT. Count only the connections in the `ESTABLISHED` state to public IP addresses. Ignore any connections in the `ESTABLISHED` state to private IP addresses. -Repeat the steps in this section for all other pods running on the node. The pod that has the most connections in the `ESTABLISHED` state to public IP addresses hosts the application that causes SNAT port exhaustion on the node. Work with your application developers to tune the application for improved network performance using the recommendations mentioned in [Design connection-efficient applications](/en-us/azure/load-balancer/troubleshoot-outbound-connection#design-connection-efficient-applications). After implementing the recommendations, verify that you see less SNAT port exhaustion.+Repeat the steps in this section for all other pods running on the node. The pod with the most connections in the `ESTABLISHED` state to public IP addresses hosts the application that causes SNAT port exhaustion on the node. Work with your application developers to tune the application for improved network performance by using the recommendations in [Design connection-efficient applications](/en-us/azure/load-balancer/troubleshoot-outbound-connection#design-connection-efficient-applications). After implementing the recommendations, verify that you see less SNAT port exhaustion.

@@ -17,7 +17,7 @@ At a high level, Application Gateway for Containers provides: - **Shared ingress platform**: Separate platform-owned ingress infrastructure from application-owned routing configuration. - **Incremental adoption**: Use Ingress or Gateway API resources so workloads can move to Application Gateway for Containers over time. - **AI inference support**: Route inference traffic to self-hosted model servers with model-aware routing and request-time endpoint selection, configured through the Gateway API Inference Extension.-- **Application delivery controls**: Use layer 7 routing, traffic splitting, health probes, retries, TLS, mutual authentication, and Web Application Firewall (WAF) policies.+- **Application delivery controls**: Use layer 7 routing, traffic splitting, health probes, retries, TLS, mutual authentication, JWT validation, and Web Application Firewall (WAF) policies.  ## How does it work? @@ -69,6 +69,7 @@ Application Gateway for Containers supports the following features for traffic m     - SSL termination     - End to End SSL - Ingress and Gateway API support+- JWT validation - Layer 7 HTTP/HTTPS request forwarding based on prefix/exact match on:     - Hostname     - Path@@ -77,6 +78,7 @@ Application Gateway for Containers supports the following features for traffic m     - Methods     - Ports (80/443) - Mutual authentication (mTLS) to frontend, backend, or end-to-end+- Private endpoint support - Request mirroring - Server-sent event (SSE) support - TLS policies@@ -130,7 +132,7 @@ Application Gateway for Containers is currently available in the following regio  ### Gateway API support -ALB Controller implements version [v1.5](https://gateway-api.sigs.k8s.io/docs/implementations/versions/v1.5/) of the [Gateway API](https://gateway-api.sigs.k8s.io/).+ALB Controller implements version [v1.6](https://gateway-api.sigs.k8s.io/docs/implementations/versions/v1.6/) of the [Gateway API](https://gateway-api.sigs.k8s.io/).  | Gateway API Resource | Support | Comments | | --- | --- | --- |@@ -139,8 +141,14 @@ ALB Controller implements version [v1.5](https://gateway-api.sigs.k8s.io/docs/im | [HTTPRoute](https://gateway-api.sigs.k8s.io/reference/api-types/httproute/) | Yes |  | | [GRPCRoute](https://gateway-api.sigs.k8s.io/reference/api-types/grpcroute/) | Yes |  | | [ReferenceGrant](https://gateway-api.sigs.k8s.io/reference/api-types/referencegrant/) | Yes | Currently supports version v1alpha1 of this API |++### Gateway API Inference Extension support++ALB Controller implements version [v1.6](https://gateway-api-inference-extension.sigs.k8s.io/reference/spec/) of the [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/).++| Gateway API Inference Extension Resource | Support | Comments |+| --- | --- | --- | | [InferencePool](https://gateway-api-inference-extension.sigs.k8s.io/reference/spec/#inferencepool) | Yes |  |-| [InferenceObjective](https://gateway-api-inference-extension.sigs.k8s.io/?h=inferenceobjective#deploy-inferenceobjective-optional) | Yes | Currently implements version v1alpha1 of this API |  ### Ingress API support 

@@ -2,26 +2,28 @@  ## Summary -This article explains basic troubleshooting of outbound connections from an Azure Kubernetes Service (AKS) cluster. By following these steps, you can identify faulty components and restore connectivity faster.+This article explains how to troubleshoot outbound connections from an Azure Kubernetes Service (AKS) cluster. Follow these steps to identify faulty network components, isolate blockers, and restore egress connectivity faster.  ## Prerequisites +Ensure you have the following prerequisites:+ - The Kubernetes [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) tool, or a similar tool to connect to the cluster. To install kubectl by using [Azure CLI](/en-us/cli/azure/install-azure-cli), run the [az aks install-cli](/en-us/cli/azure/aks#az-aks-install-cli) command. - The [apt-get](https://linux.die.net/man/8/apt-get) command-line tool for handling packages. - The Client URL ([cURL](https://www.tecmint.com/install-curl-in-linux/)) tool, or a similar command-line tool. - The `nslookup` command-line ([dnsutils](https://www.tecmint.com/install-dig-and-nslookup-in-linux/)) tool for checking DNS resolution. -## Scenarios for outbound traffic in Azure Kubernetes Service+## Outbound traffic scenarios  Traffic that originates from within the AKS cluster, whether it's from a pod or a worker node, is considered outbound traffic from the cluster. If there's an issue in the outbound flow for an AKS cluster, before you troubleshoot, first look at the scenarios for outbound traffic flow.  You can classify the outbound traffic from an AKS cluster into the following categories: -1. Traffic to a pod or service in the same cluster (internal traffic).-2. Traffic to a network resource or endpoint in the same virtual network or a different virtual network that uses virtual network peering.-3. Traffic to an on-premises environment through a VPN connection or an Azure ExpressRoute connection.-4. Traffic outside the AKS network through Azure Load Balancer (public outbound traffic).-5. Traffic outside the AKS network through Azure Firewall or a proxy server (public outbound traffic).+- Traffic to a pod or service in the same cluster (internal traffic).+- Traffic to a network resource or endpoint in the same virtual network or a different virtual network that uses virtual network peering.+- Traffic to an on-premises environment through a VPN connection or an Azure ExpressRoute connection.+- Traffic outside the AKS network through Azure Load Balancer (public outbound traffic).+- Traffic outside the AKS network through Azure Firewall or a proxy server (public outbound traffic).  #### Internal traffic @@ -35,7 +37,7 @@ If the traffic is for a destination on the internet, the default method is to se  [![Screenshot of outbound internet traffic from an AKS cluster through Azure Load Balancer.](media/basic-troubleshooting-outbound-connections/external-traffic-load-balancer.svg)](media/basic-troubleshooting-outbound-connections/external-traffic-load-balancer.svg#lightbox) -The preceding discussion about Load Balancer applies specifically to `outboundType=loadBalancer`. You also need to check the managed NAT gateway outbound architectures [from here](/en-us/azure/aks/nat-gateway).+The preceding discussion about Load Balancer applies specifically to `outboundType=loadBalancer`. You also need to check the managed network address translation (NAT) gateway outbound architectures [from here](/en-us/azure/aks/nat-gateway).  #### Public outbound traffic through Azure Firewall or a proxy server @@ -53,9 +55,9 @@ It's important to understand the nature of egress flow for your cluster so that  When you troubleshoot outbound traffic in AKS, it's important to know what network resources are present (that is, the hops through which the traffic passes). Here, the network resource could be one of the following components: -- Azure Load Balancer+- Load Balancer - Azure Firewall or a custom firewall-- A network address translation (NAT) gateway+- A NAT gateway - A proxy server - Network security group (NSG) - Network policy@@ -89,7 +91,7 @@ If other troubleshooting steps don't provide any conclusive outcome, take packet - [Capture a TCP dump from a Windows node in an AKS cluster](../logs/capture-tcp-dump-windows-node-aks) - [Capture TCP packets from a pod on an AKS cluster](../logs/packet-capture-pod-level) -Before starting the troubleshooting flow, determine the cluster outbound type (loadBalancer, managedNATGateway, userAssignedNATGateway, userDefinedRouting) and follow scenario-specific validation.+Before starting the troubleshooting flow, determine the cluster outbound type (loadBalancer, managedNATGateway, userAssignedNATGateway, userDefinedRouting) and follow scenario-specific validation:  - [Customize Cluster Egress with Outbound Types in Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Learn](/en-us/azure/aks/egress-outboundtype) - [Customize cluster egress with a user-defined routing table in Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Learn](/en-us/azure/aks/egress-udr)@@ -97,10 +99,10 @@ Before starting the troubleshooting flow, determine the cluster outbound type (l  ## Troubleshooting checklists -For basic troubleshooting for egress traffic from an AKS cluster, follow these steps:+To troubleshoot egress traffic from an AKS cluster, follow these steps:  1. Check if Azure network resources are blocking traffic to the endpoint using Azure Virtual Network Verifier (Preview).-2. Make sure that the Domain Name System (DNS) resolution for the endpoint works correctly.+2. Check that the DNS resolution for the endpoint works correctly. 3. Make sure that you can reach the endpoint through an IP address. 4. [Make sure that you can reach the endpoint from another source](troubleshoot-connection-pods-services-same-cluster). 5. [Make sure that the endpoint is working](troubleshoot-connections-endpoints-same-virtual-network).@@ -118,10 +120,10 @@ This checklist assumes no service mesh when you do basic troubleshooting. If you  To determine if Azure network resources block traffic to the endpoint, run a connectivity analysis from your AKS cluster nodes to the endpoint by using the [Azure Virtual Network Verifier (Preview)](/en-us/azure/virtual-network-manager/concept-virtual-network-verifier#supported-features-of-the-reachability-analysis) tool. The connectivity analysis covers the following resources: -- Azure Load Balancer+- Load Balancer - Azure Firewall-- A network address translation (NAT) gateway-- Network security group (NSG)+- A NAT gateway+- NSG - Network policy - User defined routes (route tables) - Virtual network peering@@ -130,6 +132,8 @@ Note  Azure Virtual Network Verifier (Preview) can't access any external or third-party networking resources, such as a custom firewall. If the connectivity analysis doesn't detect any blocked traffic, perform a manual check of any external networking to cover all hops in the traffic flow. +To run a connectivity analysis from your AKS cluster nodes to an endpoint, follow these steps:+ 1. Go to your cluster in the Azure portal. In the sidebar, go to the **Settings** &gt; **Node pools** blade. 2. Identify the node pool you want to run a connectivity analysis from. Select the node pool as the scope. 3. Select **Connectivity analysis (Preview)** from the toolbar at the top of the page. If you don't see it, select the three dots **...** in the toolbar at the top of the page to open the expanded menu.@@ -137,9 +141,9 @@ Azure Virtual Network Verifier (Preview) can't access any external or third-part 5. Select a public domain name or endpoint as the destination for the analysis, such as `mcr.microsoft.com`. The destination IP addresses are also populated automatically. 6. Run the analysis and wait up to two minutes for the results. In the resulting diagram, identify the associated Azure network resources and where traffic is blocked. To view the detailed analysis output, select the **JSON output** tab or select the arrows in the diagram. -#### Check that the Domain Name Service (DNS) resolution for the endpoint works correctly+#### Check that the DNS resolution for the endpoint works correctly -You can run a DNS lookup to the endpoint by running a debugging pod on one of your AKS nodes. If the issue is isolated to a specific problematic pod or namespace, run the DNS lookup from within the same namespace where you notice the problem.+You can run a Domain Name Service (DNS) lookup to the endpoint by running a debugging pod on one of your AKS nodes. If the issue is isolated to a specific problematic pod or namespace, run the DNS lookup from within the same namespace where you notice the problem.  If you can't run the [kubectl exec](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#exec) command to connect to an existing pod, you can [start a test pod in the same namespace as the problematic pod](https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#create-a-simple-pod-to-use-as-a-test-environment) to run the tests. @@ -149,20 +153,22 @@ If DNS resolution or egress traffic doesn't let you install the necessary networ  ##### Example procedure for checking DNS resolution -1. Start a test pod in the problematic namespace:+Follow these steps to check the DNS resolution for the endpoint:++1. Start a test pod in the problematic namespace.      ```bash     kubectl run -it --rm aks-ssh --namespace <namespace> --image=debian:stable --overrides='{"spec": { "nodeSelector": {"kubernetes.io/os": "linux"}}}'     ```      After the test pod is running, you gain access to the pod.-2. Run the following `apt-get` commands to install other tool packages:+2. Run the following `apt-get` commands to install other tool packages.      ```bash     # Update and install tool packages     apt-get update && apt-get install -y dnsutils curl     ```-3. After the packages are installed, run the [nslookup](/en-us/windows-server/administration/windows-commands/nslookup) command to test the DNS resolution to the endpoint:+3. After the packages are installed, run the [nslookup](/en-us/windows-server/administration/windows-commands/nslookup) command to test the DNS resolution to the endpoint.      ```console     $ nslookup microsoft.com # Microsoft.com is used as an example@@ -173,7 +179,7 @@ If DNS resolution or egress traffic doesn't let you install the necessary networ     Name:   microsoft.com     Address: 20.53.203.50     ```-4. Try the DNS resolution from the upstream DNS server directly. This example uses Azure DNS:+4. Try the DNS resolution from the upstream DNS server directly. This example uses Azure DNS.      ```console     $ nslookup microsoft.com 168.63.129.16@@ -186,17 +192,17 @@ If DNS resolution or egress traffic doesn't let you install the necessary networ  Sometimes, there's a problem with the endpoint itself rather than a cluster DNS. In such cases, consider the following checks: -1. Check whether the desired port is open on the remote host:+1. Check whether the desired port is open on the remote host.      ```bash     curl -Ivm5 telnet://microsoft.com:443     ```-2. Check the HTTP response code:+2. Check the HTTP response code.      ```bash     curl -Ivm5 https://microsoft.com     ```-3. Check whether you can connect to any other endpoint:+3. Check whether you can connect to any other endpoint.      ```bash     curl -Ivm5 https://kubernetes.io@@ -205,7 +211,7 @@ Sometimes, there's a problem with the endpoint itself rather than a cluster DNS. To verify that the endpoint is reachable and DNS is functioning from the node hosting the problematic pod, follow these steps:  1. Enter the node hosting the problematic pod by using the debug pod. For more information, see [Connect to Azure Kubernetes Service (AKS) cluster nodes for maintenance or troubleshooting](/en-us/azure/aks/node-access).-2. Test the DNS resolution to the endpoint:+2. Test the DNS resolution to the endpoint.      ```console     $ nslookup  microsoft.com@@ -224,7 +230,7 @@ To verify that the endpoint is reachable and DNS is functioning from the node ho     Name:   microsoft.com     Address: 20.53.203.50     ```-3. Check the **resolv.conf** file to determine whether the expected name servers are added:+3. Check the **resolv.conf** file to determine whether the expected name servers are added.      ```bash     cat /etc/resolv.conf@@ -233,18 +239,20 @@ To verify that the endpoint is reachable and DNS is functioning from the node ho  ##### Example procedure for checking DNS resolution of a Windows pod -1. Run a test pod in the Windows node pool:+Follow these steps to check DNS resolution for a Windows pod:++1. Run a test pod in the Windows node pool.      ```bash     # For a Windows environment, use the Resolve-DnsName cmdlet.     kubectl run dnsutil-win --image='mcr.microsoft.com/windows/servercore:ltsc2022' --overrides='{"spec": { "nodeSelector": {"kubernetes.io/os": "windows"}}}' -- powershell "Start-Sleep -s 3600"     ```-2. Run the [kubectl exec](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#exec) command to connect to the pod by using PowerShell:+2. Run the [kubectl exec](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#exec) command to connect to the pod by using PowerShell.      ```bash     kubectl exec -it dnsutil-win -- powershell     ```-3. Run the [Resolve-DnsName](/en-us/powershell/module/dnsclient/resolve-dnsname) cmdlet in PowerShell to check whether the DNS resolution is working for the endpoint:+3. Run the [Resolve-DnsName](/en-us/powershell/module/dnsclient/resolve-dnsname) cmdlet in PowerShell to check whether the DNS resolution is working for the endpoint.      ```console     PS C:\> Resolve-DnsName www.microsoft.com@@ -284,6 +292,10 @@ If the DNS resolution is successful, continue to the network tests. Otherwise, v  Microsoft provides third-party contact information to help you find additional information about this topic. This contact information may change without notice. Microsoft does not guarantee the accuracy of third-party contact information. -If the basic troubleshooting steps don't identify the root cause, continue with advanced diagnostics. Use Azure Virtual Network Verifier to validate the network path through Azure resources and identify blocked hops. For deeper cluster-network analysis, use Advanced Container Networking Services (ACNS) observability capabilities to investigate DNS failures, packet drops, network policy enforcement, connection tracking behavior, and Layer 7 traffic issues. If you suspect packet loss or intermittent connectivity, collect packet captures from the affected pod and node at the same time to determine where traffic is dropped. Also verify that all required AKS outbound endpoints and FQDN dependencies are permitted through firewalls, proxies, and egress filtering devices before focusing on application-level troubleshooting.+If the basic troubleshooting steps don't identify the root cause, continue with the following advanced diagnostics:++- Use Azure Virtual Network Verifier to validate the network path through Azure resources and identify blocked hops.+- For deeper cluster-network analysis, use Advanced Container Networking Services (ACNS) observability capabilities to investigate DNS failures, packet drops, network policy enforcement, connection tracking behavior, and Layer 7 traffic issues.+- If you suspect packet loss or intermittent connectivity, collect packet captures from the affected pod and node at the same time to determine where traffic is dropped. Also verify that all required AKS outbound endpoints and FQDN dependencies are permitted through firewalls, proxies, and egress filtering devices before focusing on application-level troubleshooting.  [Source](/en-us/azure/aks/container-network-observability-guide)

@@ -176,7 +176,7 @@ Use the simple scenario of deploying two `ConfigMap` resources across selected c 2. Create a `ResourcePlacement` to propagate the ConfigMaps. Save the following YAML to a file named `app-configs-rp.yaml`:      ```yaml-    apiVersion: placement.kubernetes-fleet.io/v1beta1+    apiVersion: placement.kubernetes-fleet.io/v1     kind: ResourcePlacement     metadata:       name: distribute-app-configs

@@ -28,7 +28,7 @@ When you try to start or create an AKS cluster, you receive the following error  ## Cause -The cluster nodes can't resolve the cluster's fully qualified domain name (FQDN) in Azure DNS. Run the following DNS lookup command on the failed cluster node to find DNS resolutions that are valid.+The cluster nodes can't resolve the cluster's fully qualified domain name (FQDN). This issue can occur when the DNS configuration available to the node doesn't return a valid response for the Kubernetes API server FQDN.  | Node OS | Command | | --- | --- |@@ -78,3 +78,30 @@ Conditional Forwarding doesn't support subdomains. ## More information  - [General troubleshooting of AKS cluster creation issues](troubleshoot-aks-cluster-creation-issues)++Because the affected node might be in a failed provisioning state and unavailable through SSH or RDP, use `az vmss run-command invoke` to run the DNS lookup directly on the Virtual Machine Scale Set instance.++### Test DNS resolution from a Linux node++Run `dig` against the cluster FQDN:++```azurecli+az vmss run-command invoke \+  --resource-group <mc-resource-group-name> \+  --name <vmss-name> \+  --instance-id <instance-id> \+  --command-id RunShellScript \+  --scripts "dig <cluster-fqdn>"+```++### Test DNS resolution from a Windows node++Run `nslookup` against the cluster FQDN:++```azurecli+az vmss run-command invoke \+  --resource-group <mc-resource-group-name> \+  --name <vmss-name> \+  --instance-id <instance-id> \+  --command-id RunPowerShellScript+```

@@ -2,39 +2,41 @@  **Applies to**: ✔️ Fleet Manager with hub cluster -Fleet Manager keeps a history of the 10 most recently used placement scheduling policies, along with resource versions the placement selected. These policy and resource versions are held as snapshot objects:+Fleet Manager keeps a history of the 10 most recently used placement scheduling policies, along with resource versions the placement selected. It holds these policy and resource versions as snapshot objects:  - For **ClusterResourcePlacement** (CRP): `ClusterSchedulingPolicySnapshot` and `ClusterResourceSnapshot` (cluster-scoped) - For **ResourcePlacement** (RP): `SchedulingPolicySnapshot` and `ResourceSnapshot` (namespace-scoped) -In this article, we explore these objects so you can understand them should you wish to work with them.+In this article, we explore these objects so you can understand them if you wish to work with them.  Note -If you aren't already familiar with Fleet Manager's resource placement:+If you're not already familiar with Fleet Manager's resource placement:  - For cluster-scoped resources, read the [conceptual overview of ClusterResourcePlacement (CRP)](concepts-resource-placement). - For namespace-scoped resources, read the [conceptual overview of ResourcePlacement (RP)](concepts-namespace-scoped-resource-propagation).  ## How resource snapshots are created -Any change to resources covered by the scope of CRP or RP resource selector will automatically trigger the creation of a new resource snapshot within 60 seconds.+Any change to resources covered by the scope of CRP or RP resource selector automatically triggers the creation of a new resource snapshot within 60 seconds.  ## Viewing resource snapshots -You can view the resource snapshots held by Fleet Manager using the following steps when connected to the [Fleet Manager hub cluster](access-fleet-hub-cluster-kubernetes-api).+You can view the resource snapshots held by Fleet Manager by using the following steps when connected to the [Fleet Manager hub cluster](access-fleet-hub-cluster-kubernetes-api).++::: zone pivot="cluster-scope"  ### Viewing ClusterResourceSnapshot (for ClusterResourcePlacement) -For cluster-scoped placements using `ClusterResourcePlacement`, view `ClusterResourceSnapshot` objects.+For cluster-scoped placements that use `ClusterResourcePlacement`, view `ClusterResourceSnapshot` objects. -In this sample, we use an updated ConfigMap, leading to two snapshots.+In this sample, we use an updated ConfigMap, which creates two snapshots.  ```bash kubectl get clusterresourcesnapshots --show-labels ``` -We see two snapshots, with the most recent one marked as the latest (`kubernetes-fleet.io/is-latest-snapshot=true`), with a resource-index of 1 (`kubernetes-fleet.io/resource-index=1`).+You see two snapshots, and the most recent one is marked as the latest (`kubernetes-fleet.io/is-latest-snapshot=true`) and has a resource-index of 1 (`kubernetes-fleet.io/resource-index=1`).  ```output NAME                           GEN   AGE    LABELS@@ -42,7 +44,7 @@ example-placement-0-snapshot   1     17m    kubernetes-fleet.io/is-latest-snapsh example-placement-1-snapshot   1     2m2s   kubernetes-fleet.io/is-latest-snapshot=true,kubernetes-fleet.io/parent-CRP=example-placement,kubernetes-fleet.io/resource-index=1 ``` -We can inspect the contents of the `example-placement-1-snapshot` object as follows.+You can inspect the contents of the `example-placement-1-snapshot` object as follows.  ```bash kubectl get clusterresourcesnapshots example-placement-1-snapshot -o yaml@@ -79,17 +81,21 @@ spec:       namespace: test-namespace ``` +::: zone-end++::: zone pivot="namespace-scope"+ ### Viewing ResourceSnapshot (for ResourcePlacement) -For namespace-scoped placements using `ResourcePlacement`, view `ResourceSnapshot` objects within the namespace.+For namespace-scoped placements that use `ResourcePlacement`, view `ResourceSnapshot` objects within the namespace. -In this sample, we have a `ResourcePlacement` in the `my-app` namespace with updated ConfigMaps, leading to two snapshots.+In this example, you have a `ResourcePlacement` in the `my-app` namespace with updated ConfigMaps, which creates two snapshots.  ```bash kubectl get resourcesnapshots -n my-app --show-labels ``` -We see two snapshots, with the most recent one marked as the latest (`kubernetes-fleet.io/is-latest-snapshot=true`), with a resource-index of 1 (`kubernetes-fleet.io/resource-index=1`).+You see two snapshots, and the most recent one is marked as the latest (`kubernetes-fleet.io/is-latest-snapshot=true`) and has a resource-index of 1 (`kubernetes-fleet.io/resource-index=1`).  ```output NAME                               GEN   AGE    LABELS@@ -97,14 +103,14 @@ app-configs-rp-0-snapshot          1     15m    kubernetes-fleet.io/is-latest-sn app-configs-rp-1-snapshot          1     1m3s   kubernetes-fleet.io/is-latest-snapshot=true,kubernetes-fleet.io/parent-CRP=app-configs-rp,kubernetes-fleet.io/resource-index=1 ``` -We can inspect the contents of the `app-configs-rp-1-snapshot` object as follows.+You can inspect the contents of the `app-configs-rp-1-snapshot` object as follows.  ```bash kubectl get resourcesnapshots app-configs-rp-1-snapshot -n my-app -o yaml ```  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ResourceSnapshot metadata:   ...@@ -128,19 +134,23 @@ spec:         app: my-application ``` +::: zone-end+ ## Using resource snapshots -Snapshots can be used as part of staged rollouts managed via update runs as a rollback mechanism. Both `ClusterResourcePlacement` and `ResourcePlacement` support staged rollouts with their respective snapshot types:+Use snapshots as part of staged rollouts that update runs manage. They also serve as a rollback mechanism. Both `ClusterResourcePlacement` and `ResourcePlacement` support staged rollouts with their respective snapshot types: -- For `ClusterResourcePlacement`: Uses `ClusterResourceSnapshot` objects with `ClusterStagedUpdateRun`-- For `ResourcePlacement`: Uses `ResourceSnapshot` objects with `StagedUpdateRun`+- `ClusterResourcePlacement` uses `ClusterResourceSnapshot` objects with `ClusterStagedUpdateRun`.+- `ResourcePlacement` uses `ResourceSnapshot` objects with `StagedUpdateRun`. -See [Staged rollout strategy](concepts-rollout-strategy#staged-update-strategy) for details on how to implement staged rollouts for both resource placement types.+For details about how to implement staged rollouts for both resource placement types, see [Staged rollout strategy](concepts-rollout-strategy#staged-update-strategy).  ## Modifying the snapshot history limit  To change the number of items held in history, add a `revisionHistoryLimit` field to your placement and provide an integer value. The default value is 10. +::: zone pivot="cluster-scope"+ ### For ClusterResourcePlacement  ```yaml@@ -172,10 +182,14 @@ spec:       maxSurge: 50% ``` +::: zone-end++::: zone pivot="namespace-scope"+ ### For ResourcePlacement  ```yaml-apiVersion: placement.kubernetes-fleet.io/v1beta1+apiVersion: placement.kubernetes-fleet.io/v1 kind: ResourcePlacement metadata:   name: rp-example@@ -205,3 +219,5 @@ spec:       maxUnavailable: 1       maxSurge: 50% ```++::: zone-end

@@ -2,7 +2,7 @@  AI inference workloads don't behave like traditional stateless HTTP applications. Requests are often model-specific, long-running, expensive to serve, and sensitive to runtime signals such as accelerator availability, queue depth, request priority, token budget, and model server capacity. -Application Gateway for Containers inference gateway supports these workloads by integrating with the Kubernetes [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/), which adds inference-aware resources to the Gateway API model. By using the inference gateway, you can expose self-hosted model servers through Application Gateway for Containers with model-aware and load-aware routing behavior.+Application Gateway for Containers inference gateway supports these workloads by integrating with the Kubernetes [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/), which adds inference-aware backends to the Gateway API model. By using the inference gateway, you can expose self-hosted model servers through Application Gateway for Containers with model-aware and load-aware routing behavior.  The inference gateway is purpose-built for serving large language models (LLMs) and other inference workloads. It routes requests based on model server signals rather than generic load balancing, which lowers time to first token (TTFT), reduces timeouts under load, and improves GPU efficiency. Built on the ingress capabilities of Application Gateway for Containers, the inference gateway also lets you pair AI workloads with capabilities like the web application firewall (WAF) to secure traffic before it reaches your model servers. @@ -12,24 +12,50 @@ Important  The Application Gateway for Containers inference gateway is currently in preview. See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability. -## What the Gateway API Inference Extension provides+## Building blocks of the inference gateway -The Gateway API Inference Extension turns a Gateway API implementation into an inference gateway by adding inference-specific backend and scheduling concepts.+An inference gateway combines three parts: a Gateway API implementation that handles ingress, an `InferencePool` backend defined by the Gateway API Inference Extension, and an endpoint picker that chooses a model server pod for each request. Application Gateway for Containers provides the first part and a managed body-based routing processor. You provide the `InferencePool` and the endpoint picker in your cluster. -The extension introduces these primary resources and components:+- **InferencePool**: A backend resource (`inference.networking.k8s.io/v1`) that represents a group of model server pods, the port they serve on, and a reference to the endpoint picker that selects a pod for each inference request.+- **Endpoint Picker (EPP)**: The role, defined by the Gateway API Inference Extension, of the component that selects an endpoint for each request. The gateway calls the EPP by using the endpoint picker protocol defined by the extension. The EPP returns the model server pod that should serve the request. EPP describes a contract, not a single product, and any implementation of the protocol can fill the role.+- **Body-based routing (BBR)**: A request processor that inspects an OpenAI-compatible request body, extracts the model name, and makes it available to the gateway as the `X-Gateway-Model-Name` header for model-aware routing. -- **InferencePool**: A backend resource that represents a group of model server pods and the endpoint picker used to select a pod for each inference request.-- **InferenceObjective**: A resource that represents request-serving objectives, such as priority, for requests that share an InferencePool.-- **Endpoint Picker (EPP)**: A customer-provided extension that runs in your cluster and implements inference scheduling. The EPP receives request metadata, scores candidate model server pods by using configurable plugins (for example, queue depth, KV cache utilization, and prefix-cache affinity), and returns the selected endpoint. Because endpoint selection runs in the EPP, the available routing behaviors depend on the scorer plugins that your EPP enables.-- **Body-based routing (BBR)**: A request processor that can inspect an OpenAI-compatible request body, extract the model name, and make it available to the gateway as the `X-Gateway-Model-Name` header for model-aware routing.+Application Gateway for Containers runs the BBR processor as a managed part of the gateway, so there's no separate body-based routing or payload-processing tier to deploy, scale, or patch. Because the inference gateway uses the Gateway API Inference Extension API, you configure it through standard Gateway API resources, and platform teams can use a Kubernetes-native API model for inference traffic instead of introducing a separate ingress configuration model. -Application Gateway for Containers runs the BBR processor as a managed part of the gateway, so there's no separate body-based routing tier to deploy, scale, or patch. It integrates with a customer-provided EPP to support request-time routing decisions. The inference gateway supports the Gateway API Inference Extension API, so you configure these capabilities through standard Gateway API resources and platform teams can use a Kubernetes-native API model for inference traffic instead of introducing a separate ingress configuration model.+### Changes in Gateway API Inference Extension v1.6++Starting with v1.6, the Gateway API Inference Extension project focuses on the core specification: the `InferencePool` API, the endpoint picker protocol, and conformance tests. Implementation-specific components moved to the [llm-d](https://github.com/llm-d/llm-d-router) project:++| Component | Before v1.6 | v1.6 and later |+| --- | --- | --- |+| `InferencePool` | Gateway API Inference Extension (`inference.networking.k8s.io/v1`) | Unchanged |+| Full-featured endpoint picker | Gateway API Inference Extension | llm-d router |+| `InferenceObjective` | Gateway API Inference Extension (`inference.networking.x-k8s.io/v1alpha2`) | llm-d router (`llm-d.ai/v1`) |+| `InferenceModelRewrite` | Gateway API Inference Extension (`inference.networking.x-k8s.io/v1alpha2`) | llm-d router (`llm-d.ai/v1alpha2`) |+| `EndpointPickerConfig` (EPP configuration file) | Gateway API Inference Extension | llm-d router |+| Body-based routing | Gateway API Inference Extension | llm-d inference payload processor. Application Gateway for Containers provides a managed BBR, so you don't need to deploy it. |+| Lightweight endpoint picker (LWEPP) | Not available | Gateway API Inference Extension, as a reference implementation |++If you created `InferenceObjective` or `InferenceModelRewrite` resources with the `inference.networking.x-k8s.io/v1alpha2` API, install the llm-d CRDs and re-create the resources with the llm-d API versions. The `InferencePool` API doesn't change.++## Endpoint picker implementations++The routing behavior you get from an `InferencePool` depends on the EPP implementation that the pool references. Application Gateway for Containers calls the EPP through the standard protocol and doesn't depend on a specific implementation. Two implementations are maintained alongside the specification:++| Implementation | Maintained by | Intended use | Scheduling capabilities |+| --- | --- | --- | --- |+| **llm-d router** | [llm-d](https://github.com/llm-d/llm-d-router) | Production inference serving | Pluggable scorers (prefix-cache affinity, queue depth, KV cache utilization, active requests), saturation detection, flow control with priority bands, `InferenceObjective`, and `InferenceModelRewrite`. |+| **Lightweight EPP (LWEPP)** | [Gateway API Inference Extension](https://github.com/kubernetes-sigs/gateway-api-inference-extension) | Reference implementation for conformance testing and protocol validation | Minimal endpoint selection. It doesn't include the llm-d router's scorer plugins, flow control, or support for `InferenceObjective` and `InferenceModelRewrite`. |++Use the llm-d router when you want the load-aware, cache-aware, and priority-aware behaviors described in this article. Use LWEPP to validate that your gateway, `InferencePool`, and model servers are wired correctly with the smallest possible EPP.++Each `InferencePool` references one EPP deployment. The llm-d router is typically installed per pool by using its Helm chart, which creates the `InferencePool`, the EPP deployment, and its configuration together.  ## How Application Gateway for Containers uses inference resources  Application Gateway for Containers continues to use standard Gateway API resources for ingress configuration. Inference behavior is activated when an `HTTPRoute` backend reference targets an `InferencePool` instead of a Kubernetes `Service`. -For non-inference routes, Application Gateway for Containers keeps the existing Gateway API behavior. Routes that target Kubernetes `Service` backends don't invoke inference processors and don't receive inference-specific routing behavior.+For non-inference routes, Application Gateway for Containers keeps the existing Gateway API behavior. Routes that target Kubernetes `Service` backends don't invoke the EPP and don't receive inference-specific routing behavior, even when the `Service` selects the same model server pods.  For inference routes, the control plane reconciles the Gateway API and inference resources and programs the data plane so that: @@ -45,19 +71,20 @@ A typical inference request follows this path. The numbered steps map to the lab  1. **Client request**: A client sends an OpenAI-compatible request to the Application Gateway for Containers frontend, and the Gateway listener accepts it. 2. **Body-based routing (BBR)**: For model-aware routes, the managed BBR processor inspects the request body, extracts the model name, and injects the `X-Gateway-Model-Name` header. The `HTTPRoute` can then match on that value to select the appropriate `InferencePool`.-3. **Endpoint selection**: When the matched route targets an `InferencePool`, Application Gateway for Containers invokes the EPP, which evaluates the request and model server telemetry and returns the selected endpoint.+3. **Endpoint selection**: When the matched route targets an `InferencePool`, Application Gateway for Containers invokes the pool's EPP. The EPP evaluates the request and model server telemetry and returns the selected endpoint. By using the llm-d router, the EPP can also hold the request in a priority queue until capacity is available, or rewrite the requested model name, before it returns an endpoint. 4. **Route to the InferencePool**: Application Gateway for Containers forwards the request to the selected model server pod, and the model server's response returns to the client through the gateway.  [![A diagram showing the Application Gateway for Containers processing a request through the BBR and making a routing decision based on the results from the EPP.](media/inference-gateway/inference-gateway-architecture.png)](media/inference-gateway/inference-gateway-architecture.png#lightbox)  ## Routing capabilities -The inference gateway supports these routing patterns for self-hosted AI workloads. Endpoint selection runs in the EPP, so the load-aware and cache-aware behaviors depend on the scorer plugins that your EPP enables.+The inference gateway supports these routing patterns for self-hosted AI workloads. Application Gateway for Containers provides model-aware routing, traffic splitting, and failure handling. Endpoint selection, priority, and model rewrites run in the EPP, so those behaviors depend on the EPP implementation and the plugins and features that you enable in it.  - **Model-aware routing**: Route requests based on the model name in OpenAI-compatible request bodies, which the managed BBR processor extracts.-- **Traffic splitting and rollouts**: Use standard `HTTPRoute` weighted `backendRefs` to split traffic across `InferencePool` backends for canary or blue-green model rollouts.-- **Load-aware and cache-aware endpoint selection**: The EPP scores endpoints by using model server telemetry, such as queue depth and KV cache utilization. When the EPP enables prefix-cache-aware scoring, requests that share a prompt prefix go to the same replica to raise cache hits and lower TTFT.-- **Request priority and overload protection**: Use `InferenceObjective` resources and the `x-gateway-inference-objective` request header to assign serving priority. When model servers are saturated, the EPP sheds lower-priority requests first to protect latency-critical traffic.+- **Traffic splitting and rollouts across pools**: Use standard `HTTPRoute` weighted `backendRefs` to split traffic across `InferencePool` backends for canary or blue-green rollouts of a model that runs on separate pods.+- **Model rewrites within a pool**: With the llm-d router, use `InferenceModelRewrite` resources to map the model name that clients request to one or more target models served by the same pool, with weights. For example, clients keep requesting `support-assistant` while the EPP sends 90 percent of requests to the `support-v1` LoRA adapter and 10 percent to `support-v2`. The router returns the client's requested model name in the response.+- **Load-aware and cache-aware endpoint selection**: The EPP scores endpoints by using model server telemetry, such as queue depth and KV cache utilization. When the EPP enables prefix-cache-aware scoring, requests that share a prompt prefix, such as turns of the same conversation, go to the same replica to raise cache hits and lower TTFT.+- **Request priority and overload protection**: With the llm-d router, define `InferenceObjective` resources (`llm-d.ai/v1`) that assign a priority to a class of traffic on a pool, and have clients select an objective with the `x-llm-d-inference-objective` request header. When flow control is enabled and the pool is saturated, the EPP queues requests in priority bands, dispatches higher-priority requests first, and sheds lower-priority requests when their band is full or their time-to-live expires. - **Resilient endpoint selection**: Configure EPP failure behavior with `FailOpen` or `FailClose`, depending on whether availability or strict endpoint selection is more important for a pool. - **Gateway API compatibility**: Continue using Gateway, HTTPRoute, ReferenceGrant, and standard Gateway API status conditions for ingress configuration. @@ -65,15 +92,16 @@ The inference gateway supports these routing patterns for self-hosted AI workloa  Keep model servers private behind the managed gateway and apply the platform's security capabilities to inference traffic. -- **Web application firewall (WAF)**: The inference gateway pairs natively with the existing [WAF capability in Application Gateway for Containers](how-to-waf-gateway-api), applying OWASP-aligned protections to AI traffic before requests reach your model servers.-- **Protection for expensive backends**: Because policy is enforced at the managed edge, malformed or abusive requests can be inspected and blocked before they consume scarce GPU capacity.+- **Web application firewall (WAF)**: The inference gateway pairs natively with the existing [WAF capability in Application Gateway for Containers](how-to-waf-gateway-api), applying protections to AI traffic before requests reach your model servers.+- **Protection for expensive backends**: Because policy is enforced at the managed edge, malformed or abusive requests can be inspected and blocked before they reach the EPP or consume scarce GPU capacity.  ## Example scenarios  The following scenarios show common ways to use the inference gateway: -- **Serve and roll out model versions**: Route OpenAI-compatible requests to an `InferencePool` by model name, then use weighted `HTTPRoute` backendRefs to shift a percentage of traffic to a new model version for canary testing before you complete the rollout.-- **Prioritize latency-critical traffic**: Define `InferenceObjective` resources for high-priority and low-priority workloads on a shared pool. Interactive chat requests carry the high-priority objective, while batch jobs use a lower priority that's shed first when the pool is saturated.+- **Serve and roll out model versions**: Route OpenAI-compatible requests to an `InferencePool` by model name. To roll out a new LoRA adapter on the same pods, use an `InferenceModelRewrite` to shift a percentage of requests to the new adapter. To roll out a model that runs on separate pods, use weighted `HTTPRoute` backendRefs across two `InferencePool` backends.+- **Reuse cached context for multi-turn conversations**: Enable prefix-cache-aware scoring in the llm-d router so that follow-up turns of a conversation return to the replica that already holds its KV cache, which reduces prefill work and TTFT.+- **Prioritize latency-critical traffic**: Define `InferenceObjective` resources for high-priority and low-priority workloads on a shared pool. Interactive chat requests carry the high-priority objective, while batch jobs use a lower priority that waits for spare capacity and is shed first when the pool is saturated.  ## InferencePool compared with Service @@ -84,16 +112,18 @@ A Kubernetes `Service` remains the right backend abstraction for standard applic | `Service` | Standard HTTP, gRPC, and application backends | Gateway routes to service endpoints using standard load-balancing behavior. | | `InferencePool` | Self-hosted model server pods | Gateway invokes the configured EPP, then routes to the selected model server endpoint. | -An `InferencePool` includes a pod selector, target port information, and an endpoint picker reference. The EPP is responsible for selecting the endpoint for a request. A single EPP is associated with a single pool.+An `InferencePool` includes a pod selector, target port information, and an endpoint picker reference (`endpointPickerRef`). The EPP selects the endpoint for a request. A single EPP is associated with a single pool.  ## Failure behavior  Inference routing is request-time behavior, so the failure mode of the endpoint picker matters. -`InferencePool` endpoint picker references support these failure modes:+The `endpointPickerRef` of an `InferencePool` supports these failure modes: -- **FailOpen**: If the EPP is unavailable or doesn't respond, the gateway can continue with standard endpoint selection for the pool. This choice preserves availability but might reduce routing quality.-- **FailClose**: If the EPP is unavailable or doesn't respond, the gateway rejects the request. This choice prevents traffic from being sent without the required endpoint selection decision.+- **FailOpen**: If the EPP is unavailable or doesn't respond, the gateway continues with standard endpoint selection for the pool. This choice preserves availability but loses EPP behaviors such as cache affinity, priority, and model rewrites until the EPP recovers.+- **FailClose**: If the EPP is unavailable or doesn't respond, the gateway rejects the request. This choice prevents traffic from being sent without the required endpoint selection decision. `FailClose` is the default.++If your pool relies on `InferenceModelRewrite` to map a client-facing model name to a model the servers actually load, requests that bypass the EPP under `FailOpen` reach the model server with the original model name. Ensure the model servers can serve that name, or use `FailClose` for the pool.  For model-aware routing that depends on request body parsing, Application Gateway for Containers favors correctness. If the model name can't be extracted for a route that requires BBR, the request is rejected instead of being silently routed to the wrong model backend. @@ -101,15 +131,16 @@ For model-aware routing that depends on request body parsing, Application Gatewa  Plan for the following considerations when running inference workloads behind Application Gateway for Containers: -- **EPP capacity**: The EPP is on the request path for `InferencePool` backends. Size and monitor it like a critical application component.-- **Model server telemetry**: The EPP needs fresh model server metrics to make high-quality routing decisions. Confirm that your model server supports the metrics expected by your EPP configuration.-- **GPU capacity and scheduling**: Model server pods often require GPU node pools, device plugins, and model download credentials.+- **EPP capacity**: The EPP is on the request path for `InferencePool` backends. Size and monitor it like a critical application component, and run more than one replica for production pools.+- **EPP configuration**: With the llm-d router, the scorer plugins, their weights, saturation detection, and flow control are set in the router's `EndpointPickerConfig`, usually through Helm values. Some capabilities, such as flow control, are behind feature gates and are off by default.+- **Model server telemetry**: The EPP needs fresh model server metrics to make high-quality routing decisions. Confirm that your model server exposes the metrics that your EPP configuration expects.+- **GPU capacity and scheduling**: Model server pods often require GPU node pools, device plugins, and model download credentials. The gateway and the EPP don't require GPUs. - **Autoscaling**: Scale model server pods on inference signals such as queue depth and KV cache utilization by using the Horizontal Pod Autoscaler or KEDA. Autoscaling adds capacity as demand grows, while the EPP routes around replicas that are already saturated. - **Streaming responses**: Many chat completion workloads use server-sent events. Validate streaming behavior when testing end-to-end latency and timeout settings.-- **Observability**: Monitor Gateway and HTTPRoute status, InferencePool status, EPP health, model server readiness, and model server metrics such as active requests and queue depth.+- **Observability**: Monitor Gateway and HTTPRoute status, InferencePool status, EPP health, model server readiness, and model server metrics such as active requests and queue depth. The llm-d router also exposes Prometheus metrics for its scheduling, flow control, and model rewrite decisions.  ## Limitations -The inference gateway focuses on self-hosted inference workloads running on Kubernetes. Routing directly to public or managed model provider endpoints is outside the scope of this integration.--The Gateway API Inference Extension evolves independently from Application Gateway for Containers. Use API versions and manifests that match the inference extension CRDs installed on your cluster.+- The inference gateway focuses on self-hosted inference workloads running on Kubernetes. Routing directly to public or managed model provider endpoints is outside the scope of this integration.+- JWT validation defined by an `AuthPolicy` isn't currently supported on routes whose backend is an `InferencePool`.+- The Gateway API Inference Extension and the llm-d router evolve independently from Application Gateway for Containers and from each other. Use API versions and manifests that match the CRDs installed on your cluster: the `InferencePool` CRD from the Gateway API Inference Extension, and the `InferenceObjective` and `InferenceModelRewrite` CRDs from the llm-d router release that you deploy.