I spotted this one via the AKS Docs Tracker I built and run here on pixelrobots.co.uk. It picked up new docs appearing under the AKS section and this one caught my eye immediately. Microsoft has shipped a public preview of something called Container Network Insight Agent, and it is genuinely interesting enough to stop and write about.
Put simply, it is an AI-powered network diagnostics assistant that runs as a pod inside your AKS cluster. You describe a networking problem in plain English, and it collects evidence from your cluster using kubectl, cilium, and hubble, and returns a structured report with root cause analysis and actual remediation commands. Not suggestions. Commands you can copy and run.
AKS networking issues are some of the most painful to debug. They can span from Kubernetes network policies all the way down to kernel ring buffers and NIC-level drops, across multiple tools and layers. Anything that brings that into a single interface is worth paying attention to.
What it actually does
Container Network Insight Agent covers three main diagnostic categories.
DNS troubleshooting checks CoreDNS pod health, endpoint registration, configuration (including custom ConfigMaps), NodeLocal DNS status, and network policies that might be silently blocking DNS traffic. It also evaluates Cilium FQDN egress restrictions which can cause hard-to-trace external resolution failures.
Packet drop analysis is the one that stood out to me. It deploys a lightweight debug DaemonSet to each node to collect host-level network statistics and uses delta measurements to separate active drops from historical counters. It checks NIC ring buffer utilisation via ethtool, kernel softnet stats from /proc/net/softnet_stat, per-CPU SoftIRQ distribution, socket buffer saturation, and CNI-specific interface data. This is the kind of thing that normally requires SSH-ing into nodes and running commands manually one at a time.
Kubernetes networking diagnostics covers pod-to-service connectivity, service endpoint registration, network policy conflicts (both Kubernetes NetworkPolicy and CiliumNetworkPolicy), and Hubble flow analysis. A common catch here is a targetPort mismatch between the service and pod. This produces connection timeouts even when endpoints look healthy, and the agent flags it directly.
Each diagnostic returns a structured report with an evidence table, root cause identification, and specific remediation commands. The agent itself does not make any changes to your cluster. It is advisory only, with read-only cluster access.
Worth flagging upfront is that this is a cloud-only AKS feature. It does not work on AKS hybrid, AKS on Azure Stack HCI, or Arc-enabled Kubernetes clusters.
What you need before you start
This setup has more moving parts than the average AKS preview feature, so it is worth laying out the requirements clearly before jumping in.
Tools required: Azure CLI 2.77.0 or later, kubectl, jq, git, and the k8s-extension Azure CLI extension. The jq requirement matters here because the install scripts are Bash-based. This is not a PowerShell-friendly setup. Everything in this walkthrough runs in Bash.
Azure permissions: Contributor and User Access Administrator on the target resource group, plus access to create Azure OpenAI resources and Entra ID App Registrations.
Cluster requirements: workload identity and OIDC issuer enabled, a supported Kubernetes version, and minimum node size of Standard_D4_v3 with three nodes recommended. Your cluster also needs outbound HTTPS access to your Azure OpenAI endpoint on port 443, and must be able to pull images from acnpublic.azurecr.io.
Supported regions: centralus, eastus, eastus2, uksouth, westus2. If your cluster is not in one of these, you are blocked for now.
Azure OpenAI: you need a deployed model. GPT-4o or later is recommended. If you do not already have one, the walkthrough below covers creating it.
The agent works on clusters without Cilium or ACNS, but with reduced capabilities. Without Advanced Container Networking Services (ACNS) you lose Hubble flow analysis and Cilium policy diagnostics, but DNS, packet drop, and standard Kubernetes networking diagnostics all still work. If you are running Azure CNI powered by Cilium with ACNS enabled, you get the full feature set including Hubble flow observation.
Setting it up
The setup is eight steps. I will walk through each one. All commands are Bash.
Step 1: Set variables and create the resource group
Set your environment variables first. Keeping them all in one place makes the subsequent steps much cleaner.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
export SUBSCRIPTION_ID="<your-subscription-id>" export LOCATION="uksouth" # Supported regions: centralus, eastus, eastus2, uksouth, westus2 export RESOURCE_GROUP="rg-pixelrobots-cna" export CLUSTER_NAME="aks-pixelrobots-cna" export OPENAI_SERVICE_NAME="pixelrobots-openai" export OPENAI_DEPLOYMENT_NAME="pixelrobots-gpt4o" export OPENAI_MODEL_NAME="gpt-4o" export OPENAI_MODEL_VERSION="2024-11-20" export SKU_CAPACITY="50" az login az account set --subscription $SUBSCRIPTION_ID az group create \ --name $RESOURCE_GROUP \ --location $LOCATION |
Once the resource group is created you should see a JSON response confirming the provisioningState is Succeeded.
Step 2: Create the AKS cluster
If you already have a cluster, the key requirements are workload identity and OIDC issuer enabled. If you need to create one, the following gives you Cilium dataplane and ACNS for full diagnostic capabilities.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
az aks create \ --resource-group $RESOURCE_GROUP \ --name $CLUSTER_NAME \ --location $LOCATION \ --node-count 3 \ --node-vm-size Standard_D4_v3 \ --network-plugin azure \ --network-plugin-mode overlay \ --network-dataplane cilium \ --enable-acns \ --enable-oidc-issuer \ --enable-workload-identity \ --generate-ssh-keys |
This takes a few minutes. Once it finishes, you will see the cluster details returned as JSON.
If you have an existing cluster without workload identity and OIDC issuer, enable them with:
|
1 2 3 4 5 |
az aks update \ --resource-group $RESOURCE_GROUP \ --name $CLUSTER_NAME \ --enable-oidc-issuer \ --enable-workload-identity |
The update will complete with no output on success.
Then pull down the credentials so kubectl can talk to the cluster:
|
1 2 3 4 |
az aks get-credentials \ --resource-group $RESOURCE_GROUP \ --name $CLUSTER_NAME \ --overwrite-existing |
You should see Merged "aks-pixelrobots-cna" as current context confirming the kubeconfig is ready.
Step 3: Create an Azure OpenAI resource and deploy a model
Skip this step if you already have an Azure OpenAI resource with a deployed model. Just export OPENAI_SERVICE_NAME, OPENAI_DEPLOYMENT_NAME, and retrieve the endpoint:
|
1 2 3 4 |
export AZURE_OPENAI_ENDPOINT=$(az cognitiveservices account show \ --name $OPENAI_SERVICE_NAME \ --resource-group $RESOURCE_GROUP \ --query "properties.endpoint" -o tsv) |
Once AZURE_OPENAI_ENDPOINT is set, skip ahead to Step 4.
If you need to create one from scratch:
|
1 2 3 4 5 6 7 |
az cognitiveservices account create \ --name $OPENAI_SERVICE_NAME \ --resource-group $RESOURCE_GROUP \ --location $LOCATION \ --kind "OpenAI" \ --sku "S0" \ --custom-domain $OPENAI_SERVICE_NAME |
The command returns immediately but the resource is not yet ready. Wait for provisioning, then deploy the model. The loop below polls every 10 seconds rather than using a fixed sleep.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
echo "Waiting for Azure OpenAI resource to provision..." while true; do STATE=$(az cognitiveservices account show \ --name $OPENAI_SERVICE_NAME \ --resource-group $RESOURCE_GROUP \ --query "properties.provisioningState" -o tsv 2>/dev/null) if [[ "$STATE" == "Succeeded" ]]; then echo "Provisioned." break fi echo " State: ${STATE:-Creating}. Retrying in 10s..." sleep 10 done az cognitiveservices account deployment create \ --name $OPENAI_SERVICE_NAME \ --resource-group $RESOURCE_GROUP \ --deployment-name $OPENAI_DEPLOYMENT_NAME \ --model-name $OPENAI_MODEL_NAME \ --model-version $OPENAI_MODEL_VERSION \ --model-format OpenAI \ --sku-name "GlobalStandard" \ --sku-capacity "$SKU_CAPACITY" export AZURE_OPENAI_ENDPOINT=$(az cognitiveservices account show \ --name $OPENAI_SERVICE_NAME \ --resource-group $RESOURCE_GROUP \ --query "properties.endpoint" -o tsv) echo "OpenAI Endpoint: $AZURE_OPENAI_ENDPOINT" |
With AZURE_OPENAI_ENDPOINT now set, you have everything you need from the OpenAI side. Move on to the managed identity.
Steps 4 and 5: Create a managed identity and assign roles
The agent authenticates to Azure services using a user-assigned managed identity with workload identity federation.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
export IDENTITY_NAME="pixelrobots-cna-identity" az identity create \ --name $IDENTITY_NAME \ --resource-group $RESOURCE_GROUP \ --location $LOCATION export IDENTITY_CLIENT_ID=$(az identity show \ --name $IDENTITY_NAME \ --resource-group $RESOURCE_GROUP \ --query "clientId" -o tsv) export IDENTITY_PRINCIPAL_ID=$(az identity show \ --name $IDENTITY_NAME \ --resource-group $RESOURCE_GROUP \ --query "principalId" -o tsv) |
You should see both IDENTITY_CLIENT_ID and IDENTITY_PRINCIPAL_ID printed. You will need both in the next commands.
The identity needs four roles: Cognitive Services OpenAI User on the OpenAI resource, Azure Kubernetes Service Cluster User Role and Azure Kubernetes Service Contributor Role on the cluster, and Reader on the resource group.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
export OPENAI_RESOURCE_GROUP="${OPENAI_RESOURCE_GROUP:-$RESOURCE_GROUP}" az role assignment create \ --assignee $IDENTITY_PRINCIPAL_ID \ --role "Cognitive Services OpenAI User" \ --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$OPENAI_RESOURCE_GROUP/providers/Microsoft.CognitiveServices/accounts/$OPENAI_SERVICE_NAME" az role assignment create \ --assignee $IDENTITY_PRINCIPAL_ID \ --role "Azure Kubernetes Service Cluster User Role" \ --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.ContainerService/managedClusters/$CLUSTER_NAME" az role assignment create \ --assignee $IDENTITY_PRINCIPAL_ID \ --role "Azure Kubernetes Service Contributor Role" \ --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.ContainerService/managedClusters/$CLUSTER_NAME" az role assignment create \ --assignee $IDENTITY_PRINCIPAL_ID \ --role "Reader" \ --scope "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP" |
Role assignments can take up to 10 minutes to propagate. If you see 401 or 403 errors from the agent pod shortly after install, wait a few minutes and restart the pod before debugging further.
Step 6: Configure federated credentials
Link the managed identity to the Kubernetes service account the agent uses:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
export OIDC_ISSUER_URL=$(az aks show \ --resource-group $RESOURCE_GROUP \ --name $CLUSTER_NAME \ --query "oidcIssuerProfile.issuerUrl" -o tsv) az identity federated-credential create \ --name "pixelrobots-cna-fed-cred" \ --identity-name $IDENTITY_NAME \ --resource-group $RESOURCE_GROUP \ --issuer $OIDC_ISSUER_URL \ --subject "system:serviceaccount:kube-system:container-networking-agent-reader" \ --audiences "api://AzureADTokenExchange" |
This returns the federated credential resource as JSON with a name and issuer field. Confirm the issuer URL matches your cluster’s OIDC endpoint before moving on.
Step 7: Create an App Registration for Entra ID authentication
This is required for production deployments. For development and testing only you can use simple username login, but for anything resembling a real environment you want Entra ID.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
export APP_DISPLAY_NAME="pixelrobots-cna-oauth2-auth" export APP_REDIRECT_URI="http://localhost:8080/auth/callback" export APP_CLIENT_ID=$(az ad app create \ --display-name $APP_DISPLAY_NAME \ --web-redirect-uris $APP_REDIRECT_URI \ --sign-in-audience AzureADMyOrg \ --enable-id-token-issuance true \ --enable-access-token-issuance true \ --query "appId" -o tsv) export TENANT_ID=$(az account show --query tenantId -o tsv) echo "App Client ID: $APP_CLIENT_ID" echo "Tenant ID: $TENANT_ID" |
Both values should be printed. Keep them handy as you will need them in the extension install step.
If your tenant requires a SERVICE_MANAGEMENT_REFERENCE, add --service-management-reference $SERVICE_MANAGEMENT_REFERENCE to the az ad app create command above.
Add the required Microsoft Graph delegated permissions (openid, profile, User.Read, offline_access):
|
1 2 3 4 5 6 7 8 |
az ad app permission add \ --id $APP_CLIENT_ID \ --api 00000003-0000-0000-c000-000000000000 \ --api-permissions \ e1fe6dd8-ba31-4d61-89e7-88639da4683d=Scope \ 14dad69e-099b-42c9-810b-d002981feec1=Scope \ 37f7f235-527c-4136-accd-4a02d197296e=Scope \ 7427e0e9-2fba-42fe-b0c0-848c9e6a8182=Scope |
This produces no output on success. If your tenant requires admin consent, run az ad app permission admin-consent --id $APP_CLIENT_ID before continuing.
Then add a federated credential on the App Registration itself. This is separate from the managed identity federated credential in step 6, and links the app registration to the same Kubernetes service account:
|
1 2 3 4 5 6 7 8 |
az ad app federated-credential create \ --id $APP_CLIENT_ID \ --parameters "{ \"name\": \"pixelrobots-cna-app-fed-cred\", \"issuer\": \"$OIDC_ISSUER_URL\", \"subject\": \"system:serviceaccount:kube-system:container-networking-agent-reader\", \"audiences\": [\"api://AzureADTokenExchange\"] }" |
Note that only localhost redirect URIs are currently supported. Public LoadBalancer URLs are not supported for redirect URIs, which means you access the interface via port-forward even in a more permanent setup.
Step 8: Install the extension
This is where the ACNS and non-ACNS paths diverge.
For clusters with ACNS and Cilium dataplane (full capabilities):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
export AZURE_OPENAI_DEPLOYMENT=$OPENAI_DEPLOYMENT_NAME az k8s-extension create \ --cluster-name $CLUSTER_NAME \ --resource-group $RESOURCE_GROUP \ --cluster-type managedClusters \ --name containernetworkingagent \ --extension-type microsoft.containernetworkingagent \ --release-train stable \ --auto-upgrade-minor-version false \ --scope cluster \ --configuration-settings config.AZURE_CLIENT_ID=$IDENTITY_CLIENT_ID \ --configuration-settings config.AZURE_TENANT_ID=$TENANT_ID \ --configuration-settings config.AZURE_SUBSCRIPTION_ID=$SUBSCRIPTION_ID \ --configuration-settings config.AKS_CLUSTER_NAME=$CLUSTER_NAME \ --configuration-settings config.AKS_RESOURCE_GROUP=$RESOURCE_GROUP \ --configuration-settings config.ENTRA_TENANT_ID=$TENANT_ID \ --configuration-settings config.ENTRA_CLIENT_ID=$APP_CLIENT_ID \ --configuration-settings config.AZURE_OPENAI_DEPLOYMENT=$AZURE_OPENAI_DEPLOYMENT \ --configuration-settings config.AZURE_OPENAI_API_VERSION=2025-03-01-preview \ --configuration-settings config.AZURE_OPENAI_ENDPOINT=$AZURE_OPENAI_ENDPOINT \ --configuration-settings config.AKS_MCP_ENABLED_COMPONENTS=kubectl \ --configuration-settings config.DISABLE_TELEMETRY=false |
The command returns immediately and the extension provisions in the background. Skip the next command and go straight to verifying the extension state.
For clusters without ACNS (Hubble disabled):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
az k8s-extension create \ --cluster-name $CLUSTER_NAME \ --resource-group $RESOURCE_GROUP \ --cluster-type managedClusters \ --name containernetworkingagent \ --extension-type microsoft.containernetworkingagent \ --scope cluster \ --release-train stable \ --configuration-settings config.AZURE_CLIENT_ID=$IDENTITY_CLIENT_ID \ --configuration-settings config.AZURE_TENANT_ID=$TENANT_ID \ --configuration-settings config.AZURE_SUBSCRIPTION_ID=$SUBSCRIPTION_ID \ --configuration-settings config.AKS_CLUSTER_NAME=$CLUSTER_NAME \ --configuration-settings config.AKS_RESOURCE_GROUP=$RESOURCE_GROUP \ --configuration-settings config.ENTRA_TENANT_ID=$TENANT_ID \ --configuration-settings config.ENTRA_CLIENT_ID=$APP_CLIENT_ID \ --configuration-settings config.AZURE_OPENAI_DEPLOYMENT=$AZURE_OPENAI_DEPLOYMENT \ --configuration-settings config.AZURE_OPENAI_API_VERSION=2025-03-01-preview \ --configuration-settings config.AZURE_OPENAI_ENDPOINT=$AZURE_OPENAI_ENDPOINT \ --configuration-settings hubble.enabled=false \ --configuration-settings config.AKS_MCP_ENABLED_COMPONENTS=kubectl |
As with the ACNS variant, the extension provisions in the background. Check the state with the following command:
Verify the extension installed correctly:
|
1 2 3 4 5 6 |
az k8s-extension show \ --cluster-name $CLUSTER_NAME \ --resource-group $RESOURCE_GROUP \ --cluster-type managedClusters \ --name containernetworkingagent \ --query "{provisioningState:provisioningState, version:version}" -o table |
The provisioningState should return Succeeded. You can also verify the Kubernetes service account was created with the right annotation:
|
1 |
kubectl get serviceaccount container-networking-agent-reader -n kube-system -o yaml |
Check that azure.workload.identity/client-id matches your managed identity client ID.
Accessing it
Once installed, port-forward to reach the interface:
|
1 |
kubectl port-forward svc/container-networking-agent-service -n kube-system 8080:80 |
Open http://localhost:8080 in your browser. You’ll be prompted to sign in with either your simple username (if you configured development mode) or Microsoft Entra ID credentials.
Once in, type your networking questions in plain English. The agent classifies the issue, collects evidence from the cluster, and returns a structured report. A few things worth knowing about the interface. Sessions time out after 30 minutes of inactivity and have an absolute 8-hour limit. Chat history lives in memory only and is lost if the pod restarts. At around 15 exchanges, the agent starts summarising older messages, so for very long sessions you may notice it losing context on earlier findings.
Things to watch for
The packet drop diagnostic deploys a debug DaemonSet called rx-troubleshooting-debug to the kube-system namespace. It needs hostNetwork, hostPID, hostIPC, and NET_ADMIN capabilities to collect host-level network stats. The DaemonSet is shared across sessions and cleaned up automatically, but if the agent pod crashes during a diagnostic it can be left behind. Clean it up manually if needed:
|
1 |
kubectl delete ds rx-troubleshooting-debug -n kube-system |
On larger clusters, packet drop diagnostics become slower and more resource-intensive. The docs recommend limiting concurrent users to three on a 25-node cluster and to one on a 50-node cluster. For most teams this is fine, but something to be aware of if you are planning on making this available to multiple engineers simultaneously.
If your extension install fails, the most common causes are an unsupported region, workload identity or OIDC issuer not enabled, or insufficient permissions. Running az k8s-extension create a second time on an already-installed extension will error. Use az k8s-extension update to change configuration settings.
When debugging startup issues, pod logs tell you most of what you need to know:
|
1 |
kubectl logs -n kube-system -l app=container-networking-agent --tail=200 |
Look for any lines referencing Missing required, 401, 403, or bootstrap.validation_agent_failed as these point to the most common setup problems.
Quick health check once the pod is running:
|
1 2 |
kubectl port-forward svc/container-networking-agent-service -n kube-system 8080:80 curl -s http://localhost:8080/ready | jq |
The /ready endpoint returning HTTP 200 means at least one pre-warmed agent is initialised and ready to handle requests. The first query after a pod restart typically takes 10 to 30 seconds as the warmup pool initialises. Subsequent queries are faster.
For common error patterns, the table below covers the main ones:
| Error | Likely cause | Fix |
|---|---|---|
401 Unauthorized / 403 Forbidden | Missing RBAC role or workload identity misconfiguration | Check roles with az role assignment list --assignee <principal-id> --all -o table |
RuntimeError: Missing required Azure OpenAI environment variable(s) | ConfigMap has placeholder values | Run az k8s-extension update with the correct settings |
FailedMount / volume mount error | Missing Hubble certificate secrets | Deploy with hubble.enabled=false or ensure ACNS is enabled |
Pod stuck in CrashLoopBackOff | Azure OpenAI unreachable or misconfigured | Check the endpoint URL, deployment name, and outbound HTTPS on port 443 |
redirect_uri mismatch on Entra login | Redirect URI not set to http://localhost:8080/auth/callback | Update the App Registration in the Azure portal |
My thoughts
Honestly, I think the concept here is really strong. AKS networking issues are some of the most time-consuming problems to debug, precisely because they do not live in one place. You bounce between kubectl, cilium, node SSH sessions, Azure Monitor, and your own institutional knowledge hoping something connects. The idea of describing the problem in plain English and getting a structured, evidence-backed report with actual commands to run is the right direction.
What impresses me most is the packet drop diagnostic. Getting host-level network stats like NIC ring buffer state and kernel softnet counters out of an AI-driven chat interface, with delta measurements to distinguish active drops from historical noise, is genuinely non-trivial. That is work that used to take an experienced engineer a fair chunk of an afternoon.
The current limitations are real but predictable for a public preview. Port-forward only access, in-memory session state, localhost redirect URIs, and a short supported region list are the kind of rough edges that tend to get addressed before GA. The Azure OpenAI dependency adds cost and some infra overhead, which is worth factoring in if you are thinking about this for a team.
This feature is also a really compelling reason to look seriously at Advanced Container Networking Services if you have not already. ACNS brings you Hubble flow analysis, Cilium policy diagnostics, FQDN-based egress filtering, Layer 7 network policy, deep DNS observability, and now Container Network Insight Agent on top of all of that. The cost is around $0.025 per node per hour. On a three-node cluster that is around $54 a month. On a ten-node cluster you are looking at roughly $180 a month. For most teams running AKS in production that is a reasonable price for the observability and security capabilities you get, and Container Network Insight Agent makes the value easier to demonstrate by turning that data into actionable diagnostics you can actually use when something breaks at 2am.
If you are running ACNS today, I would set this up in a non-production cluster now and get familiar with it. The full diagnostic set including Hubble flow analysis and Cilium policy diagnostics makes it significantly more capable than the baseline. If you are not running ACNS, it is still worth a look for the DNS and packet drop coverage alone, and it might just be the nudge you needed to enable ACNS properly.
Give it a try and let me know what you think in the comments. And if you want to stay on top of new AKS features like this as they land in the docs, the AKS Docs Tracker is how I caught this one.
0 Comments