AKS Flex Nodes Are in Preview: Attaching Your Own Compute to a Managed Cluster

Reading Time: 18 minutes

Microsoft has added flex nodes to Azure Kubernetes Service (AKS), and it’s one of the more interesting preview features I’ve seen land in a while. I picked it up through the AKS docs, had to read it twice before it clicked, and then stood the whole thing up end to end on a real cluster with an Azure VM as the flex host to see how it actually behaves. The short version is that you can now attach your own virtual machines and bare metal hosts to a managed AKS cluster and run them as real Kubernetes nodes, without those machines ever becoming a standard AKS node pool.

If you’ve ever needed compute that AKS couldn’t give you in the right region, on the right hardware, or in the right building, this is worth paying attention to. This is a long one. I want to go past the announcement and into how the reconciliation model actually works, how identity and networking hang together, what the day-two lifecycle looks like once hosts are attached, how you upgrade and remove them cleanly, who owns what under the support policy, and where the preview edges are sharp enough to draw blood. If you just want the headline, flex nodes let AKS schedule onto machines you own. If you want to run it, read on.

Why this matters

A standard AKS node pool is Azure virtual machines that AKS provisions, manages, and owns. That model is great until you hit one of its walls. You want capacity in a region where AKS has no quota. You have bare metal sat in your own data centre that you’d rather not throw away. You have a data residency rule that says certain jobs run on your infrastructure, full stop.

Flex nodes let the AKS control plane schedule onto machines you own and manage, while you keep control of the underlying hardware. You get Kubernetes lifecycle operations through AKS, and you keep the box. That’s a genuinely different shape to anything AKS has offered before, and it puts AKS into edge, hybrid, and bring-your-own-hardware territory that used to mean Arc-enabled Kubernetes or a self-managed cluster.

To be clear up front, this is a public preview. It’s opt-in, excluded from the SLA, and not meant for production. You also need a subscription that’s approved for the flex nodes preview, so this isn’t just a feature flag you flip and go.

What a flex node actually is

A flex node runs on a VM or bare metal host that you manage. A flex node agent runs on that host, prepares it, joins it to your AKS cluster, and then keeps the node configuration aligned with the state AKS asks for.

The part I like is that the agent runs the node in an isolated environment on the host. That means AKS can update the node without replacing the underlying machine. On a normal node pool, a change means a new VM. Here, the machine stays and the node environment on top of it gets reconciled.

There are five moving parts worth knowing before you start.

• User-managed host is the VM or bare metal box that provides the compute and runs the agent.
• Flex node agent is the software that bootstraps the node, joins the cluster, and reconciles configuration. It is open source in the AKSFlexNode repository, which is a nice touch for something you are running on your own hardware.
• AKS cluster is the control plane, scheduling across both standard node pools and your flex nodes.
• ARM machine resource is an Azure Resource Manager resource that stores the authoritative desired state for a single flex node, including its target Kubernetes version.
• AKS management APIs are the lifecycle operations you use to manage the pool.

That ARM machine resource is the bit that makes this feel like Azure rather than a bolt-on. The desired state for each host lives in Azure Resource Manager, and the agent on the host reconciles towards it.

The reconciliation loop is the whole trick

It is worth slowing down on how this actually works, because it is not how a normal node pool behaves and it explains most of the later operational quirks.

With a Virtual Machine Scale Sets node pool, the source of truth is the scale set model, and a change to the model rolls VMs. With flex nodes, the source of truth for one node is its Azure Machine resource. You express intent by writing to that Machine resource through the AKS APIs, for example a new target Kubernetes version. The agent on the host watches for the desired state, then reconciles the local node environment towards it.

The agent does that reconciliation inside an isolated worker environment on the host rather than against the host operating system directly. That isolation shows up as a systemd-nspawn machine you can see with machinectl on the host. On the box I attached, it registered as a machine called kube1 sitting alongside the base OS. The upshot is that the agent can rebuild or re-version the node without touching your base OS install, your other packages, or anything else you run on that box. Your machine is a stable substrate, and the Kubernetes node is a disposable layer on top.

That separation is genuinely clever, and it is also the mental model you need for everything that follows. When you upgrade a flex node later, you are not reimaging a VM. You are asking the agent to reconcile the node layer to a new version while the host underneath stays exactly where it is.

Host, Machine, and Node are three different things

This tripped me up early, so I am pulling it out on its own. A single flex node is represented by three names that are not interchangeable.

• The host has a Linux hostname, the one hostname returns over SSH.
• The Azure Machine is the ARM resource, with a name recorded during attachment.
• The Kubernetes Node is the object kubectl get nodes shows.

The docs warn you explicitly not to assume the Machine name equals the host name equals the Node name, and the management commands lean on that distinction. Some operations target the Azure Machine by its Machine name through az aks machine, and others target the Kubernetes Node by its node name through kubectl. When I attached an Azure VM, all three came out identical, which lulls you into thinking they’re the same thing. They don’t have to be. The agent lowercases the host name to form the node name, so WEB-01 registers as web-01, and a custom Machine name can diverge from both. If you script this and casually reuse one variable for all three, you will eventually point a drain or a delete at the wrong object. Record all three during attach and keep them separate. Future you will be grateful.

The identity model, and why there are two

Because AKS did not create the host, the cluster has no built-in reason to trust a connection request coming from it. So a flex node needs an identity to prove itself to Azure before it can register and join.

Flex nodes give you three ways to do that, and which one you pick depends entirely on where the host lives.

Authentication method	When to use it
System-assigned or user-assigned managed identity	The host is an Azure VM, so Azure can hand it a secretless identity directly.
Azure Arc managed identity	The host is off-Azure, but Arc extends Azure’s trust to it, so it can also get a secretless identity.
Service principal	Azure has no existing relationship with the host, so you create and manage an application identity yourself.

The detail that caught me out is that the agent actually uses two separate identities, each with limited permissions. One talks to Azure to read the node’s desired configuration and report status. The other talks to Kubernetes to monitor the node and to remove it from the cluster after a reset or delete. If you are used to a single service account doing everything, keep that split in mind when you are reasoning about least privilege.

A couple of finer points are worth having straight before you pick. If your host is an Azure VM, you choose between a system-assigned managed identity, which is tied to that one VM’s lifecycle and disappears when the VM does, and a user-assigned managed identity, which is its own resource you can reuse across machines or keep when you replace a VM. For anything off-Azure that you can Arc-enable, the Arc managed identity gives you the same secretless story as an Azure VM, but Arc enrolment is a genuine prerequisite and not something you bolt on mid-attach. If you are forced onto a service principal, prefer a certificate credential over a long-lived client secret, because you can rotate and protect a certificate on your own schedule, and you are the one now responsible for getting that credential onto the host over a secure channel.

Whichever option you pick, every path ends at the same checkpoint. The identity needs a role assignment scoped to that one AKS cluster before the host can register. Scoping it to a single cluster rather than the subscription is the difference between a compromised host identity reaching one cluster and reaching your whole estate, so do not be tempted to widen the scope to make the setup easier.

My honest take is that if your host is an Azure VM, use a managed identity and do not think about it again. If it is bare metal or lives in another cloud, Arc is the cleaner path because it gives you secretless auth without generating and rotating a service principal credential you then have to protect. Reach for a service principal only when neither of the first two is possible.

The networking reality check

This is the section where preview really shows. Flex nodes use a bring-your-own networking model, and you are responsible for a lot of it.

The AKS-managed network plugins do not run on flex nodes during preview. Azure CNI and Azure CNI Powered by Cilium do not extend their pod networking, network identity, or network policy to a flex node. You can attach a flex node to a cluster that uses one of those plugins, but the plugin simply does not reach the flex node. The documented end-to-end path instead creates the cluster with --network-plugin none and deploys Unbounded-Net on both the AKS-managed nodes and the flex nodes.

There are two network decisions to plan, and they are independent. Choosing a public or private API endpoint does not decide how workload traffic moves between nodes and pods.

• AKS API exposure is how your admin workstation and each flex host reach the Kubernetes API server. Public uses a secured public endpoint. Private uses a private endpoint and needs private DNS plus routing.
• Flex node connectivity is how AKS nodes, flex nodes, pods, and services talk to each other. Either you already have Layer 3 reachability between the two node networks, or you use an Unbounded-Net WireGuard gateway for topologies that cross a public network.

Because flex nodes can sit in a different virtual network, region, data centre, or third-party environment, pod networking has to add address allocation and routing that a normal CNI assumes it already has. Unbounded-Net models each location as a Site. The AKS nodes live in the cluster Site, the flex hosts live in a flex-site, and a SitePeering connects them.

Underneath, Unbounded-Net runs a controller plus a node agent on every participating node, AKS-managed and flex alike. The controller handles node placement, pod address allocation, and cross-location routing state. When a flex node joins, the controller matches its internal IP address to a Site’s node CIDR, then hands the node pod addresses from that Site’s pod CIDR. That is how you get non-overlapping pod IPs across two locations that a normal CNI would never expect to coexist. The SitePeering I create in the walkthrough sets meshNodes: true for node-to-node mesh connectivity and tunnelProtocol: Auto so Unbounded-Net picks its datapath over whatever underlay you already have.

The address planning is not optional and it is not forgiving. You need nonoverlapping ranges for the AKS node subnet, the AKS pod CIDR, the flex node subnet, the flex pod CIDR, and the Kubernetes service CIDR, and none of them can clash with anything reachable over peering, VPN, or ExpressRoute. Each subnet has to sit inside its Site node CIDR, and the DNS service IP has to sit inside the service CIDR. Azure can catch some conflicts but it cannot validate every routed external network, so this is a conversation to have with whoever owns your networking before you type a single command.

There is a second path for topologies without Layer 3 reachability, using an Unbounded-Net WireGuard gateway, but the docs are clear that it is a lab rather than the primary workflow, and a private cluster with a WireGuard gateway is not covered at all yet. If your two networks cannot already reach each other at Layer 3, factor that gap into your timeline.

If you take one thing from this section, take this. The compute story is elegant. The networking story is where the preview work lands on you, and it is the part most likely to cost you an afternoon.

How to set it up

The full deployment is a five-article journey in the docs, so I am not going to reproduce every command. Instead here is the shape of it end to end, with the commands that matter and the gotchas at each stage. Prerequisites are an approved preview subscription, the Azure CLI, kubectl, and a Linux-compatible Bash environment with network access to both Azure and your host.

Register the preview and build the cluster

Everything starts with the preview extension and two feature flags. You need aks-preview at version 22.0.0b8 or later, and both feature registrations have to reach Registered before the resource provider will play along.

Registration can take a few minutes. Poll az feature show ... --query properties.state until both read Registered, then refresh the provider with az provider register --namespace Microsoft.ContainerService --wait. Skip the refresh and the flags will not take effect, which is an easy few minutes to lose.

Next you create the cluster with no network plugin. This example follows the documented path of a public cluster with a single Azure-managed system node and SSH disabled.

Do not panic when the system node sits at NotReady after this. With --network-plugin none there is no pod networking yet, so that state is expected until Unbounded-Net is installed. The first time I saw it I went looking for a problem that was not there.

Lay down networking and create the pool

Before you can run kubectl unbounded anything, you need the Unbounded-Net CLI itself, and this is the pre-step that’s easy to miss. It ships as a kubectl plugin that you download from the Azure/unbounded releases, extract, and install onto your PATH. You need curl and tar in your Bash environment and outbound HTTPS to github.com.

UNBOUNDED_VERSION is one of the component versions the planning article pins in the environment file, so it matches the flex node release you’re deploying. The kubectl unbounded version check is your signal the plugin is on your PATH and kubectl can find it. Nothing that follows works until that returns cleanly.

With the plugin in place, install Unbounded-Net into the cluster, initialize the two Sites from your address plan, then peer them over the Layer 3 path.

After the peering applies and the Unbounded-Net controller and node daemonset roll out, the AKS-managed node finally goes Ready. There is also a temporary preview step where you apply an aks-flex-node-daemon ClusterRole and ClusterRoleBinding by hand if they do not already exist. Skip it and the agent will authenticate during attach but then exit when its controller cache cannot sync, which is a confusing failure mode if you did not know the RBAC was missing.

Now the pool itself. This is where flex nodes look different from anything else in AKS. You do not pass a node count, a VM size, an OS type, or a subnet, because you are not asking Azure to provision anything. You are declaring a logical grouping for compute you will attach later.

The --vm-set-type FlexNodes value is the giveaway. Note the NoSchedule taint in the example too. Tainting the pool from the start is sensible, because it stops general workloads landing on hosts you have not finished attaching or validating yet.

Prepare the host and attach it

The last stage is host-side. You give the host its identity, grant that identity access at the AKS cluster scope, then bootstrap the agent. For a managed identity the bootstrap looks like this on the host.

The workflow is careful about secrets, and rightly so. You verify the agent archive against a SHA-256 checksum before running it, you install the protected bootstrap config with tight file permissions, and you clean up the transient files afterwards. The docs are firm that credentials, kubeconfig content, and bootstrap data never go in the shared environment file. If you are scripting this for more than one host, keep that discipline, because you are now running Azure-issued bootstrap material on machines you own.

The bit that caught me out here was a race. My first bootstrap ran every preflight check, installed the agent, set up the isolated node environment, then failed at the very last step creating the Azure Machine with a 409 OperationNotAllowed because there was already an in-progress operation on the pool. Retrying straight away made it worse, because each attempt kicks off its own pool operation and collides with the one before it. The fix was to stop hammering it, wait for az aks nodepool show to report Succeeded again, then run bootstrap once and leave it alone. It went through cleanly the moment the pool was idle. If you automate the attach, build in a wait-for-idle check before the bootstrap rather than a naive retry loop.

Once bootstrap reports the agent service has started, the host shows up as a Kubernetes Node, an Azure Machine resource, and a member of the flex-site. From there it schedules workloads like any other node, subject to the taints and tolerations you set. I dropped a test pod on mine with a matching toleration and it came up with a pod IP straight out of the flex pod CIDR, which is Unbounded-Net doing its job across the two Sites.

To remove a host, you reset or delete it through the AKS management APIs, and the second Kubernetes-facing identity is what lets the agent cleanly pull the node out of the cluster.

Day two is where the design shows

Getting a flex node to Ready is the easy bit. The interesting part is everything after, when you have to update it, upgrade it, and eventually take it away, and that’s where the reconciliation model stops being a neat idea and starts shaping what you actually type. Day-two work runs through a small set of az aks machine and az aks nodepool commands alongside ordinary kubectl, and how those two sides line up is where things either stay boring or quietly catch you out.

Start with inventory, because you are now tracking three views of the same fleet. az aks machine list gives you the Azure Machines with their node names, versions, and provisioning state. kubectl get nodes --selector kubernetes.azure.com/nodepool-type=FlexNodes gives you the Kubernetes side with kubelet versions and UIDs. az aks nodepool show gives you the pool-level version and state. I keep all three handy during any change, because a flex node is only healthy when all three agree.

The first real gotcha is that pool settings and Machine settings are not the same thing. Updating labels or taints on the pool with az aks nodepool update only changes the defaults for hosts that join later. It does not touch a single existing node. To change a node that is already attached, you update its Machine with az aks machine update, and that command replaces the whole set of values it manages rather than merging, so you have to pass every label and taint you want the node to keep. Miss one and you have silently dropped it. This is the sort of thing that reads fine in a script review and then removes a scheduling taint in production.

Upgrades reimage the node, not the host

Upgrades are the clearest illustration of the reconciliation model, so they are worth walking through properly.

The first rule is ordering. A flex node cannot run a Kubernetes version newer than the control plane, so you upgrade the control plane first with az aks upgrade --control-plane-only, confirm it, and only then move the nodes. That is the same discipline as any AKS upgrade, but here it is a hard dependency rather than a recommendation.

The part that surprised me is how a single-node upgrade actually happens. You set the target version on the Machine with az aks machine update --kubernetes-version, and at that point nothing visibly changes. The existing Node keeps reporting the old kubelet version. The new version only lands when you drain and then delete the Kubernetes Node.

Deleting the Node here is not a host removal. It is the trigger. The agent notices the Node is gone, and because the Machine’s desired state now says a newer version, it registers a replacement Node at that version. The host never leaves. You can watch it come back with kubectl wait --for=create node/${FLEX_NODE_NAME} followed by a readiness wait, and the tell that it worked is that the Node returns with the same name but a new UID and the target kubelet version. The Machine reports that version without a v prefix and the Node reports it with one, which is a small inconsistency but a useful fingerprint when you are checking.

Once you have validated the target version on one node, az aks nodepool upgrade walks the whole pool. Nodes that get recreated pick up new UIDs, and controller-managed pods on them are rescheduled, so re-run whatever workload and connectivity validation you used at attach time afterwards. Anything you created as a bare pod rather than through a controller will not come back on its own, which is worth saying out loud because a drain plus delete is unforgiving of pets.

Removing a host cleanly

Removal has more steps than you might expect, and each one matters because you are handing hardware back to yourself rather than deleting a VM Azure owns.

First you drain the Node so controller-managed workloads move off. If the drain blocks on a Pod Disruption Budget, unmanaged pods, or local storage, treat that as a signal and not an obstacle to force past, because those protections exist for a reason and directly created pods will not be recreated elsewhere.

Then you reset the host itself by running the agent’s reset command over SSH. This is the real cleanup, and it does a lot. It stops and removes the agent service, removes the flex node worker machine, reverts the network configuration the agent applied, and deletes /etc/aks-flex-node and /var/log/aks-flex-node, which also removes any installed service principal credential. If you delivered a credential to some other path on the host, you have to remove that copy yourself according to your own process. Reset deliberately leaves the agent program files under /usr/local/lib/aks-flex-node so you can reattach the host later without downloading the release again. If you want the machine returned to a state that has never run a flex node, reimage it rather than trusting reset alone.

Reset deregisters the node but the Kubernetes Node object and sometimes the Azure Machine can linger, so you finish by deleting the Node with kubectl delete node --ignore-not-found and confirming the Machine is gone. Skip that and you leave ghosts in both the cluster and ARM. The order matters here, drain then reset then clean up the residual objects, and doing it in that sequence is the difference between a tidy detach and an afternoon of chasing phantom nodes.

Who owns what

The support policy is worth reading in full before you build anything you care about, because flex nodes move a lot of responsibility onto you and the boundary is not where people assume it is.

Microsoft owns the managed control plane, the resource-provider integration, the flex node agent and bootstrap workflow, the Azure Machine operations, and the documented management interfaces. You own almost everything on the host side. That means host provisioning, capacity, hardware, firmware, and recovery. It means the operating system and its patching, security, and monitoring. It means identity creation, credential protection, the cluster-scoped role assignment, and rotation. Above all it means the network, so routing, peering, firewalls, network security groups, DNS, proxies, address planning, VPN or ExpressRoute, and the entire Unbounded-Net layer including its installation, configuration, and upgrades.

That is a fair split for what this is, but it is a bigger surface than a standard node pool by a wide margin. With a normal pool, the VM, the OS image, and the CNI are Microsoft’s problem. Here they are yours. There is also a practical support wrinkle. Bootstrap and lifecycle commands run as root and change host state, and Microsoft Support will not touch a customer-managed host without your permission and hands-on help, so you have to keep a supported management path open and be ready to collect agent logs, OS logs, and network state yourself. Budget for the operational load, not just the setup.

Limitations worth knowing before you commit

A few hard limitations will shape whether this fits at all, and they are easier to swallow now than after you have designed around them.

Flex nodes are Linux only. The hosts are Linux, full stop, so this is not a route to bring Windows workloads onto your own hardware. During preview the AKS-managed CNIs do not extend to flex nodes, so you are on a bring-your-own networking model with Unbounded-Net or another compatible customer-managed CNI, and the pod networking, network identity, and network policy from Azure CNI or Cilium simply do not reach these nodes. There is no Azure portal support either, so pool and Machine management is CLI and API only. And the one that catches people out operationally is that you cannot stop a cluster that contains a flex node pool, even if the rest of the cluster is standard node pools, so the cost-saving trick of stopping a dev cluster overnight is off the table the moment a flex pool exists.

None of these are dealbreakers for the scenarios flex nodes are aimed at, but every one of them is the kind of constraint that is annoying to discover halfway through a proof of concept rather than at the planning stage.

Where flex nodes fit, and where they do not

Flex nodes do not replace standard node pools, and you should not try to make them. The guidance is to run standard pools for general workloads and reserve flex nodes for the cases that genuinely need a different host model.

Those cases are fairly specific. Capacity outside the AKS cluster’s Azure region. Bare metal you need to bring into the cluster. Data residency rules that pin selected jobs to your own infrastructure. Lab and test environments where you want to use hardware you already have, including mixing amd64 and arm64 hosts to test across architectures.

If none of those describe you, standard node pools, node auto provisioning, or Virtual Machines node pools are still the right answer and will stay far simpler to run.

Wrapping up

We covered a lot, so here’s the short version. Flex nodes let the AKS control plane schedule onto machines you own, using an Azure Machine resource as the per-node source of truth and an agent that reconciles a disposable Kubernetes node layer on top of a stable host. Identity is where you prove the host to Azure, networking is where the preview work lands on you, and the day-two lifecycle for upgrades and removal all flows from that reconciliation model. In exchange for that flexibility you take on a much bigger slice of the shared responsibility split than a standard node pool, and you live with real limitations like Linux-only hosts, no managed CNI, no portal, and no stopping a cluster that holds a flex pool.

What I keep coming back to is how strategically interesting the shape is. AKS is quietly turning into a control plane for compute you own, wherever that compute happens to sit. Today that means your bare metal, your edge sites, and capacity outside the cluster’s region. That alone is a big deal for anyone who’s been forced onto Arc-enabled Kubernetes or a self-managed cluster to solve those problems.

Where I’d love to see this go is one step further. Right now you bring and attach each host yourself. Imagine flex nodes growing the ability to automatically provision and attach capacity in other Azure regions, or even other clouds, under a single AKS control plane. You’d get a genuinely multi-location cluster without stitching together three separate clusters and a Fleet on top just to place workloads near data or capacity. One control plane, many locations, nodes appearing and disappearing on demand. That’s a different way of thinking about a Kubernetes footprint, and the reconciliation and Site model here feels like the groundwork for exactly that kind of future.

It’s early, and I’m not pretending the preview is smooth. The networking will cost you an afternoon and the operational surface is larger than people expect. But this is one of the more forward-looking previews AKS has shipped, and if the location story fills out the way the architecture hints at, it could reshape how we run clusters that span regions and providers. If any of the scenarios in this post are yours, get a lab subscription approved and start learning the flows now, because this is the kind of feature you want muscle memory for before it matters.

The official Microsoft docs are worth reading alongside any hands-on testing:

• Flex nodes for AKS overview
• Plan flex nodes for AKS
• Networking concepts for flex nodes
• Identity and access concepts for flex nodes
• Manage and remove flex nodes
• Support policy for flex nodes

If you get preview access and try it, I’d love to hear how you get on, especially how the networking behaves across regions. If you spin it up, let me know how it goes.