Flatcar Container Linux on AKS: First Look

Reading Time: 6 minutes

Recently I spotted something new in the AKS CLI preview. Flatcar showed up as a valid OS SKU for Linux node pools. As someone who’s been running AKS clusters for years, I was curious to see what this CNCF-backed immutable OS brings to the table, so I spun up a test cluster to see how it performs.

What is Flatcar Container Linux?

Flatcar Container Linux is a container-optimized, immutable operating system that was forked from CoreOS Container Linux after Red Hat acquired CoreOS in 2018. The community wanted to keep the original CoreOS vision alive, so Kinvolk (later acquired by Microsoft) created Flatcar as a continuation. It’s now a CNCF project, which means it’s vendor-neutral and community-stewarded. The key selling point? An immutable filesystem designed specifically for running containers, with a focus on security and multi-cloud portability.

Unlike general-purpose distributions like Ubuntu where you can SSH in and apt install whatever you want, Flatcar’s root filesystem is read-only. All binaries, libraries, and static configurations are locked down. Updates come as full OS images rather than individual package updates. For Kubernetes workloads that should already be containerized and stateless, this is actually a feature, not a limitation.

Understanding Immutable Operating Systems

An immutable operating system means the core OS files can’t be changed at runtime. All binaries, libraries, and system configurations are read-only. You can’t install packages, modify system files, or change the OS configuration once it’s running.

In the container world, this makes perfect sense. Containers are supposed to be self-contained units with all their dependencies baked in. Your application shouldn’t need anything from the host OS except the container runtime. An immutable OS enforces this best practice by design.

The trade-off is that certain types of containers won’t work:

Containers that need host process namespace – If your container runs with hostPID: true and expects to interact with or modify host processes, you’ll run into issues on an immutable OS.

Containers that mount and write to host paths – DaemonSets or privileged containers that mount /host or specific host directories and try to write files (like installing agents into /usr/local) will fail. The filesystem is read-only.

Containers expecting to install host packages – Init containers or sidecars that run apt install or yum install to add OS-level packages to the node won’t work. Everything needs to be in the container image.

If your workloads follow true cloud-native patterns where containers are isolated and don’t reach outside their boundaries, immutable OS options like Flatcar work beautifully. If you have legacy workloads that treat nodes like pets and modify them at runtime, you’ll need to refactor before moving to an immutable OS.

Why Would You Use Flatcar on AKS?

The main reasons you’d reach for Flatcar over Ubuntu or Azure Linux are:

Multi-cloud consistency – If you’re running Kubernetes across Azure, AWS, GCP, or on-premises environments, Flatcar gives you the same OS foundation everywhere. You’re not dealing with Ubuntu on one cloud and Bottlerocket on another.

Immutability – The read-only filesystem eliminates entire classes of exploits. No one can modify binaries at runtime, even if they escape a container. You know exactly what’s running because the OS state can’t drift.

Minimal attack surface – Flatcar ships only what’s necessary to run containers. No package manager, no extra services, just the essentials. Fewer components means fewer CVEs to track.

Reproducible nodes – Since configuration happens at provisioning time and the OS can’t be modified afterward, every node with the same image is identical. No configuration drift, no surprises.

Getting Started with Flatcar on AKS

Flatcar support is currently in preview, so you’ll need to install or update the aks-preview extension first:

Next, register the feature flag:

You can check the registration status with:

Wait until it shows "Registered" before proceeding. Once registered, refresh the provider:

Once that’s registered (it can take a few minutes), creating a cluster with Flatcar is straightforward:

The --os-sku Flatcar parameter is doing the work here. You can also add Flatcar node pools to existing clusters:

One thing I noticed: the --os-sku parameter now supports quite a few options beyond the old Ubuntu/CBLMariner days. The CLI help text shows Ubuntu, Ubuntu2204, Ubuntu2404, AzureLinux, AzureLinux3, AzureLinuxOSGuard, and now Flatcar. It’s good to see more choice in the OS layer.

What I Found Testing Flatcar Nodes

When my cluster came up, I used kubectl debug to connect to a Flatcar node and poke around. A few observations:

The filesystem really is read-only – Trying to write to /usr or /bin fails immediately. /var and a few other paths are writable for logs and runtime data, but the core OS is locked down.

SELinux is enabled by default – Flatcar uses SELinux for mandatory access control rather than AppArmor (which Ubuntu uses). This might matter if you have workloads with specific security context requirements.

SystemD and familiar tooling – Even though it’s immutable, Flatcar still uses systemd, so checking service status with systemctl works as expected. Standard tools like journalctl for logs are all there.

Updates are atomic – OS updates replace the entire root partition rather than updating individual packages. The system reboots into the new image. This is similar to how Bottlerocket and Talos work.

Gotchas and Considerations

After testing a few workloads, here are some things to watch out for:

DaemonSets with host filesystem access – If you have DaemonSets that mount /host and expect to write to the host filesystem, they’ll break. You’ll need to rethink those patterns or use ConfigMaps and volume mounts instead.

Init containers that modify the host – Similarly, init containers that expect to install packages or modify the OS won’t work. I’ve seen patterns where init containers run apt-get install to add kernel modules or system libraries. These need to be baked into your container images instead.

Privileged containers with host mounts – Containers running with privileged: true that expect to write to mounted host directories will fail. Common culprits include log collectors that write to /var/log on the host. Check your security contexts and volume mounts carefully.

Host process containers – Workloads that use hostPID: true, hostNetwork: true, or hostIPC: true and interact with host-level processes may behave unexpectedly. While they can run, if they try to modify anything on the host filesystem, those changes won’t persist or will outright fail.

Documentation is still coming – Since Flatcar support just landed in the preview extension, there’s no official AKS documentation page yet. I’m sure there are some feature limitations or gotchas we’ll discover as the community tests this more broadly, but for now we’ll have to wait and see what makes it into the official docs.

No Secure Boot yet – Unlike Azure Linux OS Guard (another immutable option from Microsoft), Flatcar on AKS doesn’t currently support Secure Boot. If that’s a hard requirement, OS Guard might be a better fit.

When to Use Flatcar vs. Other OS Options

AKS now supports several OS choices, and picking the right one depends on your priorities:

Use Flatcar if:

• You need consistency across multiple clouds or on-premises Kubernetes environments
• You want a vendor-neutral, community-stewarded OS
• Immutability and minimal attack surface are priorities
• You’re comfortable with the constraints of a read-only filesystem

Consider Azure Linux OS Guard if:

• You’re primarily Azure-focused and want Microsoft-backed support
• You need Secure Boot and other Azure-specific security features
• You want immutability but prefer Microsoft’s enterprise tooling

For more details on OS Guard, check out the official documentation.

Stick with Ubuntu or Azure Linux if:

• You need a general-purpose OS with package management
• You have workloads that require host filesystem modifications
• You want maximum flexibility and are willing to manage security through other means

My Take

I’m cautiously optimistic about Flatcar on AKS. The immutable OS model makes a lot of sense for Kubernetes workloads that are already containerized and stateless. The CNCF backing and multi-cloud support are appealing if you’re running hybrid or multi-cloud environments.

That said, this is still a preview feature, and the constraints of an immutable OS mean you’ll need to audit your workloads before migrating. Anything that expects to modify the host or relies on traditional package management will need to be refactored.

For new clusters running cloud-native workloads, Flatcar is worth testing. For existing clusters with legacy components, proceed carefully and test thoroughly in dev/staging first.

Next Steps

If you want to try Flatcar yourself:

1. Register the preview feature: az feature register --namespace Microsoft.ContainerService --name AKSFlatcarPreview
2. Create a test cluster or node pool with --os-sku Flatcar
3. Deploy your workloads and validate they run correctly
4. Check logs and monitoring to ensure everything behaves as expected

Flatcar is available on all currently supported AKS Kubernetes versions, which makes it easy to test alongside your existing node pools without upgrading the control plane.

Have you tried Flatcar or other immutable OS options on Kubernetes? I’d be interested to hear what your experience has been, especially if you’re running multi-cloud environments.