URL has been copied successfully!
URL has been copied successfully!
URL has been copied successfully!
URL has been copied successfully!
URL has been copied successfully!
Share:
Twitter
LinkedIn
Facebook
Reddit
Follow by Email
Copy link
Threads
Bluesky
Reading Time: 9 minutes

Microsoft has shipped Automatic Pod Disruption Budget management for Azure Kubernetes Service (AKS). If you have ever sat watching an AKS upgrade stall because Kubernetes refused to evict a pod, this one is worth your attention.

I have been on the wrong side of it more than once. AKS starts draining a node, a Pod Disruption Budget (PDB) blocks the eviction, the upgrade stalls, and now you are working out whether to scale something manually, bypass the PDB, or deal with a stuck node.

Automatic PDB management tries to make that smoother. It can create PDBs for deployments that do not have one, and it can temporarily scale a deployment when a PDB blocks a drain. Once the drain is finished, it scales things back down again.

Microsoft Learn has the full reference for Automatic Pod Disruption Budget management in AKS.

Note: This feature is currently in preview. As with all AKS preview features, do not enable it on a production cluster without testing it properly first.

What is Automatic PDB management?

Automatic PDB management is an AKS cluster extension. Under the covers it is based on the open source Eviction Autoscaler project.

It has two main jobs.

The first job is automatic PDB creation. If you enable controllerConfig.pdb.create=true, the extension watches deployments in the namespaces you have enabled. When it finds a deployment without a matching PDB, it creates one. For plain deployments, it sets minAvailable to the current replica count. For deployments managed by HPA or KEDA, it tracks the autoscaler’s minimum replica floor instead, keeping the PDB aligned with what the autoscaler will allow. It also continuously reconciles those PDBs, so if the replica count changes later the PDB is updated to match.

The second job is replica scaling during drain operations. If AKS cordons a node during an upgrade and a PDB blocks pod eviction, the extension temporarily adds replicas up to the deployment’s configured maxSurge limit so the disruption budget is satisfied. The drain can then continue without ignoring the PDB.

After the eviction has completed and things have settled, the extension scales the deployment back down. One important scope note: this only covers voluntary disruptions such as upgrade drains and manual cordon/drain operations. Node failures and pod crashes do not trigger it.

How it works with autoscalers

This is the bit I think is easy to miss. The extension is not just changing the deployment replica count blindly. It has to work with whatever is already controlling that workload.

With a normal deployment, the extension increases the deployment’s replicas value directly. Once the drain has finished, it restores the original replica count after a short cooldown.

With Horizontal Pod Autoscaler (HPA), it works differently. If the extension only changed the deployment replica count, the HPA could notice the extra pod and scale it back down before the drain had finished. To avoid that, the extension raises the HPA minReplicas value and adds a replica straight away. When the drain is finished, it puts minReplicas back where it was and lets HPA handle the rest.

KEDA follows the same idea, but through the ScaledObject. The extension raises minReplicaCount and adds a replica immediately. That avoids waiting for KEDA to sync to its HPA and then for the HPA to update the deployment.

One thing to watch for is using KEDA and a separate HPA against the same deployment. KEDA already creates an HPA behind the scenes. If you add another one yourself, you now have two autoscalers trying to write to the same target. The extension marks the EvictionAutoScaler as degraded and skips the surge because it cannot safely coordinate that setup.

Here is how the surge and revert behaviour looks across each configuration.

ConfigurationHow the extension surgesHow the extension reverts
Deployment onlyIncreases replicas directlyRestores original replica count after the cooldown
Deployment + HPARaises minReplicas and adds a replica immediately, so HPA cannot scale back down mid-drainRestores original minReplicas; HPA handles scale-down naturally
Deployment + KEDARaises minReplicaCount on the ScaledObject and adds a replica immediately, avoiding the KEDA-to-HPA sync lagRestores original minReplicaCount; KEDA and its HPA handle scale-down
Deployment + KEDA + separate HPANot supported. Marks EvictionAutoScaler as degraded and skips the surgen/a; remove the extra HPA to fix

Vertical Pod Autoscaler (VPA) is different. VPA changes CPU and memory requests, not the number of replicas. In some modes it can evict pods so they come back with updated requests. Those evictions still need to respect any PDBs you have, but Automatic PDB management is focused on upgrade and drain scenarios where a node has been cordoned. I would not assume it will rescue a VPA-driven eviction that is blocked by an overly strict PDB.

If you use VPA, test this carefully. A good pattern is to run VPA in recommendation mode first, then move to an active update mode only when you understand how the workload behaves. Also avoid using VPA and HPA on the same CPU or memory signals, because they can fight each other. If you need both, use VPA for resource sizing and HPA for a different signal such as requests, queue length, or another custom metric.

Why this matters

During an AKS upgrade, nodes are cordoned and drained. Pods are evicted so AKS can reimage or replace the node. PDBs control how many pods for a workload can be unavailable during voluntary disruptions like this.

The awkward part is that both missing and overly strict PDBs cause problems.

If a deployment has no PDB, Kubernetes can evict too many replicas at once during maintenance. That creates avoidable downtime.

If a deployment has a PDB that allows no disruption, the drain can block completely. A common example is a deployment with three replicas and a PDB using minAvailable: 3. Kubernetes cannot evict one pod because doing so would drop availability below the budget.

There is a third, less obvious scenario. A correctly configured PDB can still block eviction if the deployment simply does not have enough replicas to absorb one. The budget is set up correctly, there are just not enough pods for it to let one go. That catches teams out who have done the right thing but underprovisioned the replica count.

That leaves platform teams with three bad options. Force the upgrade and bypass protection, manually scale workloads during maintenance, or clean up stuck and quarantined nodes after the fact.

ApproachBehaviourTrade-off
Force upgradeBypasses PDB protections entirelyRisk of simultaneous pod eviction and service disruption
Undrainable node behaviourCordons and quarantines blocked nodes; upgrade continues on other nodesRequires manual cleanup of quarantined nodes after the upgrade
OverprovisioningPermanently run extra replicas so a PDB always has headroom to allow evictionOngoing cost for capacity that is only needed during upgrades
Automatic PDB managementCreates PDBs for unprotected deployments and temporarily scales up replicas to satisfy constraints, then scales back downRequires the extension; adds transient capacity only during drain

Automatic PDB management tries to remove that manual work.

When would you use it?

This feature fits clusters where upgrades have already been slowed down or blocked by PDB behaviour.

Good candidates include clusters where application teams do not consistently define PDBs, clusters with strict PDBs that regularly block drains, and shared platform clusters where the platform team owns upgrade reliability but individual teams own workload manifests.

It is less useful if your workloads already have well-designed PDBs, enough replicas, and predictable upgrade behaviour. In that case, this feature might still be worth testing, but it is not fixing a current pain point.

The preview status matters. Microsoft states that AKS preview features are self-service, opt-in, provided as-is, and not meant for production use. So this is something to test in development or staging first.

Also, do not use this as a reason to stop thinking about PDB design. The feature helps with upgrade reliability, but your applications still need sensible replica counts and availability expectations.

Check whether PDBs are already hurting upgrades

Before enabling anything, look for evidence. The Azure Activity Log can show failed upgrade operations with messages about disruption budgets or failed evictions.

You can also look at recent Kubernetes events. This command sorts events by timestamp and searches for disruption, eviction, and drain messages.

If you see messages like Cannot evict pod as it would violate the pod's disruption budget, this feature is directly relevant. If you see nothing, you may still have unprotected workloads, but you do not yet have proof that PDBs are blocking drains.

Install the extension

The extension requires an AKS cluster using managed identity, Azure CLI 2.64.0 or later, and the Microsoft.KubernetesConfiguration provider. It currently works with Deployments only. StatefulSets are not supported.

It also needs permission inside the cluster to do its job. During installation, the extension creates its own service account and Kubernetes role-based access control (RBAC) permissions. Those permissions allow it to read and write Deployments, PDBs, and Events in the namespaces it manages.

That access makes sense for what the extension does, but it is still worth calling out. You are installing something that can create disruption budgets and temporarily change replica counts. I would be careful about enabling it across every namespace until you have tested the behaviour.

For most clusters, the best starting point is targeted protection for specific namespaces. This enables automatic PDB creation only where you choose.

The following example enables the extension for kube-system and production.

This keeps the blast radius small. The extension can manage system workloads and the production namespace you explicitly named, while leaving everything else alone.

You can also enable it cluster-wide with controllerConfig.namespaces.enabledByDefault=true, but that is not the recommended starting point. Cluster-wide automation is attractive, but PDBs encode availability intent. Applying that automatically to every namespace can surprise teams that intentionally allow disruption for jobs, workers, test workloads, or low-priority services.

Configuration options

Three settings control how the extension behaves.

SettingDescriptionDefault
controllerConfig.pdb.createAutomatically create PDBs for deployments that do not have onefalse
controllerConfig.namespaces.enabledByDefaultEnable automatic PDB management in all namespaces, including namespaces created laterfalse
controllerConfig.namespaces.actionedNamespacesComma-separated list of namespaces to enable when enabledByDefault is false{kube-system}

The kube-system default is there for a reason. AKS-managed system components use PDBs that can block node drains during upgrades, so the basic install covers those even without specifying any namespaces explicitly.

A few common patterns show how these settings combine in practice.

PatternSettingsUse case
Conservativepdb.create=false, enabledByDefault=false, actionedNamespaces={kube-system}Manage PDBs manually; only want autoscaling for system workloads
Targeted auto-protection (recommended)pdb.create=true, enabledByDefault=false, actionedNamespaces={production,staging}Automatic PDB creation and autoscaling in specific namespaces
Cluster-wide protectionpdb.create=true, enabledByDefault=trueEvery namespace protected automatically
Monitoring onlypdb.create=false, enabledByDefault=trueAutoscaling across all namespaces, managing PDBs yourself

Targeted auto-protection is the recommended starting point for most clusters.

Namespace control

The actionedNamespaces list is set at install time. If you need to opt individual namespaces in or out at runtime without reinstalling the extension, use annotations directly on the namespace object.

To opt a namespace in when enabledByDefault is false:

To exclude a namespace when enabledByDefault is true:

You can also exclude individual deployments from automatic PDB creation by annotating the deployment:

The extension also automatically skips PDB creation for any deployment that has maxUnavailable set to a non-zero value in its rolling update strategy. Those deployments already tolerate some downtime during updates, so applying a strict PDB would contradict that intent.

Verify what it created

After installation, check whether the extension has created PDBs. This command filters for PDBs owned by the extension.

You should see PDBs in the namespaces you enabled. Manually created PDBs should not be modified or deleted by the extension.

You can also inspect the extension’s custom resources.

The YAML output is useful when autoscalers are involved. The extension supports a deployment managed by one HPA or one KEDA ScaledObject. It does not support a deployment managed by KEDA and a separate HPA at the same time. In that case, the EvictionAutoScaler status is marked as degraded and the extension skips the surge.

Test a drain in a safe cluster

If you are testing this in a dev or staging cluster, cordon a node that has a pod from one of your protected deployments.

Now watch the events, pods, and PDB status.

You are looking for the deployment to scale up temporarily and for the PDB to allow disruption again. Once you have finished testing, uncordon the node.

Things to watch for

The most important limitation is capacity. If the extension needs to add temporary replicas but the cluster has nowhere to schedule them, the drain is still blocked. Cluster autoscaler or node auto-provisioning can help, but that adds node scale-up time to the upgrade path.

The extension never modifies or deletes PDBs you created manually. It only manages the ones it created itself, marked with an ownedBy: EvictionAutoScaler annotation. If you want to take manual control of an auto-created PDB, remove the annotation:

After removing it, the extension no longer reconciles that PDB. Auto-created PDBs are cleaned up automatically in three situations: when the extension is removed from the cluster, when the parent deployment is deleted, and when a namespace is removed from the extension’s scope.

Changing extension configuration requires deleting and recreating the extension entirely. Treat the initial configuration as a real platform decision rather than a quick toggle.

Then reinstall with the new --configuration-settings values.

There is also a subtle design point. The extension defaults to pdb.create=false, which means the basic install watches for blocked evictions but does not automatically create PDBs for unprotected deployments. If you expect automatic PDB creation, you need to enable it explicitly.

Wrapping up

I have seen enough AKS upgrades grind to a halt over PDB friction to know how frustrating it can be. Automatic PDB management targets the right problem, and the approach is sound: it does not bypass PDB protections, it adds just enough temporary capacity to satisfy them and then steps back.

It is still in preview, so I would not enable it on a production cluster yet. But it is absolutely worth testing in a dev or staging environment, especially if your clusters have a mix of deployments with HPAs, KEDA workloads, or strict PDBs that have caused drain issues before. Start with targeted namespace protection, run a controlled cordon and drain, and watch the EvictionAutoScaler resources and PDB status as it happens.

If you try it, let me know how it goes. I am always interested to hear how it behaves against real workloads.

Share:
Twitter
LinkedIn
Facebook
Reddit
Follow by Email
Copy link
Threads
Bluesky

Pixel Robots.

I’m Richard Hooper aka Pixel Robots. I started this blog in 2016 for a couple reasons. The first reason was basically just a place for me to store my step by step guides, troubleshooting guides and just plain ideas about being a sysadmin. The second reason was to share what I have learned and found out with other people like me. Hopefully, you can find something useful on the site.

0 Comments

Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *