Prepared Image Specification (PIS) for Azure Kubernetes Service (AKS) is in preview, and I’ve been putting it through its paces. The premise is straightforward. You pre-bake stable container images and host customisations into an AKS-managed node image so pods do not wait on every download when a new node boots.
That makes it a cleaner alternative to the DaemonSet and CronJob image-puller patterns that many teams use today for cold-start scenarios.
It mostly works, but I hit an undocumented RBAC requirement and a genuine bug in the script generator that tripped me up along the way. Both are worth knowing about before you start.
Because this is still a preview feature, I would treat it as something to test in a disposable or non-production environment first. AKS previews are opt-in, provided as-is, and not intended for production workloads under normal service-level agreement expectations.
Why this matters
Image pull latency at cold start is easy to ignore until workloads are time-sensitive. When a new node joins the cluster, Kubernetes has to pull images from your registry before pods can run. With large images or a slow registry path, that pull time adds directly to your scale-out latency.
The usual workaround is a DaemonSet that pre-pulls images onto every node. It works, but it’s operational overhead. You need to keep the image list in sync with what your workloads actually use, and it doesn’t help on a brand-new node because the pull happens after the node joins rather than being baked in before boot.
PIS bakes the images in before the node even joins the cluster. That’s a meaningfully different model.
It is also worth being precise about the boundary. This is not a bring-your-own-node-image feature. AKS still builds from a supported AKS node image, then adds the prepared content. PIS does not remove virtual machine allocation, node registration, networking, scheduling, driver startup, or your application startup time. It only removes work that can be prepared ahead of time.
The trade-off is that PIS is a point-in-time snapshot. Images are baked when you create a PIS version and they don’t update automatically. When your images change, you create a new version and roll the node pool. For workloads on a release cadence, that maps well onto a CI/CD step. For workloads with constantly changing images, it’s more friction.
Microsoft’s testing points in the same direction. The biggest gains showed up where the repeated work was large and stable, including Windows image pulls, multi-node bursts, portable runtimes, dependency bundles, and some model-serving paths.
The published numbers include a 74% median reduction for a three-node Windows burst, 92% and 98% reductions for the slowest Linux and Windows image pulls, and smaller but still measurable improvements for a T4 GPU model-serving test. Those are useful signals, not universal promises. You still need to measure your own workload from scale event to useful traffic, not just from node creation to Ready.
The best PIS candidates have three properties:
- Stable: the content changes less often than the pool scales.
- Material: the image, model, runtime, or dependency download is big enough to affect readiness.
- Deterministic: the inputs can be pinned, versioned, and verified.
Small bootstrap tasks may not be worth baking. If the task only saves a few seconds, normal node provisioning variation can swamp the benefit.
For anything beyond a lab, pin images by digest and verify downloaded artifacts with hashes. If the prepared image needs private artifacts, use managed identity and least-privilege data-plane role-based access control (RBAC) rather than embedding secrets in scripts.
How to set it up
You’ll need Azure CLI 2.85.0 or later and the aks-preview extension at version 21.0.0b5 or above. PIS is currently available in public Azure regions, excluding sovereign clouds and air-gapped environments. It supports Ubuntu, Azure Linux, and Windows node images.
Check your versions before registering the feature flag.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
az extension add --name aks-preview az extension update --name aks-preview az feature register \ --namespace Microsoft.ContainerService \ --name AKSPreparedImageSpecificationPreview az feature show \ --namespace Microsoft.ContainerService \ --name AKSPreparedImageSpecificationPreview \ --query properties.state -o tsv az provider register --namespace Microsoft.ContainerService |
Wait for Registered before continuing.
With the feature registered, get your images into Azure Container Registry (ACR). PIS requires explicit image references with no wildcard support, so the practical approach is to import everything into a single repo and use tags to distinguish images. I used .NET runtime images for testing since they’re small and pull quickly from Microsoft Container Registry.
For repeatable performance tests or production-style validation, use digest-pinned image references instead of mutable tags.
|
1 2 3 4 5 6 7 8 9 10 11 |
ACR="<your-acr-name>" az acr import --name $ACR \ --source mcr.microsoft.com/dotnet/aspnet:8.0 \ --image pis-images:aspnet-8.0 az acr import --name $ACR \ --source mcr.microsoft.com/dotnet/runtime:8.0 \ --image pis-images:runtime-8.0 az acr repository show-tags --name $ACR --repository pis-images -o table |
The last command confirms the tags are in place before you move on.
Create a test cluster attached to your ACR for pull access.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
RG="rg-pis-test" CLUSTER="aks-pis-test" az group create --name $RG --location uksouth az aks create \ --resource-group $RG \ --name $CLUSTER \ --node-count 1 \ --node-vm-size Standard_D2s_v3 \ --attach-acr $ACR \ --generate-ssh-keys |
The cluster takes a few minutes to provision.
Build the image list from your ACR repos and create the PIS version. A PIS can include custom Bash or PowerShell scripts as well as images, but start with images first unless you have a clear host-level dependency to prepare.
Custom scripts run during image build, so a broken script can fail the bake or later scale-up path.
Expect the VHD bake to take 10 to 30 minutes depending on how many images you’re including.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
PIS_NAME="pis-test" PIS_VER="v1" images=() for repo in $(az acr repository list --name $ACR -o tsv); do for tag in $(az acr repository show-tags --name $ACR --repository $repo -o tsv); do images+=("$ACR.azurecr.io/{repo}:${tag}") done done az aks prepared-image-specification create \ --resource-group $RG \ --name $PIS_NAME \ --version $PIS_VER \ --location uksouth \ --container-images "${images[@]}" |
There’s no built-in progress indicator while the VHD bakes. Poll the provisioning state with az aks prepared-image-specification version show --query properties.provisioningState -o tsv until it returns Succeeded before moving on.
Before you add a node pool referencing the PIS, there’s a step the current docs don’t mention. The AKS control plane identity needs Reader access on the resource group containing the PIS resource. Without it the node pool add fails and the error message doesn’t point at permissions. I spent more time on this than I should have.
|
1 2 3 4 5 6 7 8 9 |
CLUSTER_IDENTITY=$(az aks show \ --resource-group $RG \ --name $CLUSTER \ --query identity.principalId -o tsv) az role assignment create \ --role "Reader" \ --assignee $CLUSTER_IDENTITY \ --scope "/subscriptions/<sub-id>/resourceGroups/$RG" |
Give the assignment about 30 seconds to propagate, then add the node pool.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
PIS_ID=$(az aks prepared-image-specification version show \ --resource-group $RG \ --pis-name $PIS_NAME \ --name $PIS_VER \ --query id -o tsv) az aks nodepool add \ --resource-group $RG \ --cluster-name $CLUSTER \ --name pispool \ --node-count 1 \ --node-vm-size Standard_D2s_v3 \ --prepared-image-specification-id $PIS_ID |
Check the node pool’s provisioningState and wait for Succeeded before moving to the verification step.
To confirm the images are actually pre-cached, run a privileged pod on the pispool node and check what crictl reports.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
az aks get-credentials --resource-group $RG --name $CLUSTER --overwrite-existing kubectl apply -f - <<'EOF' apiVersion: v1 kind: Pod metadata: name: image-check namespace: default spec: nodeSelector: agentpool: pispool hostPID: true containers: - name: check image: mcr.microsoft.com/azure-cli:latest command: ["sleep", "3600"] securityContext: privileged: true volumeMounts: - name: containerd mountPath: /run/containerd/containerd.sock volumes: - name: containerd hostPath: path: /run/containerd/containerd.sock EOF kubectl wait --for=condition=Ready pod/image-check --timeout=120s kubectl exec image-check -- crictl \ --runtime-endpoint unix:///run/containerd/containerd.sock images |
Your ACR images should appear in the list with no recent pull event. They were already there when the node booted.
A bug worth knowing about
The biggest issue I hit is a bug in the PIS script generator. AKS generates a bash script to pull and cache images during the VHD bake. The script runs with set -euo pipefail. For certain image tag strings, the generator emits echo"STAGE=pull image=..." without a space between echo and the opening quote. Bash treats that as an unknown command and exits with code 1, failing the entire bake.
When this happens the node pool shows FailedToCreateNodeCustomizationVHD. AKS leaves the build VM running so you can inspect the script.
|
1 2 3 |
az vmss run-command show \ --ids /subscriptions/<sub-id>/resourceGroups/MC_<rg>_<cluster>_<region>/providers/Microsoft.Compute/virtualMachineScaleSets/<pispool-vmss>/virtualMachines/0 \ --name pis-image-cache -o json |
Look at the source.script field. If you see echo"STAGE=..." without a space, that’s the bug. The workaround is to exclude the triggering tag and create a new PIS version. In my testing, sdk-6.0 was the trigger. Something about that tag string hits the generator bug. Other versions of the same image were fine.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# Exclude the offending tag and rebuild the list images=() for tag in $(az acr repository show-tags --name $ACR --repository pis-images -o tsv); do if [ "$tag" != "sdk-6.0" ]; then images+=("$ACR.azurecr.io/pis-images:$tag") fi done PIS_VER_FIXED="v1-fix" az aks prepared-image-specification create \ --resource-group $RG \ --name $PIS_NAME \ --version $PIS_VER_FIXED \ --location uksouth \ --container-images "${images[@]}" |
Once that version reaches Succeeded, update the node pool to reference it and the problematic tag is out of the picture.
Updating when your images change
Creating a new PIS version and rolling the node pool is the same process each time. Re-enumerate ACR to pick up new images and create the next version.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
PIS_VER_NEW="v2" images=() for repo in $(az acr repository list --name $ACR -o tsv); do for tag in $(az acr repository show-tags --name $ACR --repository $repo -o tsv); do images+=("$ACR.azurecr.io/{repo}:${tag}") done done az aks prepared-image-specification create \ --resource-group $RG \ --name $PIS_NAME \ --version $PIS_VER_NEW \ --location uksouth \ --container-images "${images[@]}" NEW_PIS_ID=$(az aks prepared-image-specification version show \ --resource-group $RG \ --pis-name $PIS_NAME \ --name $PIS_VER_NEW \ --query id -o tsv) az aks nodepool update \ --resource-group $RG \ --cluster-name $CLUSTER \ --name pispool \ --prepared-image-specification-id $NEW_PIS_ID |
This maps cleanly onto a CI/CD step. After your build pipeline pushes images to ACR, a follow-up step enumerates the repo, creates a new PIS version, and rolls the node pools. Cold start on those pools is then fast for the life of the sprint.
For a more controlled rollout, keep a standard node pool around as a comparison point. Use the same virtual machine size, AKS node image, networking, and workload on both pools, then measure the thing users actually feel. That means time from scale event to the app serving a representative request.
Track node provisioning duration, image pull duration, pod startup, and any model or dependency readiness checks separately so you can see which part PIS helped.
Wrapping up
PIS is a solid approach to the cold start problem if your image set is stable enough to version. The VHD bake is slow, but that’s a one-time cost per version rather than per node, and it’s work the cluster would otherwise do at scale-out time when you can least afford the latency.
The preview is rough in places. The echo bug in the script generator is a real blocker if it hits your image set, and the missing RBAC documentation will catch people out. Neither of those is a reason to avoid testing it, but they are good reasons to use a disposable resource group and leave yourself time to debug.
The official Microsoft material is also worth reading alongside hands-on testing:
- Prepared Image Specification overview
- Create and manage a Prepared Image Specification
- AKS preview feature support policy
If you’re running workloads where scale-out latency matters and you’re currently using a DaemonSet image puller, this is worth testing now. I’m particularly interested to hear whether others hit the echo bug with different tag patterns. If you try it, let me know how it goes.
0 Comments