Azure Kubernetes Service (AKS)

Introduction

Azure Kubernetes Service (AKS) is Azure’s managed Kubernetes offering. Azure operates the control plane while you manage node pools of worker machines that run your workloads. For more information, see What is Azure Kubernetes Service?.

LocalStack for Azure creates real, working Kubernetes clusters on your machine. az aks create produces a cluster backed by k3d that you can reach with kubectl, so manifests, Helm charts, and operators behave as they would against a cluster in the cloud. The supported APIs are available on our API Coverage section, which provides information on the extent of AKS’s integration with LocalStack.

Getting started

This guide is designed for users new to AKS and assumes basic knowledge of the Azure CLI, kubectl, and our lstk az proxy.

Launch LocalStack using your preferred method. For more information, see Introduction to LocalStack for Azure. Once the container is running, enable Azure CLI interception by running:

lstk az start-interception

This command points the az CLI away from the public Azure management REST API and toward the LocalStack for Azure emulator API. To revert this configuration, run:

lstk az stop-interception

This reconfigures the az CLI to send commands to the official Azure management REST API.

Create a resource group

Create a resource group to hold the cluster:

az group create \
  --name rg-aks-demo \
  --location westeurope

Output

{
  "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-aks-demo",
  "location": "westeurope",
  "managedBy": null,
  "name": "rg-aks-demo",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}

Create a cluster

Create a cluster with a single node in its system node pool:

az aks create \
  --resource-group rg-aks-demo \
  --name aks-demo \
  --node-count 1 \
  --generate-ssh-keys

The command returns when the cluster is ready to use. Locally that takes a couple of minutes: the emulator provisions a k3d cluster, so what you get back is a live API server, not a mock.

Output

{
  "currentKubernetesVersion": "1.34.4",
  "fqdn": "aks-demo-rg-aks-demo-000000-oq7mpqgx.hcp.westeurope.azmk8s.io",
  "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/rg-aks-demo/providers/Microsoft.ContainerService/managedClusters/aks-demo",
  "kubernetesVersion": "1.34",
  "location": "westeurope",
  "name": "aks-demo",
  "nodeResourceGroup": "MC_rg-aks-demo_aks-demo_westeurope",
  "powerState": {
    "code": "Running"
  },
  "provisioningState": "Succeeded",
  ...
}

Show and list clusters

Retrieve the details of a single cluster:

az aks show \
  --resource-group rg-aks-demo \
  --name aks-demo

Output

{
  "currentKubernetesVersion": "1.34.4",
  "dnsPrefix": "aks-demo-rg-aks-demo-000000",
  "kubernetesVersion": "1.34",
  "location": "westeurope",
  "name": "aks-demo",
  "nodeResourceGroup": "MC_rg-aks-demo_aks-demo_westeurope",
  "provisioningState": "Succeeded",
  ...
}

List the clusters in a resource group:

az aks list \
  --resource-group rg-aks-demo \
  --output table

Output

Name      Location    ResourceGroup    KubernetesVersion    CurrentKubernetesVersion    ProvisioningState    Fqdn
--------  ----------  ---------------  -------------------  --------------------------  -------------------  -------------------------------------------------------------
aks-demo  westeurope  rg-aks-demo      1.34                 1.34.4                      Succeeded            aks-demo-rg-aks-demo-000000-oq7mpqgx.hcp.westeurope.azmk8s.io

Update a cluster

az aks update changes the properties of an existing cluster. The following example sets resource tags:

az aks update \
  --resource-group rg-aks-demo \
  --name aks-demo \
  --tags environment=local team=platform

Output

{
  "name": "aks-demo",
  "provisioningState": "Succeeded",
  "tags": {
    "environment": "local",
    "team": "platform"
  },
  ...
}

Connect with kubectl

Merge the cluster credentials into your local kubeconfig:

az aks get-credentials \
  --resource-group rg-aks-demo \
  --name aks-demo \
  --overwrite-existing

Output

Merged "aks-demo" as current context in /home/user/.kube/config

Query the nodes:

kubectl get nodes

Output

NAME                               STATUS   ROLES           AGE    VERSION
aks-nodepool1-5829393-vmss000000   Ready    <none>          99s    v1.36.2+k3s1
k3d-aks-demo-7da4c24d-server-0     Ready    control-plane   2m1s   v1.36.2+k3s1

Note

Unlike in the cloud, where the control plane is hidden, the local cluster also lists its k3d control-plane node. Agent nodes carry the same aks-<pool>-...-vmss naming scheme as real AKS nodes, and the VERSION column reflects the underlying k3s runtime rather than the cluster’s kubernetesVersion.

Manage node pools

Add a user node pool with two nodes:

az aks nodepool add \
  --resource-group rg-aks-demo \
  --cluster-name aks-demo \
  --name workers \
  --mode User \
  --node-count 2

Output

{
  "count": 2,
  "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/rg-aks-demo/providers/Microsoft.ContainerService/managedClusters/aks-demo/agentPools/workers",
  "mode": "User",
  "name": "workers",
  "orchestratorVersion": "1.34",
  "osType": "Linux",
  "provisioningState": "Succeeded",
  ...
}

List the node pools of the cluster:

az aks nodepool list \
  --resource-group rg-aks-demo \
  --cluster-name aks-demo \
  --output table

Output

Name       OsType    VmSize    Count    MaxPods    ProvisioningState    Mode
---------  --------  --------  -------  ---------  -------------------  ------
nodepool1  Linux               1        250        Succeeded            System
workers    Linux               2        250        Succeeded            User

Inspect a single node pool:

az aks nodepool show \
  --resource-group rg-aks-demo \
  --cluster-name aks-demo \
  --name workers

Output

{
  "count": 2,
  "mode": "User",
  "name": "workers",
  "orchestratorVersion": "1.34",
  "osType": "Linux",
  "powerState": {
    "code": "Running"
  },
  "provisioningState": "Succeeded",
  ...
}

Delete the node pool when you no longer need it:

az aks nodepool delete \
  --resource-group rg-aks-demo \
  --cluster-name aks-demo \
  --name workers

Stop, start, and delete

Stop the cluster to free local resources. Unlike AKS, the emulator does not keep the cluster’s Kubernetes objects while it is stopped (see Limitations):

az aks stop \
  --resource-group rg-aks-demo \
  --name aks-demo

Verify that the cluster has stopped by confirming that powerState reports Stopped:

az aks show \
  --resource-group rg-aks-demo \
  --name aks-demo \
  --query powerState.code \
  --output tsv

Start the cluster again. It comes back with its node pools, their number of agent nodes, and its add-ons, and its API server can move to a different port, so run az aks get-credentials again before you use kubectl:

az aks start \
  --resource-group rg-aks-demo \
  --name aks-demo

Delete the cluster once you are done. A deleted cluster cannot be recovered:

az aks delete \
  --resource-group rg-aks-demo \
  --name aks-demo \
  --yes

Teardown

Remove the resource group and any resources it still contains:

az group delete \
  --name rg-aks-demo \
  --yes

Disable Azure CLI interception to point the az CLI back to the official Azure management REST API:

lstk az stop-interception

Autoscale workloads with KEDA

Kubernetes Event-driven Autoscaling (KEDA) scales workloads based on events from external sources, such as the number of messages waiting in a queue, and can scale them down to zero when there is no work. KEDA extends the Kubernetes Horizontal Pod Autoscaler (HPA) rather than replacing it.

The Service Bus and Storage Queue scalers authenticate with a managed identity, which needs the OIDC issuer and Workload Identity. The emulator enables both only when a cluster is created (see Limitations), and the Teardown above removed the demo resources, so start the interception again and create a cluster with the managed KEDA add-on, the OIDC issuer, and Workload Identity:

lstk az start-interception

az group create \
  --name rg-aks-demo \
  --location westeurope

az aks create \
  --resource-group rg-aks-demo \
  --name aks-demo \
  --node-count 1 \
  --enable-keda \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --generate-ssh-keys

az aks get-credentials \
  --resource-group rg-aks-demo \
  --name aks-demo \
  --overwrite-existing

Verify that the add-on is enabled and its deployments are ready:

az aks show \
  --resource-group rg-aks-demo \
  --name aks-demo \
  --query workloadAutoScalerProfile.keda.enabled \
  --output tsv

kubectl get deployments \
  --namespace kube-system \
  --selector app.kubernetes.io/part-of=keda-operator

Output

true

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
keda-admission             1/1     1            1           1m
keda-metrics-apiserver     1/1     1            1           1m
keda-operator              1/1     1            1           1m

Configure Workload Identity

KEDA 2.15 and later use Microsoft Entra Workload ID instead of Azure AD Pod Identity. Create a user-assigned managed identity and federate it to the KEDA operator service account:

KEDA_IDENTITY=aks-keda-identity

az identity create \
  --name "$KEDA_IDENTITY" \
  --resource-group rg-aks-demo \
  --location westeurope

KEDA_CLIENT_ID=$(az identity show \
  --name "$KEDA_IDENTITY" \
  --resource-group rg-aks-demo \
  --query clientId \
  --output tsv)

OIDC_ISSUER=$(az aks show \
  --resource-group rg-aks-demo \
  --name aks-demo \
  --query oidcIssuerProfile.issuerUrl \
  --output tsv)

az identity federated-credential create \
  --name keda-operator \
  --identity-name "$KEDA_IDENTITY" \
  --resource-group rg-aks-demo \
  --issuer "$OIDC_ISSUER" \
  --subject system:serviceaccount:kube-system:keda-operator \
  --audiences api://AzureADTokenExchange

Annotate the operator service account with the identity’s client ID. Then restart the operator so that the Workload Identity webhook injects the federated token into its pods:

kubectl annotate serviceaccount keda-operator \
  --namespace kube-system \
  "azure.workload.identity/client-id=${KEDA_CLIENT_ID}" \
  --overwrite

kubectl rollout restart deployment keda-operator \
  --namespace kube-system

kubectl rollout status deployment keda-operator \
  --namespace kube-system \
  --timeout=300s

Grant the identity the data plane role required by the scaler. This Service Bus example creates a queue and grants Azure Service Bus Data Owner on the namespace:

SERVICE_BUS_NAMESPACE=sbkedademo
SERVICE_BUS_QUEUE=work-items

az servicebus namespace create \
  --name "$SERVICE_BUS_NAMESPACE" \
  --resource-group rg-aks-demo \
  --location westeurope \
  --sku Standard

az servicebus queue create \
  --name "$SERVICE_BUS_QUEUE" \
  --namespace-name "$SERVICE_BUS_NAMESPACE" \
  --resource-group rg-aks-demo

KEDA_PRINCIPAL_ID=$(az identity show \
  --name "$KEDA_IDENTITY" \
  --resource-group rg-aks-demo \
  --query principalId \
  --output tsv)

SERVICE_BUS_ID=$(az servicebus namespace show \
  --name "$SERVICE_BUS_NAMESPACE" \
  --resource-group rg-aks-demo \
  --query id \
  --output tsv)

az role assignment create \
  --role "Azure Service Bus Data Owner" \
  --assignee-object-id "$KEDA_PRINCIPAL_ID" \
  --assignee-principal-type ServicePrincipal \
  --scope "$SERVICE_BUS_ID"

Configure private scaler endpoints

Service Bus and Storage Queue scalers use cloud: Private when targeting the emulator. Derive the endpointSuffix from the endpoints returned by Azure Resource Manager instead of hardcoding it:

SERVICE_BUS_ENDPOINT=$(az servicebus namespace show \
  --name "$SERVICE_BUS_NAMESPACE" \
  --resource-group rg-aks-demo \
  --query serviceBusEndpoint \
  --output tsv)

ARM_ENDPOINT=$(az cloud show \
  --query endpoints.resourceManager \
  --output tsv)

SERVICE_BUS_HOST="${SERVICE_BUS_ENDPOINT#*://}"
SERVICE_BUS_HOST="${SERVICE_BUS_HOST%%/*}"
SERVICE_BUS_HOST="${SERVICE_BUS_HOST%%:*}"
ARM_ENDPOINT="${ARM_ENDPOINT%/}"
ARM_PORT="${ARM_ENDPOINT##*:}"
if [[ "$ARM_PORT" == "$ARM_ENDPOINT" || "$ARM_PORT" == *"/"* ]]; then
  ARM_PORT=443
fi

SERVICE_BUS_SUFFIX="${SERVICE_BUS_HOST#${SERVICE_BUS_NAMESPACE}.}:${ARM_PORT}"
printf '%s\n' "$SERVICE_BUS_SUFFIX"

Output

servicebus.azure.localhost.localstack.cloud:4566

Use the derived suffix in the ScaledObject. The target Deployment can start with zero replicas; KEDA creates and manages its Horizontal Pod Autoscaler:

service-bus-scaler.yaml

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: service-bus-auth
spec:
  podIdentity:
    provider: azure-workload
    identityId: "<managed-identity-client-id>"
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: service-bus-scaler
spec:
  scaleTargetRef:
    name: service-bus-consumer
  pollingInterval: 5
  cooldownPeriod: 30
  minReplicaCount: 0
  maxReplicaCount: 4
  triggers:
    - type: azure-servicebus
      metadata:
        queueName: work-items
        namespace: sbkedademo
        messageCount: "5"
        cloud: Private
        endpointSuffix: "<service-bus-endpoint-suffix>"
      authenticationRef:
        name: service-bus-auth

For an Azure Storage Queue scaler, derive the suffix from the storage account’s primaryEndpoints.queue property and use the same private-cloud pattern:

triggers:
  - type: azure-queue
    metadata:
      queueName: jobs
      accountName: stkedademo
      queueLength: "5"
      cloud: Private
      endpointSuffix: "<storage-queue-endpoint-suffix>"

The Event Hubs scaler doesn’t support Workload Identity in the emulator, so it authenticates with connection strings. Connection strings returned by LocalStack use an sb:// endpoint and include UseDevelopmentEmulator=true, which KEDA and the Azure SDKs use to connect to the emulator without TLS:

triggers:
  - type: azure-eventhub
    metadata:
      consumerGroup: keda-consumer
      unprocessedEventThreshold: "5"
      blobContainer: eh-checkpoints
      checkpointStrategy: blobMetadata
      connectionFromEnv: EVENTHUB_CONNECTION
      storageConnectionFromEnv: STORAGE_CONNECTION

The AKS KEDA tutorials contain complete producer and consumer applications for Service Bus, Storage Queue, and Event Hubs. Each tutorial verifies that a backlog scales its consumer from zero to multiple replicas, drains without dead-lettering messages, and scales back to zero.

Service Bus tutorial output

Observed scale-out: 0 -> 2 -> 3 -> 4
PASS: the consumer drained the work-items queue with no dead-lettered messages
PASS: the sb-consumer deployment scaled back to zero replicas
SUCCESS: KEDA scaled the sb-consumer deployment from zero to 4 replicas and back to zero

Disable KEDA

Delete your KEDA custom resources before disabling the add-on:

kubectl delete \
  scaledobject/service-bus-scaler \
  triggerauthentication/service-bus-auth
kubectl wait \
  --for=delete \
  scaledobject/service-bus-scaler \
  triggerauthentication/service-bus-auth \
  --timeout=60s

Then disable the managed add-on with az aks update:

az aks update \
  --resource-group rg-aks-demo \
  --name aks-demo \
  --disable-keda \
  --query workloadAutoScalerProfile.keda.enabled \
  --output tsv

Output

false

Caution

Delete all ScaledObject, ScaledJob, and TriggerAuthentication resources, or their containing namespaces, before disabling KEDA.

Clean up

Delete the resource group, which removes the cluster, the Service Bus namespace, and the managed identity, then disable Azure CLI interception:

az group delete \
  --name rg-aks-demo \
  --yes

lstk az stop-interception

Run GPU workloads

AKS runs GPU workloads on node pools of NVIDIA GPU VM sizes. LocalStack for Azure advertises the GPUs of these sizes on their nodes even when your machine has no GPU, so the manifests you deploy to AKS schedule unchanged, while the workloads find no CUDA device (see Limitations).

The following walkthrough adds a Spot GPU node pool that scales from zero, deploys the NVIDIA device plugin, runs a job that asks for a GPU, and simulates the eviction of its Spot node. Start the interception again, then create a resource group and a cluster with a system node pool:

lstk az start-interception

az group create \
  --name rg-aks-gpu-demo \
  --location italynorth

az aks create \
  --resource-group rg-aks-gpu-demo \
  --name aks-gpu-demo \
  --node-count 1 \
  --node-vm-size Standard_D2s_v5 \
  --generate-ssh-keys

az aks get-credentials \
  --resource-group rg-aks-gpu-demo \
  --name aks-gpu-demo \
  --overwrite-existing

Add a Spot GPU node pool

Add a pool of Standard_NV18ads_A10_v5 Spot nodes that starts with no node and that the cluster autoscaler grows to one node when a pod needs it. The sku=gpu:NoSchedule taint keeps other workloads off its nodes:

az aks nodepool add \
  --resource-group rg-aks-gpu-demo \
  --cluster-name aks-gpu-demo \
  --name gpupool \
  --node-vm-size Standard_NV18ads_A10_v5 \
  --os-sku AzureLinux \
  --priority Spot \
  --eviction-policy Delete \
  --spot-max-price -1 \
  --enable-cluster-autoscaler \
  --node-count 0 \
  --min-count 0 \
  --max-count 1 \
  --labels workload=gpu \
  --node-taints sku=gpu:NoSchedule

Output

{
  "count": 0,
  "enableAutoScaling": true,
  "gpuProfile": {
    "driver": "Install"
  },
  "maxCount": 1,
  "minCount": 0,
  "mode": "User",
  "name": "gpupool",
  "nodeLabels": {
    "kubernetes.azure.com/scalesetpriority": "spot",
    "workload": "gpu"
  },
  "nodeTaints": [
    "sku=gpu:NoSchedule",
    "kubernetes.azure.com/scalesetpriority=spot:NoSchedule"
  ],
  "osSku": "AzureLinux",
  "provisioningState": "Succeeded",
  "scaleSetEvictionPolicy": "Delete",
  "scaleSetPriority": "Spot",
  "spotMaxPrice": -1.0,
  "vmSize": "Standard_NV18ads_A10_v5",
  ...
}

As on AKS, the pool reports the Spot label and taint along with your own, and it has no node yet:

kubectl get nodes \
  --selector kubernetes.azure.com/agentpool=gpupool

Output

No resources found

Deploy the NVIDIA device plugin

AKS installs the NVIDIA driver on GPU nodes, and the NVIDIA device plugin advertises their GPUs to Kubernetes. The following manifest is the one from Microsoft’s guide, with a node selector for the accelerator=nvidia label AKS gives GPU nodes and a toleration for the Spot taint:

nvidia-device-plugin-ds.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
  namespace: gpu-resources
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-device-plugin-ds
    spec:
      nodeSelector:
        accelerator: nvidia
      tolerations:
      - key: "sku"
        operator: "Equal"
        value: "gpu"
        effect: "NoSchedule"
      - key: "kubernetes.azure.com/scalesetpriority"
        operator: "Equal"
        value: "spot"
        effect: "NoSchedule"
      priorityClassName: "system-node-critical"
      containers:
      - image: nvcr.io/nvidia/k8s-device-plugin:v0.18.0
        name: nvidia-device-plugin-ctr
        env:
          - name: FAIL_ON_INIT_ERROR
            value: "false"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
        volumeMounts:
        - name: device-plugin
          mountPath: /var/lib/kubelet/device-plugins
      volumes:
      - name: device-plugin
        hostPath:
          path: /var/lib/kubelet/device-plugins

Create the namespace and the DaemonSet:

kubectl create namespace gpu-resources

kubectl apply \
  --filename nvidia-device-plugin-ds.yaml

Note

On LocalStack for Azure, the device plugin finds no GPU on the node and waits, because FAIL_ON_INIT_ERROR is false. LocalStack advertises the node’s GPUs itself.

Run a GPU job

Create a job whose pod asks for one GPU, selects the nodes of the GPU pool, and tolerates their taints:

gpu-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: gpu-job
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: Never
      nodeSelector:
        workload: gpu
      tolerations:
      - key: "sku"
        operator: "Equal"
        value: "gpu"
        effect: "NoSchedule"
      - key: "kubernetes.azure.com/scalesetpriority"
        operator: "Equal"
        value: "spot"
        effect: "NoSchedule"
      containers:
      - name: train
        image: busybox:1.36
        command: ["sh", "-c", "while [ ! -f /tmp/release ]; do sleep 1; done"]
        resources:
          limits:
            nvidia.com/gpu: 1

Apply it and wait for its pod to run. The pod stays pending until the cluster autoscaler adds a node to the GPU pool, the only pool whose nodes can run it:

kubectl apply \
  --filename gpu-job.yaml

kubectl wait \
  --for=jsonpath='{.status.ready}'=1 \
  job/gpu-job \
  --timeout=30m

List the node with some of the labels AKS gives it:

kubectl get nodes \
  --selector kubernetes.azure.com/agentpool=gpupool \
  --label-columns accelerator,kubernetes.azure.com/scalesetpriority,topology.kubernetes.io/zone

Output

NAME                             STATUS   ROLES    AGE   VERSION        ACCELERATOR   SCALESETPRIORITY   ZONE
aks-gpupool-9839207-vmss000000   Ready    <none>   10s   v1.36.2+k3s1   nvidia        spot               0

Read the node’s name from the pod, then check the GPU it advertises and its taints:

GPU_NODE=$(kubectl get pods \
  --selector job-name=gpu-job \
  --output jsonpath='{.items[0].spec.nodeName}')

kubectl get node "$GPU_NODE" \
  --output jsonpath='{.status.allocatable.nvidia\.com/gpu}{"\n"}'

kubectl get node "$GPU_NODE" \
  --output jsonpath='{range .spec.taints[*]}{.key}={.value}:{.effect}{"\n"}{end}'

Output

1
sku=gpu:NoSchedule
kubernetes.azure.com/scalesetpriority=spot:NoSchedule

Simulate a Spot eviction

Simulating the eviction of a Spot instance tests how a workload copes with losing its node. Find the scale set and the instance behind the GPU node, and evict it:

NODE_RESOURCE_GROUP=$(az aks show \
  --resource-group rg-aks-gpu-demo \
  --name aks-gpu-demo \
  --query nodeResourceGroup \
  --output tsv)

GPU_SCALE_SET=$(az vmss list \
  --resource-group "$NODE_RESOURCE_GROUP" \
  --query "[?contains(name, 'gpupool')].name" \
  --output tsv)

PROVIDER_ID=$(kubectl get node "$GPU_NODE" \
  --output jsonpath='{.spec.providerID}')
INSTANCE_ID="${PROVIDER_ID##*/}"

az vmss simulate-eviction \
  --resource-group "$NODE_RESOURCE_GROUP" \
  --name "$GPU_SCALE_SET" \
  --instance-id "$INSTANCE_ID"

As on AKS, the node is tainted remediator.kubernetes.azure.com/unschedulable:NoSchedule without being cordoned, its pods are evicted, and the node and its instance are removed. The job’s replacement pod waits for the next node the autoscaler adds. Wait for it to be ready, then release it to complete:

kubectl wait \
  --for=delete \
  "node/$GPU_NODE" \
  --timeout=15m

kubectl wait \
  --for=jsonpath='{.status.ready}'=1 \
  job/gpu-job \
  --timeout=30m

GPU_POD=$(kubectl get pods \
  --selector job-name=gpu-job \
  --field-selector=status.phase=Running \
  --output jsonpath='{.items[0].metadata.name}')

kubectl exec "$GPU_POD" -- touch /tmp/release

kubectl wait \
  --for=condition=Complete \
  job/gpu-job \
  --timeout=30m

kubectl get pods \
  --selector job-name=gpu-job \
  --output wide

Output

NAME            READY   STATUS      RESTARTS   AGE   IP          NODE                             NOMINATED NODE   READINESS GATES
gpu-job-6wxlm   0/1     Completed   0          6m    10.42.5.4   aks-gpupool-9839207-vmss000001   <none>           <none>

Once its node is idle, the pool scales back to zero.

Clean up the GPU walkthrough

Delete the resource group, which removes the cluster and its GPU pool, then disable Azure CLI interception:

az group delete \
  --name rg-aks-gpu-demo \
  --yes

lstk az stop-interception

Features

The local control plane implements the following capabilities:

• Cluster lifecycle: You can create, show, list, update, stop, start, and delete clusters, and az aks get-credentials returns user and admin kubeconfigs that work with kubectl and Helm. The node resource group holds the scale sets, VM instances, network interfaces, load balancers, and kubelet identity that AKS creates for a cluster, along with the managed identities of the Key Vault secrets provider and of cluster extensions.
• Node pools: System and User node pools run as k3d agent nodes that follow the AKS naming scheme. Adding, scaling, and deleting a pool adds and removes real nodes, and the pool’s node labels and taints are applied to its Kubernetes nodes and updated in place. Nodes also carry the labels and taints AKS sets itself, such as kubernetes.azure.com/agentpool, the region and zone labels, accelerator=nvidia on GPU sizes, and the Spot label and taint on Spot pools, and a User pool can start with no nodes.
• Scaling: The cluster autoscaler adds a node to a pool when an unschedulable pod could run on the pool’s nodes, choosing among such pools with the priority expander when the cluster uses it, and removes an idle node, within the pool’s minimum and maximum counts. Node auto-provisioning runs the AKS Karpenter provider, which creates and removes nodes through the AKS Machines API. The Kubernetes Event-driven Autoscaling (KEDA) and Vertical Pod Autoscaler add-ons can be enabled and disabled on a running cluster.
• GPU node pools: Pools of the NVIDIA GPU VM sizes AKS supports advertise their GPUs as nvidia.com/gpu, so GPU workloads, the NVIDIA device plugin DaemonSet, and the cluster autoscaler behave as on AKS on a machine without a GPU.
• Spot evictions: az vmss simulate-eviction evicts a node of a Spot pool as AKS does: the node is tainted remediator.kubernetes.azure.com/unschedulable:NoSchedule, its pods are evicted, and the node and its scale set instance are removed.
• Networking: The Cilium data plane and the Calico network policy engine run on the cluster and enforce Kubernetes network policies as well as their own policy resources, and Advanced Container Networking Services observability enables Hubble on the Cilium data plane. The managed Gateway API installs the standard Gateway API CRDs, which you pair with an implementation such as NGINX Gateway Fabric.
• Load balancing: The Azure cloud controller manager reconciles public and internal LoadBalancer services into the kubernetes and kubernetes-internal load balancers, public IP addresses, and network security group rules of the node resource group. Internal load balancers take their frontend IP address from the cluster’s subnet, including a subnet you bring with --vnet-subnet-id.
• Storage: The Azure Disk CSI driver is enabled by default and gives persistent volumes the behavior of managed disks: data survives a detach and a reattach on another node, and volumes can be expanded online. The Azure Files CSI driver provisions SMB and NFS file shares, the CSI snapshot controller is installed, and the built-in storage classes have the same names as on AKS.
• Secrets: The Azure Key Vault provider for the Secrets Store CSI driver mounts Key Vault secrets and certificates into pods, authenticating with the add-on’s managed identity or with Microsoft Entra Workload ID.
• Identity: Microsoft Entra Workload ID works with a real OIDC issuer, so pods can exchange service account tokens for Azure credentials without secrets. The control plane can use a system-assigned or a user-assigned managed identity, and each cluster gets a kubelet identity in its node resource group.
• Run Command: az aks command invoke runs a command, with any files attached through --file, in a transient pod on the cluster and returns its output and exit code, and az aks command result retrieves a result by its ID. az aks update --disable-run-command turns the feature off.
• Managed namespaces: az aks namespace add creates the namespace on the cluster with its default resource quota and network policy, and applies the adoption and delete policies.
• Cluster extensions: The Microsoft.AppConfiguration extension, created with az k8s-extension create, installs the Azure App Configuration Kubernetes Provider with its own managed identity. It is the only extension type available: creating any other type, such as microsoft.flux, fails with ExtensionTypeRegistrationGetFailed.
• Management APIs: Maintenance configurations, agent pool snapshots, trusted access role bindings, upgrade profiles, the supported Kubernetes versions, and Azure Kubernetes Fleet Manager resources are available through Azure Resource Manager, so scripts and templates that create or read them run unchanged.
• Tooling: The same clusters can be provisioned with the Azure CLI, Terraform, or Bicep.

Limitations

• Stopping a cluster discards its workloads: az aks stop deletes the backing k3d cluster, and az aks start creates a new one from the stored configuration. AKS keeps every object except standalone pods while a cluster is stopped, whereas the restarted emulator cluster has none of the workloads or other Kubernetes objects you created. Managed namespaces and cluster extensions are not applied again either, although Azure Resource Manager still lists them, so recreate them after a start. Its API server can also move to a different port, so run az aks get-credentials again after a start.
• Clusters do not survive an emulator restart: LocalStack deletes the k3d clusters when it shuts down and does not recreate them when it starts again.
• Upgrades and maintenance operations have no effect: Every cluster runs the same k3s release, whatever kubernetesVersion or orchestratorVersion you request, and the emulator also accepts versions that AKS does not offer. az aks upgrade, az aks nodepool upgrade, and node image upgrades do not replace any node or change the Kubernetes version that the cluster runs, and az aks rotate-certs and az aks operation-abort are accepted without rotating certificates or cancelling operations.
• The network profile is stored, not applied: The network plugin and its mode, such as Azure CNI Overlay or kubenet, and the pod and service address ranges are recorded without changing how the cluster is networked. Pods get their addresses from the cluster’s own CNI, which is flannel unless you select Cilium or Calico, so --network-plugin none does not let you install your own CNI. Of the Advanced Container Networking Services features, only observability is configured, and only on the Cilium data plane. az aks update ignores network profile changes, such as --enable-acns: they are neither stored nor applied.
• Network policies are always enforced: Every cluster created without Cilium or Calico runs the k3s network policy controller, so NetworkPolicy objects take effect even without a network policy engine, where AKS does not enforce them.
• Load balancer IP addresses are not reachable: The external IP address that the cloud controller manager assigns to a LoadBalancer service exists only in the emulated network, so use kubectl port-forward to reach the service from your machine. The cloud controller manager also needs the cluster’s managed identity: on a cluster that uses a service principal, LoadBalancer services stay pending.
• API server access settings are ignored: Private clusters, authorized IP ranges, and API server VNet integration are not emulated. The API server is always reachable from your machine, and disableRunCommand is the only apiServerAccessProfile setting that the emulator stores.
• Microsoft Entra ID and Azure RBAC are not enforced: The Microsoft Entra ID settings, such as --enable-azure-rbac, are stored, and --disable-local-accounts is ignored. Every kubeconfig, with or without --admin, carries the same cluster administrator credentials and needs no kubelogin, and Azure role assignments do not control access to Kubernetes resources.
• OIDC issuer and Workload ID must be enabled at creation: Every cluster reports an OIDC issuer URL, as AKS does for new clusters on Kubernetes 1.34 and later, but the API server issues tokens from it only on clusters created with --enable-oidc-issuer or --enable-workload-identity, and the Workload ID webhook is installed only with --enable-workload-identity. az aks update does not enable either feature on an existing cluster, so pass both flags to az aks create.
• Most add-ons are not installed: The Azure Key Vault secrets provider is the only entry of addonProfiles that the emulator installs, and only if it was enabled when the cluster was created. Other add-ons, such as monitoring and Azure Policy, are stored without installing anything, and az aks enable-addons and az aks disable-addons do not change an existing cluster. The application routing add-on, the Istio-based service mesh add-on, Azure Monitor managed Prometheus, and the Azure Blob CSI driver are not supported.
• Add-on installation failures are only logged: Add-ons are installed on a best-effort basis. If an add-on fails to install, the cluster still reports Succeeded and the add-on as enabled, so check the LocalStack logs when a component does not come up. A cluster extension that fails to install reports Failed.
• The cluster autoscaler is emulated: An emulator control loop takes the place of the upstream cluster autoscaler. It checks an unschedulable pod against the nodes of each pool for the node selector, required node affinity, taints and tolerations, and CPU, memory, and GPU requests, but not for pod affinity, topology spread constraints, or volume topology. It adds one node to a pool at a time, and doesn’t grow a pool that already has a node running only DaemonSet pods. Of autoScalerProfile, it reads only expander: priority takes the pool priorities from the cluster-autoscaler-priority-expander ConfigMap, and any other expander picks a pool at random. It removes the node that has been idle the longest, without draining it. Set LS_AZURE_AKS_SCALING_INTERVAL to change how often it checks the cluster (15 seconds by default) and LS_AZURE_AKS_SCALE_DOWN_AFTER to change how long a node must stay idle before the pool shrinks (30 seconds by default).
• KEDA differs from the managed add-on: The metrics server and admission webhooks run as the keda-metrics-apiserver and keda-admission deployments of the upstream KEDA release, where the AKS add-on names them keda-operator-metrics-apiserver and keda-admission-webhooks. The Event Hubs scaler cannot authenticate with Workload Identity against the emulator, so it uses connection strings.
• Node auto-provisioning creates no default node pools: AKS creates the default and system-surge Karpenter NodePool resources, while the emulator creates none, so define your own NodePool and AKSNodeClass before Karpenter provisions any node. Karpenter is installed only when the cluster starts, so enable node auto-provisioning with az aks create: az aks update --node-provisioning-mode Auto stores the setting without installing Karpenter.
• Node sizes and operating systems are approximated: For the VM sizes in the emulator’s catalog, a pool’s size shapes the allocatable CPU and memory of its nodes, and any other size is shaped as Standard_D2s_v5. A pool created without a VM size reports none, as the az aks nodepool list output in Getting started shows, and its nodes are not shaped, whereas AKS picks a default size. Node capacity is that of your machine, so a size larger than your machine gets its whole capacity. The NVIDIA GPU sizes advertise their GPUs without a GPU behind them: GPU pods schedule as on AKS, but the NVIDIA device plugin finds no device and CUDA isn’t available to the pods. Every node runs Linux on your machine’s CPU architecture, including the nodes of a Windows node pool.
• Most node pool settings have no effect: Settings such as maxPods, the OS SKU, the OS disk options, FIPS, node public IP addresses, and the maxSurge and maxUnavailable upgrade settings don’t change the nodes, apart from the node labels AKS derives from some of them, such as kubernetes.azure.com/os-sku. Others, such as pod subnets, custom kubelet or Linux OS configuration, and the drain timeout, node soak duration, and undrainable node behavior upgrade settings, are neither applied nor returned by the API. az aks nodepool stop and az aks nodepool start leave the pool’s nodes running.
• Spot nodes are evicted only on request: Azure evicts Spot nodes when it needs the capacity back, while LocalStack evicts one only when you run az vmss simulate-eviction on an instance of a Spot node pool’s scale set, and it removes the node sooner than AKS does. A pool with the Deallocate eviction policy is evicted as with Delete.
• Run Command runs as cluster administrator: The command pod runs with a cluster-admin service account rather than your Microsoft Entra identity, each cluster keeps only its latest 100 results, and a command times out after the number of seconds set in LS_AZURE_AKS_RUN_COMMAND_TIMEOUT (600 by default).
• Managed namespaces are not locked: AKS blocks changes made through the Kubernetes API to a managed namespace and to its defaultresourcequota and defaultnetworkpolicy objects, while the emulator does not, so they stay editable with kubectl.
• Some resources are stored only: Maintenance configurations and the auto-upgrade channels never schedule or run an upgrade, and the pricing tier is always reported as Free. The emulator ignores the creationData of an agent pool: az aks nodepool add --snapshot-id still gets the snapshot’s Kubernetes version, VM size, OS type, and OS SKU because the Azure CLI copies them into the request, but a template that sets only creationData.sourceResourceId gets the default configuration. Trusted access role bindings grant no permissions inside the cluster, and Azure Kubernetes Fleet Manager update runs do not upgrade member clusters.
• Azure Files NFS volumes depend on the host: The NFS backend image has no linux/arm64 build, and its NFS server does not start on every Docker host, so an NFS share can be created and still fail to mount, for example on Apple Silicon.

Cluster-creation scripts

The aks-samples repository provides two interchangeable scripts that provision a production-shaped cluster, complete with a virtual network, a container registry, a Log Analytics workspace, and system and user node pools. Both scripts run unchanged against real Azure and the emulator; they differ only in the cluster identity:

Script	Cluster identity	When to use
01-system-assigned-managed-identity.sh	System-assigned managed identity	Simplest option: Azure creates and manages the identity lifecycle together with the cluster.
01-user-assigned-managed-identity.sh	User-assigned managed identity	Use when you need a stable, pre-created identity that can be reused across resources and granted role assignments ahead of time.

Both scripts are idempotent and safe to re-run. The same folder also contains optional add-on installers for Prometheus, the NGINX ingress controller, the Gateway API CRDs, NGINX Gateway Fabric, and cert-manager.

Samples

Every sample deploys the same Vacation Planner single-page web application, available in two versions: a Python Flask app and an ASP.NET Core Razor Pages app. Only the data service, its provisioning, and the way the app authenticates to it change from one sample to the next.

Sample	Description	Languages
web-app-sql-database	Stores activities in an Azure SQL Database, connecting with a SQL login over TDS.	Python, .NET
web-app-mysql-flexible-server	Stores activities in an Azure Database for MySQL flexible server.	Python, .NET
web-app-postgresql-flexible-server	Stores activities in an Azure Database for PostgreSQL flexible server.	Python, .NET
web-app-in-cluster-postgresql	Stores activities in an in-cluster PostgreSQL database deployed as a Kubernetes StatefulSet, with a primary and two streaming replicas.	Python, .NET
web-app-cosmosdb-mongodb-api	Stores activities in a collection of an Azure Cosmos DB for MongoDB account.	Python, .NET
web-app-cosmosdb-nosql-api	Stores activities in a container of an Azure Cosmos DB for NoSQL account.	Python, .NET
web-app-blob-storage	Stores activities in an Azure Blob Storage container, using a connection string.	Python, .NET
web-app-file-storage	Stores activities as text files on an Azure Files share mounted by the Azure Files CSI driver, over either SMB or NFS.	Python, .NET
web-app-managed-identity	Stores activities in Azure Blob Storage, authenticating with Microsoft Entra Workload ID instead of a secret, and optionally exposes the app through the Gateway API with a managed TLS certificate.	Python, .NET
web-app-app-configuration	Stores activities in an Azure Database for PostgreSQL flexible server, reading every connection setting from an Azure App Configuration store through the Microsoft.AppConfiguration cluster extension, with the credentials kept in Azure Key Vault as Key Vault references.	Python, .NET

Tutorials

The same repository includes standalone tutorials that exercise individual AKS capabilities. Unlike the samples, they do not deploy the web application:

Tutorial	Description
policies	Kubernetes network policies that enforce zero-trust traffic control with Calico and Cilium: cluster-wide default-deny, DNS-aware egress, and L3/L4/L7 ingress.
ccm	Exercises the Azure cloud controller manager: public and internal LoadBalancer services, source ranges, the nodeIP backend-pool variant, and an NGINX ingress controller.
gateway-api	Enables the managed Gateway API CRDs, installs NGINX Gateway Fabric, and routes traffic to a backend through a Gateway and an HTTPRoute.
keda	Event-driven autoscaling with the KEDA add-on: a producer creates a backlog on an Azure event source, and a ScaledObject scales a consumer from zero to four replicas and back.
keda/service-bus	Scales a consumer on an Azure Service Bus queue with the azure-servicebus scaler, authenticating with Microsoft Entra Workload ID. Start here if you are new to KEDA.
keda/queue-storage	Scales a consumer on an Azure Storage queue with the azure-queue scaler. Workload identity is used end to end, so no data-plane secret exists anywhere.
keda/event-hubs	Scales a consumer on an Azure Event Hubs hub with the azure-eventhub scaler, whose backlog is the distance between the last enqueued event and the consumer group’s blob checkpoints.
key-vault-csi-driver	Mounts secrets from Azure Key Vault into a pod with the Secrets Store CSI driver, in both the workload identity and the user-assigned managed identity access modes.
terraform/tags-labels-taints	Deploys a modular, feature-rich AKS stack with Terraform, steering workloads across agent pools with Azure resource tags, node labels, and taints, then validates them through both the ARM and Kubernetes APIs.
bicep/tags-labels-taints	The same modular AKS stack built with Bicep, with parameters and outputs that mirror the Terraform tutorial one-to-one.

API Coverage

83 of 156 operations implemented

Formats for tools and AIPage MarkdownCoverage MarkdownCoverage JSONCoverage catalog

Operation ▲	Implemented ▼

PreviousPage 1 of 0Next