Kubernetes Fleet Manager

Introduction

Azure Kubernetes Fleet Manager manages multiple Kubernetes clusters at scale. It groups Azure Kubernetes Service (AKS) and Azure Arc-enabled Kubernetes clusters into a fleet and orchestrates Kubernetes version and node image updates across them in stages, so that the clusters in one stage are updated before the next stage starts. For more information, see What is Azure Kubernetes Fleet Manager?.

LocalStack for Azure emulates the Fleet Manager control plane. Fleets, member clusters, update strategies, update runs, and auto-upgrade profiles are Azure Resource Manager resources that you manage with the same Azure CLI commands, and the clusters you join to a fleet are LocalStack AKS clusters that you reach with kubectl. An update run builds the same stages, groups, and member placement that Azure builds from your strategy, and it tracks skip, start, and stop, but it upgrades no cluster. The supported APIs are listed in the API Coverage section.

Getting started

This guide is designed for users new to Fleet Manager and assumes basic knowledge of the Azure CLI, AKS, kubectl, and lstk az. The following example creates a fleet, joins two AKS clusters to it, and starts a two-stage node image update run for them.

The az fleet commands are part of the fleet Azure CLI extension. The Fleet Manager quickstart on Microsoft Learn requires Azure CLI 2.82.0 or later and version 1.8.3 or later of the extension. Install the extension, or update it if it is already installed:

az extension add --name fleet --upgrade

Launch LocalStack using your preferred method. For more information, see Introduction to LocalStack for Azure. Once the container is running, enable Azure CLI interception by running:

lstk az start-interception

This command points the az CLI away from the public Azure management REST API and toward the LocalStack for Azure emulator API. To revert this configuration, run:

lstk az stop-interception

This reconfigures the az CLI to send commands to the official Azure management REST API.

Create a resource group

Create a resource group to hold the fleet and its member clusters:

az group create --name rg-fleet-demo --location westeurope

{
  "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-fleet-demo",
  "location": "westeurope",
  "name": "rg-fleet-demo",
  "properties": {
    "provisioningState": "Succeeded"
  },
  ...
}

Create a fleet

Create a fleet without a hub cluster, which is the Fleet Manager type for multi-cluster Kubernetes and node image updates:

az fleet create \
  --resource-group rg-fleet-demo \
  --name fleet-demo \
  --location westeurope

{
  "hubProfile": null,
  "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-fleet-demo/providers/Microsoft.ContainerService/fleets/fleet-demo",
  "location": "westeurope",
  "name": "fleet-demo",
  "provisioningState": "Succeeded",
  "resourceGroup": "rg-fleet-demo",
  "type": "Microsoft.ContainerService/fleets",
  ...
}

The hubProfile is null because the fleet has no hub cluster.

Show, list, and update fleets

Retrieve the details of the fleet:

az fleet show \
  --resource-group rg-fleet-demo \
  --name fleet-demo

{
  "hubProfile": null,
  "location": "westeurope",
  "name": "fleet-demo",
  "provisioningState": "Succeeded",
  "status": {
    "lastOperationError": null,
    "lastOperationId": "cd9ff461-8153-49f9-87e6-7490804041e4"
  },
  ...
}

List the fleets in the resource group:

az fleet list \
  --resource-group rg-fleet-demo \
  --output table

ETag                                    Location    Name        ProvisioningState    ResourceGroup
--------------------------------------  ----------  ----------  -------------------  ---------------
"59e33f94-aa1d-46c9-959e-fc964d56145a"  westeurope  fleet-demo  Succeeded            rg-fleet-demo

Tag the fleet. az fleet update replaces the fleet’s tags with the ones you pass:

az fleet update \
  --resource-group rg-fleet-demo \
  --name fleet-demo \
  --tags environment=dev team=platform

{
  "name": "fleet-demo",
  "provisioningState": "Succeeded",
  "tags": {
    "environment": "dev",
    "team": "platform"
  },
  ...
}

Create the member clusters

Fleet members are existing clusters. Create two single-node AKS clusters, one for each update group this guide uses. Each command returns when the cluster is ready; the Azure Kubernetes Service (AKS) article describes what LocalStack backs a cluster with.

az aks create \
  --resource-group rg-fleet-demo \
  --name aks-canary \
  --node-count 1 \
  --generate-ssh-keys

{
  "currentKubernetesVersion": "1.34.4",
  "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/rg-fleet-demo/providers/Microsoft.ContainerService/managedClusters/aks-canary",
  "name": "aks-canary",
  "nodeResourceGroup": "MC_rg-fleet-demo_aks-canary_westeurope",
  "powerState": {
    "code": "Running"
  },
  "provisioningState": "Succeeded",
  ...
}

az aks create \
  --resource-group rg-fleet-demo \
  --name aks-production \
  --node-count 1 \
  --generate-ssh-keys

Read the resource ID of each cluster, which is how a fleet member refers to its cluster:

export CANARY_CLUSTER_ID=$(az aks show --resource-group rg-fleet-demo --name aks-canary --query id --output tsv)
export PRODUCTION_CLUSTER_ID=$(az aks show --resource-group rg-fleet-demo --name aks-production --query id --output tsv)

Join the clusters to the fleet

Join each cluster to the fleet as a member, and assign it to an update group. An update strategy selects the members of each of its groups by this update group.

az fleet member create \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --name aks-canary \
  --member-cluster-id "$CANARY_CLUSTER_ID" \
  --update-group canary

{
  "clusterResourceId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/rg-fleet-demo/providers/Microsoft.ContainerService/managedClusters/aks-canary",
  "group": "canary",
  "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-fleet-demo/providers/Microsoft.ContainerService/fleets/fleet-demo/members/aks-canary",
  "name": "aks-canary",
  "provisioningState": "Succeeded",
  "type": "Microsoft.ContainerService/fleets/members",
  ...
}

az fleet member create \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --name aks-production \
  --member-cluster-id "$PRODUCTION_CLUSTER_ID" \
  --update-group production

List the members of the fleet:

az fleet member list \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --query "[].{Name:name, Group:group, State:provisioningState}" \
  --output table

Name            Group       State
--------------  ----------  ---------
aks-canary      canary      Succeeded
aks-production  production  Succeeded

Connect to a member cluster

With --member, az fleet get-credentials merges the kubeconfig of that member’s cluster into your local configuration, under a context named after the member:

az fleet get-credentials \
  --resource-group rg-fleet-demo \
  --name fleet-demo \
  --member aks-canary

WARNING: Merged "aks-canary" as current context in .../.kube/config

Query the nodes of the member cluster:

kubectl get nodes

NAME                               STATUS   ROLES           AGE     VERSION
aks-nodepool1-7400169-vmss000000   Ready    <none>          4m48s   v1.36.2+k3s1
k3d-aks-canary-ee80e94c-server-0   Ready    control-plane   5m42s   v1.36.2+k3s1

The member is a real local cluster, so its node list includes the k3d control-plane node, as described in Connect with kubectl. Without --member, the command asks for the credentials of the fleet’s hub cluster, which a fleet without a hub does not have, and it fails with InvalidHubOperation.

Define an update strategy

An update strategy is a reusable order for update runs: its stages run one after another, and the groups inside a stage run in parallel. Save a strategy that places the canary group before the production group:

cat > stages.json <<'EOF'
{
  "stages": [
    {
      "name": "canary",
      "groups": [{ "name": "canary" }]
    },
    {
      "name": "production",
      "groups": [{ "name": "production" }]
    }
  ]
}
EOF

az fleet updatestrategy create \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --name canary-first \
  --stages stages.json

{
  "name": "canary-first",
  "provisioningState": "Succeeded",
  "strategy": {
    "stages": [
      {
        "afterStageWaitInSeconds": 0,
        "groups": [
          {
            "name": "canary",
            ...
          }
        ],
        "name": "canary",
        ...
      },
      {
        "afterStageWaitInSeconds": 0,
        "groups": [
          {
            "name": "production",
            ...
          }
        ],
        "name": "production",
        ...
      }
    ]
  },
  "type": "Microsoft.ContainerService/fleets/updateStrategies",
  ...
}

Create an update run

Create an update run that targets the latest node image for the members and follows the order the strategy defines:

az fleet updaterun create \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --name node-image-rollout \
  --upgrade-type NodeImageOnly \
  --node-image-selection Latest \
  --update-strategy-name canary-first

After successfully creating the run, you need to use the following command to start the run:az fleet updaterun start --resource-group=rg-fleet-demo --fleet=fleet-demo --name=node-image-rollout
{
  "managedClusterUpdate": {
    "nodeImageSelection": {
      "customNodeImageVersions": [],
      "type": "Latest"
    },
    "upgrade": {
      "kubernetesVersion": null,
      "type": "NodeImageOnly"
    }
  },
  "name": "node-image-rollout",
  "provisioningState": "Succeeded",
  "status": {
    "stages": [
      ...
    ],
    "status": {
      "completedTime": null,
      "error": null,
      "startTime": null,
      "state": "NotStarted"
    }
  },
  "strategy": {
    "stages": [
      ...
    ]
  },
  "updateStrategyId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-fleet-demo/providers/Microsoft.ContainerService/fleets/fleet-demo/updateStrategies/canary-first",
  ...
}

The run copies the strategy into its own strategy when it is created, so a later change to the strategy does not affect it. Show the stages of the run and the members in each one:

az fleet updaterun show \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --name node-image-rollout \
  --query "status.stages[].{Stage:name, State:status.state, Members:join(', ', groups[].members[].join(': ', [name, status.state]))}" \
  --output table

Stage       State       Members
----------  ----------  --------------------------
canary      NotStarted  aks-canary: NotStarted
production  NotStarted  aks-production: NotStarted

Skip a stage and start the run

Skip the production stage, so that the run leaves the production cluster out. Skipping a stage skips its groups and their members with it. The --targets argument is in preview, so the CLI prints a warning about it.

az fleet updaterun skip \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --name node-image-rollout \
  --targets Stage:production \
  --query "status.stages[].{Stage:name, State:status.state, Members:join(', ', groups[].members[].join(': ', [name, status.state]))}" \
  --output table

WARNING: Argument '--targets' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Stage       State       Members
----------  ----------  -----------------------
canary      NotStarted  aks-canary: NotStarted
production  Skipped     aks-production: Skipped

Start the run, and wait until it completes:

az fleet updaterun start \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --name node-image-rollout \
  --output none

az fleet updaterun wait \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --update-run-name node-image-rollout \
  --custom "status.status.state=='Completed'"

Show the result of the run:

az fleet updaterun show \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --name node-image-rollout \
  --query "status.stages[].{Stage:name, State:status.state, Members:join(', ', groups[].members[].join(': ', [name, status.state]))}" \
  --output table

Stage       State      Members
----------  ---------  -----------------------
canary      Completed  aks-canary: Completed
production  Skipped    aks-production: Skipped

On Azure, the run stays Running while the canary cluster is upgraded, and a member whose node images are already the latest is reported Skipped. In LocalStack, a started run completes at once and upgrades nothing; see Limitations.

Generate update runs from an auto-upgrade profile

An auto-upgrade profile generates update runs when a new Kubernetes or node image version becomes available on its channel. Read the ID of the update strategy, then create a profile on the Stable channel that orders its runs by that strategy. --disabled keeps the profile from generating runs on its own:

export STRATEGY_ID=$(az fleet updatestrategy show --resource-group rg-fleet-demo --fleet-name fleet-demo --name canary-first --query id --output tsv)

az fleet autoupgradeprofile create \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --name stable-channel \
  --channel Stable \
  --update-strategy-id "$STRATEGY_ID" \
  --disabled

{
  "channel": "Stable",
  "disabled": true,
  "name": "stable-channel",
  "provisioningState": "Succeeded",
  "type": "Microsoft.ContainerService/fleets/autoUpgradeProfiles",
  "updateStrategyId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-fleet-demo/providers/Microsoft.ContainerService/fleets/fleet-demo/updateStrategies/canary-first",
  ...
}

Generate an update run from the profile on demand:

az fleet autoupgradeprofile generate-update-run \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --auto-upgrade-profile-name stable-channel

{
  "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-fleet-demo/providers/Microsoft.ContainerService/fleets/fleet-demo/updateRuns/stable-channel-8ed05ce8",
  "resourceGroup": "rg-fleet-demo"
}

List the update runs of the fleet. The generated run is built from a copy of the profile’s strategy:

az fleet updaterun list \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --query "[].{Name:name, State:status.status.state, Stages:join(', ', strategy.stages[].name)}" \
  --output table

Name                     State       Stages
-----------------------  ----------  ------------------
node-image-rollout       Completed   canary, production
stable-channel-8ed05ce8  NotStarted  canary, production

Remove a member and delete the fleet

Remove the production cluster from the fleet. The cluster itself is not deleted:

az fleet member delete \
  --resource-group rg-fleet-demo \
  --fleet-name fleet-demo \
  --name aks-production \
  --yes

Delete the fleet, together with its remaining member, update strategy, update runs, and auto-upgrade profile:

az fleet delete \
  --resource-group rg-fleet-demo \
  --name fleet-demo \
  --yes

Teardown

Remove the resource group and the AKS clusters it contains:

az group delete \
  --name rg-fleet-demo \
  --yes

Features

The Fleet Manager emulator supports the following features:

Fleets: Create, update, delete, get, and list fleets, per resource group and per subscription. An update replaces the fleet’s tags with the ones it carries.
Member clusters: Join, update, remove, get, and list members, each assigned to an update group. A member is a LocalStack AKS cluster, so az fleet get-credentials --member returns a kubeconfig that kubectl can use.
Update strategies: Create, update, delete, get, and list reusable strategies made of stages and groups, with an optional wait after a stage.
Update runs: Create, get, list, delete, start, stop, and skip update runs. A run takes its order from a strategy defined inline, from a copy of an update strategy of its own fleet that it references, or, with neither, from a single default stage that holds every member, and it places the members in the same stages and groups that Azure does.
Skipping: Skip stages, groups, members, and after-stage waits. A skip reaches everything under its target, a group or stage that is left with only skipped children is skipped as well, and a run whose stages are all skipped completes at once, as it does on Azure.
Auto-upgrade profiles: Create, delete, get, and list profiles, and generate an update run from a profile on demand. The generated run is built from a copy of the profile’s update strategy, or from the single default stage when the profile has none.

Limitations

Update runs upgrade nothing: Starting a run moves every stage, group, and member that is not skipped straight to Completed, and no member cluster changes its Kubernetes version or node image. On Azure, the run stays Running while members are upgraded, and a member that is already at the target version is reported Skipped.
A stopped run cannot be started again: Azure resumes a stopped or failed run from the last unprocessed cluster when you start it again. The emulator keeps a stopped run Stopped.
Members are AKS clusters only: Azure also accepts Azure Arc-enabled Kubernetes clusters as members, but LocalStack for Azure does not emulate Arc-enabled clusters.
No hub cluster is provisioned: A fleet created with --enable-hub keeps its hub profile, but no hub cluster exists. The hub’s fqdn, kubernetesVersion, and portalFqdn stay empty, and az fleet get-credentials without --member fails with InvalidHubOperation. Workload placement and DNS load balancing, which run on the hub, are not available.
Managed Fleet Namespaces are stored, not placed: The emulator accepts the preview az fleet namespace commands on any fleet, including one without a hub cluster, which Azure requires for them, and it creates no namespace on a member cluster.
Cross-cluster networking is not supported: The emulator does not implement the preview az fleet clustermeshprofile commands, which connect member clusters into a Cilium cluster mesh. Do not use them against LocalStack for Azure.
Approval gates are never reached: Gates in a strategy are stored with it, but no run stops at one, so the preview az fleet gate list command returns an empty list.
Preview strategy settings are dropped: maxAllowedFailures and memberSelector in a stage or group are not kept, so a group always selects members by their update group, and the status of a run reports no failureCount. maxConcurrency and afterStageWaitInSeconds are kept.
Members carry no system labels or status: Azure adds fleet.azure.com/* labels and a status with the last operation to every member. The emulator returns only the labels you set, and no status.
The fleet’s managed identity has no principal: With --enable-managed-identity, the identity type is stored, but principalId and tenantId are returned empty.
Auto-upgrade profiles generate runs only on demand: On Azure, an enabled profile automatically creates and executes an update run when AKS releases a new Kubernetes or node image version on its channel. The emulator follows no AKS releases, so a profile never creates a run on its own, and az fleet autoupgradeprofile generate-update-run is the only way to get one.
Generated update runs differ in name and upgrade: The emulator names a generated run after the profile with a random suffix and sets no managedClusterUpdate. Azure names it after the profile, the channel, and the target version, and sets the upgrade that the channel implies.
Some requests that Azure rejects are accepted: The emulator does not refuse an update run on a fleet without members, a member whose cluster does not exist, an unsupported Kubernetes version, stopping a run that has not started, starting or stopping a run that has completed, skipping a target that the run does not contain, or a strategy whose groups hold no member.

Samples

No sample is currently available for Azure Kubernetes Fleet Manager resources. Explore end-to-end examples in the LocalStack for Azure Samples and AKS Samples and Tutorials for LocalStack for Azure repositories.

API Coverage

36 of 36 operations implemented

Operation ▲	Implemented ▼

Page 1 of 0

Was this page helpful?