# Kubernetes Fleet Manager Source: /azure/services/kubernetes-fleet-manager/ ## Introduction Azure Kubernetes Fleet Manager manages multiple Kubernetes clusters at scale. It groups Azure Kubernetes Service (AKS) and Azure Arc-enabled Kubernetes clusters into a fleet and orchestrates Kubernetes version and node image updates across them in stages, so that the clusters in one stage are updated before the next stage starts. For more information, see [What is Azure Kubernetes Fleet Manager?](https://learn.microsoft.com/azure/kubernetes-fleet/overview). LocalStack for Azure emulates the Fleet Manager control plane. Fleets, member clusters, update strategies, update runs, and auto-upgrade profiles are Azure Resource Manager resources that you manage with the same Azure CLI commands, and the clusters you join to a fleet are LocalStack AKS clusters that you reach with `kubectl`. An update run builds the same stages, groups, and member placement that Azure builds from your strategy, and it tracks skip, start, and stop, but it upgrades no cluster. The supported APIs are listed in the [API Coverage](#api-coverage) section. ## Getting started This guide is designed for users new to Fleet Manager and assumes basic knowledge of the Azure CLI, AKS, `kubectl`, and `lstk az`. The following example creates a fleet, joins two AKS clusters to it, and starts a two-stage node image update run for them. :::note The `az fleet` commands are part of the `fleet` Azure CLI extension. The Fleet Manager quickstart on Microsoft Learn requires Azure CLI 2.82.0 or later and version 1.8.3 or later of the extension. Install the extension, or update it if it is already installed: ```bash az extension add --name fleet --upgrade ``` ::: Launch LocalStack using your preferred method. For more information, see [Introduction to LocalStack for Azure](/azure/getting-started/). Once the container is running, enable Azure CLI interception by running: ```bash lstk az start-interception ``` This command points the `az` CLI away from the public Azure management REST API and toward the LocalStack for Azure emulator API. To revert this configuration, run: ```bash lstk az stop-interception ``` This reconfigures the `az` CLI to send commands to the official Azure management REST API. ### Create a resource group Create a resource group to hold the fleet and its member clusters: ```bash az group create --name rg-fleet-demo --location westeurope ``` ```bash title="Output" { "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-fleet-demo", "location": "westeurope", "name": "rg-fleet-demo", "properties": { "provisioningState": "Succeeded" }, ... } ``` ### Create a fleet Create a fleet without a hub cluster, which is the Fleet Manager type for multi-cluster Kubernetes and node image updates: ```bash az fleet create \ --resource-group rg-fleet-demo \ --name fleet-demo \ --location westeurope ``` ```bash title="Output" { "hubProfile": null, "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-fleet-demo/providers/Microsoft.ContainerService/fleets/fleet-demo", "location": "westeurope", "name": "fleet-demo", "provisioningState": "Succeeded", "resourceGroup": "rg-fleet-demo", "type": "Microsoft.ContainerService/fleets", ... } ``` The `hubProfile` is `null` because the fleet has no hub cluster. ### Show, list, and update fleets Retrieve the details of the fleet: ```bash az fleet show \ --resource-group rg-fleet-demo \ --name fleet-demo ``` ```bash title="Output" { "hubProfile": null, "location": "westeurope", "name": "fleet-demo", "provisioningState": "Succeeded", "status": { "lastOperationError": null, "lastOperationId": "cd9ff461-8153-49f9-87e6-7490804041e4" }, ... } ``` List the fleets in the resource group: ```bash az fleet list \ --resource-group rg-fleet-demo \ --output table ``` ```bash title="Output" ETag Location Name ProvisioningState ResourceGroup -------------------------------------- ---------- ---------- ------------------- --------------- "59e33f94-aa1d-46c9-959e-fc964d56145a" westeurope fleet-demo Succeeded rg-fleet-demo ``` Tag the fleet. `az fleet update` replaces the fleet's tags with the ones you pass: ```bash az fleet update \ --resource-group rg-fleet-demo \ --name fleet-demo \ --tags environment=dev team=platform ``` ```bash title="Output" { "name": "fleet-demo", "provisioningState": "Succeeded", "tags": { "environment": "dev", "team": "platform" }, ... } ``` ### Create the member clusters Fleet members are existing clusters. Create two single-node AKS clusters, one for each update group this guide uses. Each command returns when the cluster is ready; the [Azure Kubernetes Service (AKS)](/azure/services/aks/) article describes what LocalStack backs a cluster with. ```bash az aks create \ --resource-group rg-fleet-demo \ --name aks-canary \ --node-count 1 \ --generate-ssh-keys ``` ```bash title="Output" { "currentKubernetesVersion": "1.34.4", "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/rg-fleet-demo/providers/Microsoft.ContainerService/managedClusters/aks-canary", "name": "aks-canary", "nodeResourceGroup": "MC_rg-fleet-demo_aks-canary_westeurope", "powerState": { "code": "Running" }, "provisioningState": "Succeeded", ... } ``` ```bash az aks create \ --resource-group rg-fleet-demo \ --name aks-production \ --node-count 1 \ --generate-ssh-keys ``` Read the resource ID of each cluster, which is how a fleet member refers to its cluster: ```bash export CANARY_CLUSTER_ID=$(az aks show --resource-group rg-fleet-demo --name aks-canary --query id --output tsv) export PRODUCTION_CLUSTER_ID=$(az aks show --resource-group rg-fleet-demo --name aks-production --query id --output tsv) ``` ### Join the clusters to the fleet Join each cluster to the fleet as a member, and assign it to an update group. An update strategy selects the members of each of its groups by this update group. ```bash az fleet member create \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --name aks-canary \ --member-cluster-id "$CANARY_CLUSTER_ID" \ --update-group canary ``` ```bash title="Output" { "clusterResourceId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/rg-fleet-demo/providers/Microsoft.ContainerService/managedClusters/aks-canary", "group": "canary", "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-fleet-demo/providers/Microsoft.ContainerService/fleets/fleet-demo/members/aks-canary", "name": "aks-canary", "provisioningState": "Succeeded", "type": "Microsoft.ContainerService/fleets/members", ... } ``` ```bash az fleet member create \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --name aks-production \ --member-cluster-id "$PRODUCTION_CLUSTER_ID" \ --update-group production ``` List the members of the fleet: ```bash az fleet member list \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --query "[].{Name:name, Group:group, State:provisioningState}" \ --output table ``` ```bash title="Output" Name Group State -------------- ---------- --------- aks-canary canary Succeeded aks-production production Succeeded ``` ### Connect to a member cluster With `--member`, `az fleet get-credentials` merges the kubeconfig of that member's cluster into your local configuration, under a context named after the member: ```bash az fleet get-credentials \ --resource-group rg-fleet-demo \ --name fleet-demo \ --member aks-canary ``` ```bash title="Output" WARNING: Merged "aks-canary" as current context in .../.kube/config ``` Query the nodes of the member cluster: ```bash kubectl get nodes ``` ```bash title="Output" NAME STATUS ROLES AGE VERSION aks-nodepool1-7400169-vmss000000 Ready 4m48s v1.36.2+k3s1 k3d-aks-canary-ee80e94c-server-0 Ready control-plane 5m42s v1.36.2+k3s1 ``` The member is a real local cluster, so its node list includes the k3d control-plane node, as described in [Connect with kubectl](/azure/services/aks/#connect-with-kubectl). Without `--member`, the command asks for the credentials of the fleet's hub cluster, which a fleet without a hub does not have, and it fails with `InvalidHubOperation`. ### Define an update strategy An update strategy is a reusable order for update runs: its stages run one after another, and the groups inside a stage run in parallel. Save a strategy that places the canary group before the production group: ```bash cat > stages.json <<'EOF' { "stages": [ { "name": "canary", "groups": [{ "name": "canary" }] }, { "name": "production", "groups": [{ "name": "production" }] } ] } EOF ``` ```bash az fleet updatestrategy create \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --name canary-first \ --stages stages.json ``` ```bash title="Output" { "name": "canary-first", "provisioningState": "Succeeded", "strategy": { "stages": [ { "afterStageWaitInSeconds": 0, "groups": [ { "name": "canary", ... } ], "name": "canary", ... }, { "afterStageWaitInSeconds": 0, "groups": [ { "name": "production", ... } ], "name": "production", ... } ] }, "type": "Microsoft.ContainerService/fleets/updateStrategies", ... } ``` ### Create an update run Create an update run that targets the latest node image for the members and follows the order the strategy defines: ```bash az fleet updaterun create \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --name node-image-rollout \ --upgrade-type NodeImageOnly \ --node-image-selection Latest \ --update-strategy-name canary-first ``` ```bash title="Output" After successfully creating the run, you need to use the following command to start the run:az fleet updaterun start --resource-group=rg-fleet-demo --fleet=fleet-demo --name=node-image-rollout { "managedClusterUpdate": { "nodeImageSelection": { "customNodeImageVersions": [], "type": "Latest" }, "upgrade": { "kubernetesVersion": null, "type": "NodeImageOnly" } }, "name": "node-image-rollout", "provisioningState": "Succeeded", "status": { "stages": [ ... ], "status": { "completedTime": null, "error": null, "startTime": null, "state": "NotStarted" } }, "strategy": { "stages": [ ... ] }, "updateStrategyId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-fleet-demo/providers/Microsoft.ContainerService/fleets/fleet-demo/updateStrategies/canary-first", ... } ``` The run copies the strategy into its own `strategy` when it is created, so a later change to the strategy does not affect it. Show the stages of the run and the members in each one: ```bash az fleet updaterun show \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --name node-image-rollout \ --query "status.stages[].{Stage:name, State:status.state, Members:join(', ', groups[].members[].join(': ', [name, status.state]))}" \ --output table ``` ```bash title="Output" Stage State Members ---------- ---------- -------------------------- canary NotStarted aks-canary: NotStarted production NotStarted aks-production: NotStarted ``` ### Skip a stage and start the run Skip the production stage, so that the run leaves the production cluster out. Skipping a stage skips its groups and their members with it. The `--targets` argument is in preview, so the CLI prints a warning about it. ```bash az fleet updaterun skip \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --name node-image-rollout \ --targets Stage:production \ --query "status.stages[].{Stage:name, State:status.state, Members:join(', ', groups[].members[].join(': ', [name, status.state]))}" \ --output table ``` ```bash title="Output" WARNING: Argument '--targets' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus Stage State Members ---------- ---------- ----------------------- canary NotStarted aks-canary: NotStarted production Skipped aks-production: Skipped ``` Start the run, and wait until it completes: ```bash az fleet updaterun start \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --name node-image-rollout \ --output none ``` ```bash az fleet updaterun wait \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --update-run-name node-image-rollout \ --custom "status.status.state=='Completed'" ``` Show the result of the run: ```bash az fleet updaterun show \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --name node-image-rollout \ --query "status.stages[].{Stage:name, State:status.state, Members:join(', ', groups[].members[].join(': ', [name, status.state]))}" \ --output table ``` ```bash title="Output" Stage State Members ---------- --------- ----------------------- canary Completed aks-canary: Completed production Skipped aks-production: Skipped ``` On Azure, the run stays `Running` while the canary cluster is upgraded, and a member whose node images are already the latest is reported `Skipped`. In LocalStack, a started run completes at once and upgrades nothing; see [Limitations](#limitations). ### Generate update runs from an auto-upgrade profile An auto-upgrade profile generates update runs when a new Kubernetes or node image version becomes available on its channel. Read the ID of the update strategy, then create a profile on the Stable channel that orders its runs by that strategy. `--disabled` keeps the profile from generating runs on its own: ```bash export STRATEGY_ID=$(az fleet updatestrategy show --resource-group rg-fleet-demo --fleet-name fleet-demo --name canary-first --query id --output tsv) ``` ```bash az fleet autoupgradeprofile create \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --name stable-channel \ --channel Stable \ --update-strategy-id "$STRATEGY_ID" \ --disabled ``` ```bash title="Output" { "channel": "Stable", "disabled": true, "name": "stable-channel", "provisioningState": "Succeeded", "type": "Microsoft.ContainerService/fleets/autoUpgradeProfiles", "updateStrategyId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-fleet-demo/providers/Microsoft.ContainerService/fleets/fleet-demo/updateStrategies/canary-first", ... } ``` Generate an update run from the profile on demand: ```bash az fleet autoupgradeprofile generate-update-run \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --auto-upgrade-profile-name stable-channel ``` ```bash title="Output" { "id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-fleet-demo/providers/Microsoft.ContainerService/fleets/fleet-demo/updateRuns/stable-channel-8ed05ce8", "resourceGroup": "rg-fleet-demo" } ``` List the update runs of the fleet. The generated run is built from a copy of the profile's strategy: ```bash az fleet updaterun list \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --query "[].{Name:name, State:status.status.state, Stages:join(', ', strategy.stages[].name)}" \ --output table ``` ```bash title="Output" Name State Stages ----------------------- ---------- ------------------ node-image-rollout Completed canary, production stable-channel-8ed05ce8 NotStarted canary, production ``` ### Remove a member and delete the fleet Remove the production cluster from the fleet. The cluster itself is not deleted: ```bash az fleet member delete \ --resource-group rg-fleet-demo \ --fleet-name fleet-demo \ --name aks-production \ --yes ``` Delete the fleet, together with its remaining member, update strategy, update runs, and auto-upgrade profile: ```bash az fleet delete \ --resource-group rg-fleet-demo \ --name fleet-demo \ --yes ``` ### Teardown Remove the resource group and the AKS clusters it contains: ```bash az group delete \ --name rg-fleet-demo \ --yes ``` ## Features The Fleet Manager emulator supports the following features: - **Fleets**: Create, update, delete, get, and list fleets, per resource group and per subscription. An update replaces the fleet's tags with the ones it carries. - **Member clusters**: Join, update, remove, get, and list members, each assigned to an update group. A member is a LocalStack AKS cluster, so `az fleet get-credentials --member` returns a kubeconfig that `kubectl` can use. - **Update strategies**: Create, update, delete, get, and list reusable strategies made of stages and groups, with an optional wait after a stage. - **Update runs**: Create, get, list, delete, start, stop, and skip update runs. A run takes its order from a strategy defined inline, from a copy of an update strategy of its own fleet that it references, or, with neither, from a single `default` stage that holds every member, and it places the members in the same stages and groups that Azure does. - **Skipping**: Skip stages, groups, members, and after-stage waits. A skip reaches everything under its target, a group or stage that is left with only skipped children is skipped as well, and a run whose stages are all skipped completes at once, as it does on Azure. - **Auto-upgrade profiles**: Create, delete, get, and list profiles, and generate an update run from a profile on demand. The generated run is built from a copy of the profile's update strategy, or from the single `default` stage when the profile has none. ## Limitations - **Update runs upgrade nothing**: Starting a run moves every stage, group, and member that is not skipped straight to `Completed`, and no member cluster changes its Kubernetes version or node image. On Azure, the run stays `Running` while members are upgraded, and a member that is already at the target version is reported `Skipped`. - **A stopped run cannot be started again**: Azure resumes a stopped or failed run from the last unprocessed cluster when you start it again. The emulator keeps a stopped run `Stopped`. - **Members are AKS clusters only**: Azure also accepts Azure Arc-enabled Kubernetes clusters as members, but LocalStack for Azure does not emulate Arc-enabled clusters. - **No hub cluster is provisioned**: A fleet created with `--enable-hub` keeps its hub profile, but no hub cluster exists. The hub's `fqdn`, `kubernetesVersion`, and `portalFqdn` stay empty, and `az fleet get-credentials` without `--member` fails with `InvalidHubOperation`. Workload placement and DNS load balancing, which run on the hub, are not available. - **Managed Fleet Namespaces are stored, not placed**: The emulator accepts the preview `az fleet namespace` commands on any fleet, including one without a hub cluster, which Azure requires for them, and it creates no namespace on a member cluster. - **Cross-cluster networking is not supported**: The emulator does not implement the preview `az fleet clustermeshprofile` commands, which connect member clusters into a Cilium cluster mesh. Do not use them against LocalStack for Azure. - **Approval gates are never reached**: Gates in a strategy are stored with it, but no run stops at one, so the preview `az fleet gate list` command returns an empty list. - **Preview strategy settings are dropped**: `maxAllowedFailures` and `memberSelector` in a stage or group are not kept, so a group always selects members by their update group, and the status of a run reports no `failureCount`. `maxConcurrency` and `afterStageWaitInSeconds` are kept. - **Members carry no system labels or status**: Azure adds `fleet.azure.com/*` labels and a `status` with the last operation to every member. The emulator returns only the labels you set, and no `status`. - **The fleet's managed identity has no principal**: With `--enable-managed-identity`, the identity type is stored, but `principalId` and `tenantId` are returned empty. - **Auto-upgrade profiles generate runs only on demand**: On Azure, an enabled profile automatically creates and executes an update run when AKS releases a new Kubernetes or node image version on its channel. The emulator follows no AKS releases, so a profile never creates a run on its own, and `az fleet autoupgradeprofile generate-update-run` is the only way to get one. - **Generated update runs differ in name and upgrade**: The emulator names a generated run after the profile with a random suffix and sets no `managedClusterUpdate`. Azure names it after the profile, the channel, and the target version, and sets the upgrade that the channel implies. - **Some requests that Azure rejects are accepted**: The emulator does not refuse an update run on a fleet without members, a member whose cluster does not exist, an unsupported Kubernetes version, stopping a run that has not started, starting or stopping a run that has completed, skipping a target that the run does not contain, or a strategy whose groups hold no member. ## Samples No sample is currently available for Azure Kubernetes Fleet Manager resources. Explore end-to-end examples in the [LocalStack for Azure Samples](https://github.com/localstack/localstack-azure-samples) and [AKS Samples and Tutorials for LocalStack for Azure](https://github.com/localstack-samples/aks-samples) repositories. ## API Coverage ### Kubernetes Fleet Manager API coverage Source service: `kubernetes-fleet-manager`. 36 of 36 tracked operations are implemented. Service documentation: /azure/services/kubernetes-fleet-manager/ | Operation | Status | | --- | --- | | AutoUpgradeProfileOperations.GenerateUpdateRun | Implemented | | AutoUpgradeProfiles.CreateOrUpdate | Implemented | | AutoUpgradeProfiles.Delete | Implemented | | AutoUpgradeProfiles.Get | Implemented | | AutoUpgradeProfiles.ListByFleet | Implemented | | FleetManagedNamespaces.CreateOrUpdate | Implemented | | FleetManagedNamespaces.Delete | Implemented | | FleetManagedNamespaces.Get | Implemented | | FleetManagedNamespaces.ListByFleet | Implemented | | FleetManagedNamespaces.Update | Implemented | | FleetMembers.Create | Implemented | | FleetMembers.Delete | Implemented | | FleetMembers.Get | Implemented | | FleetMembers.ListByFleet | Implemented | | FleetMembers.Update | Implemented | | FleetUpdateStrategies.CreateOrUpdate | Implemented | | FleetUpdateStrategies.Delete | Implemented | | FleetUpdateStrategies.Get | Implemented | | FleetUpdateStrategies.ListByFleet | Implemented | | Fleets.CreateOrUpdate | Implemented | | Fleets.Delete | Implemented | | Fleets.Get | Implemented | | Fleets.ListByResourceGroup | Implemented | | Fleets.ListBySubscription | Implemented | | Fleets.ListCredentials | Implemented | | Fleets.Update | Implemented | | Gates.Get | Implemented | | Gates.ListByFleet | Implemented | | Gates.Update | Implemented | | UpdateRuns.CreateOrUpdate | Implemented | | UpdateRuns.Delete | Implemented | | UpdateRuns.Get | Implemented | | UpdateRuns.ListByFleet | Implemented | | UpdateRuns.Skip | Implemented | | UpdateRuns.Start | Implemented | | UpdateRuns.Stop | Implemented |