Crossplane
Overview
Section titled “Overview”Crossplane is a cloud-native control plane framework, which offers an extensible backend that enables orchestrating applications and infrastructure via declarative APIs and resource definitions.
Crossplane offers a native AWS provider which can be used to create and manage AWS cloud resources via the Crossplane platform. For example, it can be used to create S3 buckets, SQS queues, Lambda functions, among many other resources. Crossplane AWS provider supports a comprehensive set of some 900+ resource types.
Getting started
Section titled “Getting started”In the following, we provide a step-by-step guide for installing Crossplane in a local test environment, and creating AWS resources (S3 bucket, SQS queue) in LocalStack via Crossplane.
Prerequisites
Section titled “Prerequisites”- LocalStack running in local Docker
- A local Kubernetes cluster:
- We can use the embedded Kubernetes cluster that ships with modern versions of Docker Desktop (can be easily enabled in the Docker settings)
- Alternatively, you can create a local EKS cluster in LocalStack directly, which will spin up a light-weight embedded
k3d
Kubernetes cluster in your Docker environment
- The
helm
andkubectl
command-line clients installed
Installing Crossplane in local Kubernetes
Section titled “Installing Crossplane in local Kubernetes”Once your kubectl
is configured to point to the local Kubernetes cluster, we first install Crossplane via helm
:
helm repo add crossplane-stable https://charts.crossplane.io/stablehelm repo updatehelm install crossplane crossplane-stable/crossplane --namespace crossplane-system --create-namespace
The installation may take a few minutes.
In parallel, we can install the crossplane
command-line tool.
curl -sL https://raw.githubusercontent.com/crossplane/crossplane/master/install.sh | bashsudo mv crossplane /usr/local/bin
To confirm that the installation was successful, we can run these commands, which should yield output similar to the following:
crossplane version
Client Version: v1.17.0Server Version: v1.17.0
kubectl get crds | grep crossplane
compositions.apiextensions.crossplane.io 2023-09-03T11:30:36Zconfigurations.pkg.crossplane.io 2023-09-03T11:30:36Z
Installing the Crossplane AWS Provider
Section titled “Installing the Crossplane AWS Provider”Once the basic Crossplane installation is running properly, we can proceed with installing the AWS provider. Newer versions of Crossplane promote the use of provider families, which are collections of providers for different groups of resources. For example, there is a separate provider for each individual AWS service (like S3, SQS, Lambda, etc), and in addition provider family provides shared resources for common configuration of all services (e.g., credentials, etc).
In the following, we first install the AWS provider for S3. Note that you can copy/paste the entire multi-line command below into your terminal:
cat <<EOF | kubectl apply -f -apiVersion: pkg.crossplane.io/v1kind: Providermetadata: name: provider-aws-s3spec: package: xpkg.upbound.io/upbound/provider-aws-s3:v0.40.0EOF
We also install the AWS provider for SQS:
cat <<EOF | kubectl apply -f -apiVersion: pkg.crossplane.io/v1kind: Providermetadata: name: provider-aws-sqsspec: package: xpkg.upbound.io/upbound/provider-aws-sqs:v0.40.0EOF
After some time, the providers should get into healthy state, which can be confirmed via kubectl get providers
:
kubectl get providers
NAME INSTALLED HEALTHY PACKAGE AGEupbound-provider-family-aws True True xpkg.upbound.io/upbound/provider-family-aws:v0.40.0 2mprovider-aws-s3 True True xpkg.upbound.io/upbound/provider-aws-s3:v0.40.0 2mprovider-aws-sqs True True xpkg.upbound.io/upbound/provider-aws-sqs:v0.40.0 2m
Next, we install a secret to define the test credentials for the AWS provider:
cat <<EOF | kubectl apply -f -apiVersion: v1kind: Secretmetadata: name: localstack-aws-secretstringData: creds: | [default] aws_access_key_id = test aws_secret_access_key = testEOF
Finally, we create an AWS ProviderConfig
that references the secret created above, and defines a static endpoint
pointing to the LocalStack URL http://host.docker.internal:4566
:
cat <<EOF | kubectl apply -f -apiVersion: aws.upbound.io/v1beta1kind: ProviderConfigmetadata: name: defaultspec: credentials: source: Secret secretRef: name: localstack-aws-secret namespace: default key: creds endpoint: hostnameImmutable: trueTODO: add more services to this list, as needed services: [iam, s3, sqs, sts] url: type: Static static: http://host.docker.internal:4566 skip_credentials_validation: true skip_metadata_api_check: true skip_requesting_account_id: true s3_use_path_style: trueEOF
Deploying sample resources in LocalStack
Section titled “Deploying sample resources in LocalStack”After the Crossplane AWS provider is properly installed and configured, we can proceed with creating some local resources.
First, we create an S3 bucket named crossplane-test-bucket
:
cat <<EOF | kubectl apply -f -apiVersion: s3.aws.upbound.io/v1beta1kind: Bucketmetadata: name: crossplane-test-bucketspec: forProvider: region: us-east-1EOF
If everything is wired up correctly, you should now see some activity in the LocalStack log outputs, where Crossplane starts deploying the S3 bucket against LocalStack.
After some time, the bucket should be transitioning into ready
state within Crossplane:
kubectl get buckets
NAME READY SYNCED EXTERNAL-NAME AGEcrossplane-test-bucket True True crossplane-test-bucket 30s
…
and the bucket it should also be visible when querying the local S3 buckets in LocalStack via awslocal
:
awslocal s3 ls
2023-09-03 15:18:47 crossplane-test-bucket
We can repeat the same exercise for creating a local SQS queue named crossplane-test-queue
:
cat <<EOF | kubectl apply -f -apiVersion: sqs.aws.upbound.io/v1beta1kind: Queuemetadata: name: crossplane-test-queuespec: forProvider: name: crossplane-test-queue region: us-east-1EOF
After some time, the queue should transition into ready
state in Crossplane:
kubectl get queues
NAME READY SYNCED EXTERNAL-NAME AGEcrossplane-test-queue True True http://host.docker.internal:4566/000000000000/crossplane-test-queue 40s
…and the queue should be visible when listing the SQS queues in LocalStack:
awslocal sqs list-queues
{ "QueueUrls": [ "http://localhost:4566/000000000000/crossplane-test-queue" ]}
Summary
Section titled “Summary”The Crossplane AWS provider is a great way to manage AWS resources, and by leveraging the endpoint
configuration of the provider, we can seamlessly run resource deployments against LocalStack.
In this tutorial, we have provided an end-to-end walkthrough of how to provision two simple resources - an S3 bucket, and an SQS queue. Crossplane supports a vast range of additional AWS resource types, as well as advanced operations like updating, deleting, or composing resources. You can refer to the additional reading material to learn and explore more advanced features.
Further Reading
Section titled “Further Reading”- Kubernetes on Docker Desktop: https://docs.docker.com/desktop/kubernetes
- Kubernetes getting started guide: https://kubernetes.io/docs/setup
- EKS Kubernetes clusters on LocalStack: https://docs.localstack.cloud/aws/services/eks/
- Crossplane user docs: https://docs.crossplane.io
- Crossplane AWS provider family: https://marketplace.upbound.io/providers/upbound/provider-family-aws
- Crossplane AWS provider source code: https://github.com/upbound/provider-aws