GitLab CI

Use LocalStack in GitLab CI

This page contains easily customisable snippets to show you how to manage LocalStack in a GitLab CI pipeline.

Snippets

Start up Localstack

For LocalStack versions before 3.0.0Under test>variables, add:
LOCALSTACK_HOSTNAME: localhost.localstack.cloud
HOSTNAME_EXTERNAL: localhost.localstack.cloud.

Service

...
variables:
  DOCKER_SOCK: tcp://docker:2375
  DOCKER_HOST: tcp://docker:2375
  DOCKER_TLS_CERTDIR: ""
...
services:
  - name: localstack/localstack:latest
    alias: localstack
  - name: docker:dind
    alias: docker
    command: ["--tls=false"]
...

Container

image: docker:latest

stages:
  - job

job:
  stage: job
  variables:
    ...
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
    AWS_ENDPOINT_URL: "http://localhost.localstack.cloud:4566"
    ...

  services:
    - name: docker:dind
      alias: docker
      command: ["--tls=false"]

  before_script:
    - apk update
    - apk add gcc musl-dev linux-headers py3-pip python3 python3-dev
    - python3 -m pip install localstack awscli
  script:
    - docker pull localstack/localstack:latest
    - dind_ip="$(getent hosts docker | cut -d' ' -f1)"
    - echo "${dind_ip} localhost.localstack.cloud " >> /etc/hosts
    - DOCKER_HOST="tcp://${dind_ip}:2375" localstack start -d

Configure a CI key

You can easily enable LocalStack Pro by using the localstack/localstack-pro image and adding your CI key to the repository’s environment variables. Go to your project’s Settings > CI/CD and expand the Variables section. Select the Add Variable button and fill in the necessary details. After you create a variable, you can use it in the .gitlab-ci.yml file.

However Variables set in the GitLab UI are not passed down to service containers. We need to assign them to variables in the UI, and then re-assign them in our .gitlab-ci.yml

...
variables:
  LOCALSTACK_API_KEY: $LOCALSTACK_API_KEY
...
services:
  - name: localstack/localstack-pro:latest
    alias: localstack
...

You can check the logs of the LocalStack container to see if the activation was successful. If the CI key activation fails, LocalStack container will exit with an error code.

Dump Localstack logs

...
job:
  variables:
    LOCALSTACK_HOST: <LS_HOST>:<LS_PORT>
  script:
  - localstack logs | tee localstack.log
... 

In case of the service setup LOCALSTACK_HOST will be localstack:4566.

Store Localstack state

You can preserve your AWS infrastructure with Localstack in various ways.

Artifact

...
job:
  before_script:
    - (test -f ./ls-state-pod.zip && localstack state import ./ls-state-pod.zip) || true
  script:
  ...
    - localstack state export ./ls-state-pod.zip
  ...
  artifacts:
    paths:
      - $CI_PROJECT_DIR/ls-state-pod.zip
...

More info about Localstack’s state export and import here.

Cache

...
job:
  before_script:
    - (test -f ./ls-state-pod.zip && localstack state import ./ls-state-pod.zip) || true
  script:
  ...
    - localstack state export ./ls-state-pod.zip
  ...
  cache:
    key:
      untracked: true
      files:
        - $CI_PROJECT_DIR/ls-state-pod.zip
    paths:
      - $CI_PROJECT_DIR/ls-state-pod.zip
...

Additional information about state export and import here.

Cloud Pod

...
job:
  before_script:
    - localstack pod load <POD_NAME> || true
  script:
  ...
    - localstack pod save <POD_NAME>
...

Find more information about cloud pods here.

Ephemeral Instance (Preview)

...
variables:
  LOCALSTACK_API_KEY: $LOCALSTACK_API_KEY
...
setup-job:
  stage: build
  before_script:
    - |
      response=$(curl -X POST -d '{"auto_load_pod": "false"}' \
        -H 'ls-api-key: $LOCALSTACK_API_KEY' \
        -H 'authorization: token $LOCALSTACK_API_KEY' \
        -H 'content-type: application/json' \
        https://api.localstack.cloud/v1/previews/my-gitlab-state)
      
      if [ "$endpointUrl" = "null" ] || [ "$endpointUrl" = "" ]; then
        echo "Unable to create preview environment. API response: $response"
        exit 1
      fi
      echo "Created preview environment with endpoint URL: $endpointUrl"

      echo "export AWS_ENDPOINT_URL=$endpointUrl"
      echo "$AWS_ENDPOINT_URL" >> ls-endpoint.env      
  ...
  artifacts:
    reports:
      dotenv: ls-endpoint.env

test-job:
  stage: test
  script:
    - echo "$AWS_ENDPOINT_URL"  # Output is the address of the ephemeral instance
...

Find out more about ephemeral instances here.

Limitations and Known Issues

  • Localstack must be able to reach a docker socket to provision containers for certain services, ie Lambda, EKS, ECS…etc
  • the runner must be able to resolve the Localstack domain (by default localhost.localstack.cloud), see the sample pipelines for a possible solution
  • to be able to separate steps into their own jobs one must preserve Localstack’s state, since Gitlab is not preserving job related containers/services during the pipelines
  • to start up Localstack in Gitlab CI Docker tools are necessary
  • when Localstack run as a container, it’s not accessible during the after_script phase
Last modified May 17, 2024: Add callout shortcode (#1260) (52d305d4e)