Managed Workflows for Apache Airflow (MWAA)
Categories:
2 minute read
LocalStack supports Managed Workflows for Apache Airflow, which allow setup and operation of data pipelines.
Creating and accessing environments
First create an S3 bucket which will be used for Airflow resources.
$ awslocal s3 mb s3://my-mwaa-bucket
Next create the environment with the bucket ARN as follows:
$ awslocal mwaa create-environment --dag-s3-path /dags \
--execution-role-arn r1 \
--network-configuration {} \
--source-bucket-arn arn:aws:s3:::my-mwaa-bucket \
--airflow-version 2.2.2 \
--airflow-configuration-options agent.code=007,agent.name=bond \
--name my-mwaa-env
LocalStack will create the Airflow environment and print the Airflow UI URL and access credentials in the logs:
2022-11-30T17:58:36.533 DEBUG --- [functhread13] l.services.mwaa.provider : Airflow available at http://localhost.localstack.cloud:4510 with username=localstack and password=localstack
LocalStack supports following versions of Apache Airflow:
1.10.12
2.0.2
2.2.2
2.4.3
(default)
Airflow configuration options
Configuration options can be passed to Airflow environments using the AirflowConfigurationOptions
argument.
The options are transformed as follows and then passed to Airflow as environment variables.
agent.code
:007
→AIRFLOW__AGENT__CODE:007
agent.name
:bond
→AIRFLOW__AGENT__NAME:bond
Adding or updating DAGs
To add DAGs to Airflow, upload them to the S3 bucket at the path configured by the DagS3Path
argument.
$ awslocal s3 cp sample_dag.py s3://my-mwaa-bucket/dags
Installing custom plugins
You can install custom plugins to leverage Airflow operators, interfaces or hooks. LocalStack supports plugins packaged as per AWS specifications as described here.
The plugins should be uploaded to S3 bucket configured for the MWAA environment at the /plugins.zip
path, like so:
$ awslocal s3 cp plugins.zip s3://my-mwaa-bucket/plugins.zip
Installing Python dependencies
LocalStack makes it possible to install Python dependencies for Apache Airflow in your environments.
First, create a requirements.txt
file:
boto3==1.17.54
boto==2.49.0
botocore==1.20.54
Next, upload the file to the S3 bucket configured for use by the MWAA environment.
The file must be uploaded to /requirements.txt
:
$ awslocal s3 cp requirements.txt s3://my-mwaa-bucket/requirements.txt
The environment will be updated and will be ready for use with new dependencies.
Unlike production AWS, LocalStack does not install any provider packages. These must be installed using the above steps.
Connections
When using connections to other AWS services within DAGs, please specify either the internal Docker IP address of the LocalStack container or host.docker.internal
.
LocalStack currently does not use the credentials and region from aws_conn_id
.
This information must be explicitly passed in operators, hooks and sensors.
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.