Database Migration Service (DMS)

Introduction

AWS Database Migration Service provides migration solution from databases, data warehouses, and other type of data stores (e.g. S3, SAP). The migration can be homogeneous (source and target have the same type), but often times is heterogeneous as it supports migration from various sources to various targets (self-hosted and AWS services).

LocalStack only supports selected use cases for DMS at the moment. The supported APIs are available on our API coverage page, which provides information on the extent of DMS integration with LocalStack.

Getting started

You can run a DMS sample showcasing MariaDB source and Kinesis target from our GitHub repository.

• The sample is using CDK to setup the infrastructure.
• It setups two databases: one external MariaDB (starting in a docker container) and one RDS MariaDB.
• It creates two cdc replication tasks, with different table mappings, that will run against the RDS database,
• and two full-load replication tasks with different table mappings, running against the hosted (containerized) MariaDB.

To follow the sample, simply clone the repository:

git clone https://github.com/localstack-samples/sample-dms-kinesis-rds-mariadb.git

Next, start LocalStack (there is a docker-compose included, setting the ENABLE_DMS=1 flag):

export LOCALSTACK_AUTH_TOKEN=<your-auth-token> # this must be a enterprise license token
docker-compose up

Now you can install the dependencies, deploy the resources, and run the tests:

# install dependencies
make install
# deploys cdk stack with all required resources (replication instances, tasks, endpoints)
make deploy
# starts the tasks
make run

You will then see some log output, indicating the status of the ongoing replication:

************
STARTING FULL LOAD FLOW
************
db endpoint: localhost:3306

 Cleaning tables
 Creating tables
 Inserting data

 Added the following authors
[{'first_name': 'John', 'last_name': 'Doe'}]

 Added the following accounts
[{'account_balance': Decimal('1500.00'), 'name': 'Alice'}]

 Added the following novels
[{'author_id': 1, 'title': 'The Great Adventure'},
 {'author_id': 1, 'title': 'Journey to the Stars'}]

****Full Task 1****


 Starting Full load task 1 a%
Replication Task arn:aws:dms:us-east-1:000000000000:task:FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7 status: starting
Waiting for task status stopped
task='arn:aws:dms:us-east-1:000000000000:task:FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7' status='starting'
task='arn:aws:dms:us-east-1:000000000000:task:FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7' status='stopped'

 Kinesis events

fetching Kinesis event
Received: 6 events
[{'control': {},
  'metadata': {'operation': 'drop-table',
               'partition-key-type': 'task-id',
               'partition-key-value': 'FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7',
               'record-type': 'control',
               'schema-name': 'dms_sample',
               'table-name': 'accounts',
               'timestamp': '2024-05-23T19:17:33.126Z'},
  'partition_key': 'FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7.dms_sample.accounts'},
 {'control': {},
  'metadata': {'operation': 'drop-table',
               'partition-key-type': 'task-id',
               'partition-key-value': 'FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7',
               'record-type': 'control',
               'schema-name': 'dms_sample',
               'table-name': 'authors',
               'timestamp': '2024-05-23T19:17:33.128Z'},
...
...
...

Supported Use Cases

DMS is in a preview state on LocalStack and only supports some selected use cases:

Serverless

DMS Serverless can be used for the sources and targets that LocalStack already supports, and that are also supported by AWS.

In order to simulate the different states that the replication config goes through when provisioning, you can set the env DMS_SERVERLESS_STATUS_CHANGE_WAITING_TIME, which will cause the state-change to wait the configured seconds.

The waiting time is applied for every status change before the replication is actually in running. See also the official docs for explanation about the different states.

Be aware that the replication table statistics on AWS is deleted automatically once the replication finished, and the replication configuration deprovisioned.

For parity reasons, this is also true on LocalStack. In order to delay the deprovisioning, you can use the env DMS_SERVERLESS_DEPROVISIONING_DELAY, which by default is set to 60 seconds.

Resource Browser

The LocalStack Web Application provides a Resource Browser for managing:

• Replication Instances
• Endpoints
• Replication Tasks

You can access the Resource Browser by opening the LocalStack Web Application in your browser, navigating to the Resources section, and then clicking on Database Migration Service under the Migration and transfer section.

The Resource Browser supports CRD (Create, Read, Delete) operations on DMS resources.

Replication Instances

• Create Replication Instance: To create a new replication instance, click the Create Replication Instance button and enter details such as the Replication Instance Identifier and Replication Instance class.
• View Replication Instance: To view details of a replication instance, click on its ARN.
• Delete Replication Instance: To delete a replication instance, select it, go to Actions, and choose Remove Selected.

Endpoints

• Create Endpoint: To create a new endpoint, click on the Create Endpoint button and fill in necessary details such as the Endpoint Identifier, Endpoint Type, and Engine Name.
• View Endpoint: To see the details of an endpoint, click on its ARN. You can further click Connections and test a conenction by specifying the Replication Instance ARN.
• Delete Endpoint: To remove an endpoint, select it, navigate to Actions, and click Remove Selected.

Replication Tasks

• Create Replication Task: To create a new replication task, press the Create Replication Task button and specify the Task Identifier, Source Endpoint Identifier, and Target Endpoint Identifier, among other settings.
• View Replication Task: To review a replication task, click on the task identifier.
• Delete Replication Task: To delete a replication task, choose the task, click on Actions, and select Remove Selected.

Current Limitations

For RDS MariaDB and RDS MySQL it is not yet possible to set custom db-parameters. In order to make those databases work with cdc migration for DMS, some default db-parameters are changed upon start if the ENABLE_DMS=1 flag is set:

binlog_checksum=NONE
binlog_row_image=FULL
binlog_format=ROW
server_id=1
log_bin=mysqld-bin

For S3 as a source, only the first 1000 files of a table in a bucket are considered for migration.

Enum Values for CDC data events

To support Enum values for CDC data events, you need to enable the database setting BINLOG_ROW_METADATA=FULL

Migration Type

A replication task on LocalStack does currently only support full-load (migrate existing data) or cdc (replicate data changes only). On AWS there is also a combination for those, which is not yet implemented on LocalStack.

ReplicationTaskSettings

The ReplicationTaskSettings for a replication task only considers BeforeImageSettings, FullLoadSettings.CommitRate and FullLoadSettings.TargetTablePrepMode

Other Limitations

• Data Validation is not supported
• Reload of tables is not supported
• Task Logs, specifically CloudWatch, and CloudTrail are not supported (table statistics are supported)
• Time Travel is not supported
• Target Metadata Settings: ParallelLoadThreads is not supported
• Transformation: "rule-type": "transformation" is not supported
• AWS DMS Schema Conversion Tool is not supported
• AWS DMS Fleet Advisor is not supported