Database Migration Service (DMS)
Introduction
Section titled “Introduction”AWS Database Migration Service provides migration solution from databases, data warehouses, and other type of data stores (e.g. S3, SAP). The migration can be homogeneous (source and target have the same type), but often times is heterogeneous as it supports migration from various sources to various targets (self-hosted and AWS services).
LocalStack only supports selected use cases for DMS at the moment. The supported APIs are available on our API Coverage section, which provides information on the extent of DMS integration with LocalStack.
Getting started
Section titled “Getting started”You can run a DMS sample showcasing MariaDB source and Kinesis target from our GitHub repository.
- The sample is using CDK to setup the infrastructure.
- It setups two databases: one external MariaDB (starting in a docker container) and one RDS MariaDB.
- It creates two
cdc
replication tasks, with different table mappings, that will run against the RDS database, - and two
full-load
replication tasks with different table mappings, running against the hosted (containerized) MariaDB.
To follow the sample, simply clone the repository:
git clone https://github.com/localstack-samples/sample-dms-kinesis-rds-mariadb.git
Next, start LocalStack (there is a docker-compose included, setting the ENABLE_DMS=1
flag):
export LOCALSTACK_AUTH_TOKEN=<your-auth-token> # this must be a enterprise license tokendocker-compose up
Now you can install the dependencies, deploy the resources, and run the tests:
# install dependenciesmake install# deploys cdk stack with all required resources (replication instances, tasks, endpoints)make deploy# starts the tasksmake run
You will then see some log output, indicating the status of the ongoing replication:
************STARTING FULL LOAD FLOW************db endpoint: localhost:3306
Cleaning tables Creating tables Inserting data
Added the following authors[{'first_name': 'John', 'last_name': 'Doe'}]
Added the following accounts[{'account_balance': Decimal('1500.00'), 'name': 'Alice'}]
Added the following novels[{'author_id': 1, 'title': 'The Great Adventure'}, {'author_id': 1, 'title': 'Journey to the Stars'}]
****Full Task 1****
Starting Full load task 1 a%Replication Task arn:aws:dms:us-east-1:000000000000:task:FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7 status: startingWaiting for task status stoppedtask='arn:aws:dms:us-east-1:000000000000:task:FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7' status='starting'task='arn:aws:dms:us-east-1:000000000000:task:FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7' status='stopped'
Kinesis events
fetching Kinesis eventReceived: 6 events[{'control': {}, 'metadata': {'operation': 'drop-table', 'partition-key-type': 'task-id', 'partition-key-value': 'FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7', 'record-type': 'control', 'schema-name': 'dms_sample', 'table-name': 'accounts', 'timestamp': '2024-05-23T19:17:33.126Z'}, 'partition_key': 'FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7.dms_sample.accounts'}, {'control': {}, 'metadata': {'operation': 'drop-table', 'partition-key-type': 'task-id', 'partition-key-value': 'FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7', 'record-type': 'control', 'schema-name': 'dms_sample', 'table-name': 'authors', 'timestamp': '2024-05-23T19:17:33.128Z'},.........
Supported Use Cases
Section titled “Supported Use Cases”DMS is in a preview state on LocalStack and only supports some selected use cases:
Source | Target | Migration Types | Serverless Support |
---|---|---|---|
MariaDB (external) | Kinesis | full-load, cdc | Yes |
MySQL (external) | Kinesis | full-load, cdc | Yes |
RDS MariaDB | Kinesis | full-load, cdc | Yes |
RDS MySQL | Kinesis | full-load, cdc | Yes |
S3 | Kinesis | full-load, cdc | Not supported by AWS |
Aurora PostgreSQL | Kinesis | full-load, cdc | No |
RDS PostgreSQL | Kinesis | full-load, cdc | No |
PostgreSQL (external) | Kinesis | full-load, cdc | No |
Serverless
Section titled “Serverless”DMS Serverless can be used in Localstack for the above mentioned supported use cases that are officially supported by AWS.
In order to simulate the different states that the replication config goes through when provisioning, you can set the env DMS_SERVERLESS_STATUS_CHANGE_WAITING_TIME
, which will cause the state-change to wait the configured seconds.
The waiting time is applied for every status change before the replication is actually in running
.
See also the official docs for explanation about the different states.
Be aware that the replication table statistics on AWS is deleted automatically once the replication finished, and the replication configuration deprovisioned.
For parity reasons, this is also true on LocalStack.
In order to delay the deprovisioning, you can use the env DMS_SERVERLESS_DEPROVISIONING_DELAY
, which by default is set to 60 seconds.
Resource Browser
Section titled “Resource Browser”The LocalStack Web Application provides a Resource Browser for managing:
You can access the Resource Browser by opening the LocalStack Web Application in your browser, navigating to the Resources section, and then clicking on Database Migration Service under the Migration and transfer section.
The Resource Browser supports CRD (Create, Read, Delete) operations on DMS resources.
Replication Instances
Section titled “Replication Instances”- Create Replication Instance: To create a new replication instance, click the Create Replication Instance button and enter details such as the Replication Instance Identifier and Replication Instance class.
- View Replication Instance: To view details of a replication instance, click on its ARN.
- Delete Replication Instance: To delete a replication instance, select it, go to Actions, and choose Remove Selected.
Endpoints
Section titled “Endpoints”- Create Endpoint: To create a new endpoint, click on the Create Endpoint button and fill in necessary details such as the Endpoint Identifier, Endpoint Type, and Engine Name.
- View Endpoint: To see the details of an endpoint, click on its ARN. You can further click Connections and test a connection by specifying the Replication Instance ARN.
- Delete Endpoint: To remove an endpoint, select it, navigate to Actions, and click Remove Selected.
Replication Tasks
Section titled “Replication Tasks”- Create Replication Task: To create a new replication task, press the Create Replication Task button and specify the Task Identifier, Source Endpoint Identifier, and Target Endpoint Identifier, among other settings.
- View Replication Task: To review a replication task, click on the task identifier.
- Delete Replication Task: To delete a replication task, choose the task, click on Actions, and select Remove Selected.
Current Limitations
Section titled “Current Limitations”For RDS MariaDB and RDS MySQL it is not yet possible to set custom db-parameters.
In order to make those databases work with cdc
migration for DMS, some default db-parameters are changed upon start if the ENABLE_DMS=1
flag is set:
binlog_checksum=NONEbinlog_row_image=FULLbinlog_format=ROWserver_id=1log_bin=mysqld-bin
For S3 as a source, only the first 1000 files of a table in a bucket are considered for migration.
For PostgreSQL as a source, the ReplicationTaskSettings.BeforeImageSettings
parameter is not supported.
Enum Values for CDC data events
Section titled “Enum Values for CDC data events”To support Enum values for CDC data events, you need to enable the database setting BINLOG_ROW_METADATA=FULL
Migration Type
Section titled “Migration Type”A replication task on LocalStack does currently only support full-load
(migrate existing data) or cdc
(replicate data changes only).
On AWS there is also a combination for those, which is not yet implemented on LocalStack.
ReplicationTaskSettings
Section titled “ReplicationTaskSettings”The ReplicationTaskSettings
for a replication task only considers BeforeImageSettings
, FullLoadSettings.CommitRate
and FullLoadSettings.TargetTablePrepMode
Other Limitations
Section titled “Other Limitations”- Data Validation is not supported
- Reload of tables is not supported
- Task Logs, specifically CloudWatch, and CloudTrail are not supported (table statistics are supported)
- Time Travel is not supported
- Target Metadata Settings:
ParallelLoadThreads
is not supported - Transformation:
"rule-type": "transformation"
is not supported - AWS DMS Schema Conversion Tool is not supported
- AWS DMS Fleet Advisor is not supported
API Coverage
Section titled “API Coverage”Operation ▲ | Implemented | Image |
---|