# Database Migration Service (DMS)

Source: /aws/services/dms/

## Introduction

AWS Database Migration Service provides migration solution from databases, data warehouses, and other type of data stores (e.g. S3, SAP).
The migration can be homogeneous (source and target have the same type), but often times is heterogeneous as it supports migration from various sources to various targets (self-hosted and AWS services).

LocalStack only supports selected use cases for DMS at the moment.
The supported APIs are available on our [API Coverage section](#api-coverage), which provides information on the extent of DMS integration with LocalStack.

:::note
DMS is in a preview state, supporting only [selected use cases](#supported-use-cases).
You need to set the env `ENABLE_DMS=1` in order to activate it.
:::

## Getting started

You can run a DMS sample showcasing MariaDB source and Kinesis target from our [GitHub repository](https://github.com/localstack-samples/sample-dms-kinesis-rds-mariadb/).

* The sample is using CDK to setup the infrastructure.
* It setups two databases: one external MariaDB (starting in a docker container) and one RDS MariaDB.
* It creates two `cdc` replication tasks, with different table mappings, that will run against the RDS database,
* and two `full-load` replication tasks with different table mappings, running against the hosted (containerized) MariaDB.

To follow the sample, simply clone the repository:

```bash
git clone https://github.com/localstack-samples/sample-dms-kinesis-rds-mariadb.git
```

Next, start LocalStack (there is a docker-compose included, setting the `ENABLE_DMS=1` flag):

```bash
export LOCALSTACK_AUTH_TOKEN=<your-auth-token> # this must be a enterprise license token
docker-compose up
```

Now you can install the dependencies, deploy the resources, and run the tests:

```bash
# install dependencies
make install
# deploys cdk stack with all required resources (replication instances, tasks, endpoints)
make deploy
# starts the tasks
make run
```

You will then see some log output, indicating the status of the ongoing replication:

```bash
************
STARTING FULL LOAD FLOW
************
db endpoint: localhost:3306

 Cleaning tables
 Creating tables
 Inserting data

 Added the following authors
[{'first_name': 'John', 'last_name': 'Doe'}]

 Added the following accounts
[{'account_balance': Decimal('1500.00'), 'name': 'Alice'}]

 Added the following novels
[{'author_id': 1, 'title': 'The Great Adventure'},
 {'author_id': 1, 'title': 'Journey to the Stars'}]

****Full Task 1****


 Starting Full load task 1 a%
Replication Task arn:aws:dms:us-east-1:000000000000:task:FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7 status: starting
Waiting for task status stopped
task='arn:aws:dms:us-east-1:000000000000:task:FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7' status='starting'
task='arn:aws:dms:us-east-1:000000000000:task:FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7' status='stopped'

 Kinesis events

fetching Kinesis event
Received: 6 events
[{'control': {},
  'metadata': {'operation': 'drop-table',
               'partition-key-type': 'task-id',
               'partition-key-value': 'FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7',
               'record-type': 'control',
               'schema-name': 'dms_sample',
               'table-name': 'accounts',
               'timestamp': '2024-05-23T19:17:33.126Z'},
  'partition_key': 'FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7.dms_sample.accounts'},
 {'control': {},
  'metadata': {'operation': 'drop-table',
               'partition-key-type': 'task-id',
               'partition-key-value': 'FQWFF7YIZ4VGQHBIXCLI9FJTUUS17NSECIM0UR7',
               'record-type': 'control',
               'schema-name': 'dms_sample',
               'table-name': 'authors',
               'timestamp': '2024-05-23T19:17:33.128Z'},
...
...
...
```

## Supported Use Cases

DMS is in a preview state on LocalStack and only supports some selected use cases:

| Source                | Target      | Migration Types | Serverless Support   |
| -                     | -           | -               | -                    |
| MariaDB (external)    | Kinesis     | full-load, cdc  | Yes                  |
| MySQL (external)      | Kinesis     | full-load, cdc  | Yes                  |
| RDS MariaDB           | Kinesis     | full-load, cdc  | Yes                  |
| RDS MySQL             | Kinesis     | full-load, cdc  | Yes                  |
| S3                    | Kinesis     | full-load, cdc  | Not supported by AWS |
| Aurora PostgreSQL     | Kinesis     | full-load, cdc  | No                   |
| RDS PostgreSQL        | Kinesis     | full-load, cdc  | No                   |
| PostgreSQL (external) | Kinesis     | full-load, cdc  | No                   |

## Serverless

[DMS Serverless](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Serverless.html) can be used in Localstack for the above mentioned supported use cases that are [officially supported by AWS](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Serverless.Components.html#CHAP_Serverless.SupportedVersions).

In order to simulate the different states that the replication config goes through when provisioning, you can set the env `DMS_SERVERLESS_STATUS_CHANGE_WAITING_TIME`, which will cause the state-change to wait the configured seconds.

The waiting time is applied for every status change before the replication is actually in `running`.
See also the [official docs for explanation about the different states](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Serverless.Components.html).

Be aware that the replication table statistics on AWS is deleted automatically once the replication finished, and the replication configuration deprovisioned.

For parity reasons, this is also true on LocalStack.
In order to delay the deprovisioning, you can use the env `DMS_SERVERLESS_DEPROVISIONING_DELAY`, which by default is set to 60 seconds.

## Resource Browser

The LocalStack Web Application provides a Resource Browser for managing:

* [Replication Instances](https://app.localstack.cloud/inst/default/resources/dms/replication-instances)
* [Endpoints](https://app.localstack.cloud/inst/default/resources/dms/endpoints)
* [Replication Tasks](https://app.localstack.cloud/inst/default/resources/dms/replication-tasks)

You can access the Resource Browser by opening the LocalStack Web Application in your browser, navigating to the **Resources** section, and then clicking on **Database Migration Service** under the **Migration and transfer** section.

![DMS Resource Browser](/images/aws/dms-resource-browser.png)

The Resource Browser supports CRD (Create, Read, Delete) operations on DMS resources.

### Replication Instances

* **Create Replication Instance**: To create a new replication instance, click the **Create Replication Instance** button and enter details such as the Replication Instance Identifier and Replication Instance class.
* **View Replication Instance**: To view details of a replication instance, click on its ARN.
* **Delete Replication Instance**: To delete a replication instance, select it, go to **Actions**, and choose **Remove Selected**.

### Endpoints

* **Create Endpoint**: To create a new endpoint, click on the **Create Endpoint** button and fill in necessary details such as the Endpoint Identifier, Endpoint Type, and Engine Name.
* **View Endpoint**: To see the details of an endpoint, click on its ARN.
  You can further click **Connections** and test a connection by specifying the Replication Instance ARN.
* **Delete Endpoint**: To remove an endpoint, select it, navigate to **Actions**, and click **Remove Selected**.

### Replication Tasks

* **Create Replication Task**: To create a new replication task, press the **Create Replication Task** button and specify the Task Identifier, Source Endpoint Identifier, and Target Endpoint Identifier, among other settings.
* **View Replication Task**: To review a replication task, click on the task identifier.
* **Delete Replication Task**: To delete a replication task, choose the task, click on **Actions**, and select **Remove Selected**.

## Current Limitations

For RDS MariaDB and RDS MySQL it is not yet possible to set custom db-parameters.
In order to make those databases work with `cdc` migration for DMS, some default db-parameters are changed upon start if the `ENABLE_DMS=1` flag is set:

```bash
binlog_checksum=NONE
binlog_row_image=FULL
binlog_format=ROW
server_id=1
log_bin=mysqld-bin
```

For S3 as a source, only the first 1000 files of a table in a bucket are considered for migration.

For PostgreSQL as a source, the `ReplicationTaskSettings.BeforeImageSettings` parameter is not supported.

### Enum Values for CDC data events

To support Enum values for CDC data events, you need to enable the database setting `BINLOG_ROW_METADATA=FULL`

### Migration Type

A replication task on LocalStack does currently only support `full-load` (migrate existing data) or `cdc` (replicate data changes only).
On AWS there is also a combination for those, which is not yet implemented on LocalStack.

### ReplicationTaskSettings

The `ReplicationTaskSettings` for a [replication task](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.html) only considers `BeforeImageSettings`, `FullLoadSettings.CommitRate` and `FullLoadSettings.TargetTablePrepMode`

### Other Limitations

* [Data Validation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Validating.html#CHAP_Validating.TaskStatistics) is not supported
* [Reload](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.ReloadTables.html) of tables is not supported
* [Task Logs](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Monitoring.html#CHAP_Monitoring.ManagingLogs), specifically CloudWatch, and CloudTrail are not supported (table statistics are supported)
* [Time Travel](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.TimeTravel.html) is not supported
* [Target Metadata Settings](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.TargetMetadata.html): `ParallelLoadThreads` is not supported
* [Transformation](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TableMapping.SelectionTransformation.Transformations.html): `"rule-type": "transformation"` is not supported
* [AWS DMS Schema Conversion Tool](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_SchemaConversion.html) is not supported
* [AWS DMS Fleet Advisor](https://docs.aws.amazon.com/dms/latest/userguide/CHAP_FleetAdvisor.html) is not supported

## API Coverage


### Database Migration Service API coverage

Source service: `dms`. 33 of 119 tracked operations are implemented.

Service documentation: /aws/services/dms/
License availability: available starting with the Ultimate plan. See /aws/licensing/ for current plan details.

| Operation | Status |
| --- | --- |
| AddTagsToResource | Implemented |
| ApplyPendingMaintenanceAction | Not implemented |
| BatchStartRecommendations | Not implemented |
| CancelMetadataModelConversion | Not implemented |
| CancelMetadataModelCreation | Not implemented |
| CancelReplicationTaskAssessmentRun | Not implemented |
| CreateDataMigration | Not implemented |
| CreateDataProvider | Not implemented |
| CreateEndpoint | Implemented |
| CreateEventSubscription | Not implemented |
| CreateFleetAdvisorCollector | Not implemented |
| CreateInstanceProfile | Not implemented |
| CreateMigrationProject | Not implemented |
| CreateReplicationConfig | Implemented |
| CreateReplicationInstance | Implemented |
| CreateReplicationSubnetGroup | Implemented |
| CreateReplicationTask | Implemented |
| DeleteCertificate | Not implemented |
| DeleteConnection | Implemented |
| DeleteDataMigration | Not implemented |
| DeleteDataProvider | Not implemented |
| DeleteEndpoint | Implemented |
| DeleteEventSubscription | Not implemented |
| DeleteFleetAdvisorCollector | Not implemented |
| DeleteFleetAdvisorDatabases | Not implemented |
| DeleteInstanceProfile | Not implemented |
| DeleteMigrationProject | Not implemented |
| DeleteReplicationConfig | Implemented |
| DeleteReplicationInstance | Implemented |
| DeleteReplicationSubnetGroup | Implemented |
| DeleteReplicationTask | Implemented |
| DeleteReplicationTaskAssessmentRun | Not implemented |
| DescribeAccountAttributes | Not implemented |
| DescribeApplicableIndividualAssessments | Not implemented |
| DescribeCertificates | Not implemented |
| DescribeConnections | Implemented |
| DescribeConversionConfiguration | Not implemented |
| DescribeDataMigrations | Not implemented |
| DescribeDataProviders | Not implemented |
| DescribeEndpointSettings | Implemented |
| DescribeEndpointTypes | Implemented |
| DescribeEndpoints | Implemented |
| DescribeEngineVersions | Not implemented |
| DescribeEventCategories | Not implemented |
| DescribeEventSubscriptions | Not implemented |
| DescribeEvents | Not implemented |
| DescribeExtensionPackAssociations | Not implemented |
| DescribeFleetAdvisorCollectors | Not implemented |
| DescribeFleetAdvisorDatabases | Not implemented |
| DescribeFleetAdvisorLsaAnalysis | Not implemented |
| DescribeFleetAdvisorSchemaObjectSummary | Not implemented |
| DescribeFleetAdvisorSchemas | Not implemented |
| DescribeInstanceProfiles | Not implemented |
| DescribeMetadataModel | Not implemented |
| DescribeMetadataModelAssessments | Not implemented |
| DescribeMetadataModelChildren | Not implemented |
| DescribeMetadataModelConversions | Not implemented |
| DescribeMetadataModelCreations | Not implemented |
| DescribeMetadataModelExportsAsScript | Not implemented |
| DescribeMetadataModelExportsToTarget | Not implemented |
| DescribeMetadataModelImports | Not implemented |
| DescribeMigrationProjects | Not implemented |
| DescribeOrderableReplicationInstances | Not implemented |
| DescribePendingMaintenanceActions | Not implemented |
| DescribeRecommendationLimitations | Not implemented |
| DescribeRecommendations | Not implemented |
| DescribeRefreshSchemasStatus | Not implemented |
| DescribeReplicationConfigs | Implemented |
| DescribeReplicationInstanceTaskLogs | Not implemented |
| DescribeReplicationInstances | Implemented |
| DescribeReplicationSubnetGroups | Implemented |
| DescribeReplicationTableStatistics | Implemented |
| DescribeReplicationTaskAssessmentResults | Not implemented |
| DescribeReplicationTaskAssessmentRuns | Not implemented |
| DescribeReplicationTaskIndividualAssessments | Not implemented |
| DescribeReplicationTasks | Implemented |
| DescribeReplications | Implemented |
| DescribeSchemas | Not implemented |
| DescribeTableStatistics | Implemented |
| ExportMetadataModelAssessment | Not implemented |
| GetTargetSelectionRules | Not implemented |
| ImportCertificate | Not implemented |
| ListTagsForResource | Implemented |
| ModifyConversionConfiguration | Not implemented |
| ModifyDataMigration | Not implemented |
| ModifyDataProvider | Not implemented |
| ModifyEndpoint | Implemented |
| ModifyEventSubscription | Not implemented |
| ModifyInstanceProfile | Not implemented |
| ModifyMigrationProject | Not implemented |
| ModifyReplicationConfig | Implemented |
| ModifyReplicationInstance | Not implemented |
| ModifyReplicationSubnetGroup | Implemented |
| ModifyReplicationTask | Not implemented |
| MoveReplicationTask | Not implemented |
| RebootReplicationInstance | Not implemented |
| RefreshSchemas | Not implemented |
| ReloadReplicationTables | Not implemented |
| ReloadTables | Not implemented |
| RemoveTagsFromResource | Implemented |
| RunFleetAdvisorLsaAnalysis | Not implemented |
| StartDataMigration | Not implemented |
| StartExtensionPackAssociation | Not implemented |
| StartMetadataModelAssessment | Not implemented |
| StartMetadataModelConversion | Not implemented |
| StartMetadataModelCreation | Not implemented |
| StartMetadataModelExportAsScript | Not implemented |
| StartMetadataModelExportToTarget | Not implemented |
| StartMetadataModelImport | Not implemented |
| StartRecommendations | Not implemented |
| StartReplication | Implemented |
| StartReplicationTask | Implemented |
| StartReplicationTaskAssessment | Not implemented |
| StartReplicationTaskAssessmentRun | Not implemented |
| StopDataMigration | Not implemented |
| StopReplication | Implemented |
| StopReplicationTask | Implemented |
| TestConnection | Implemented |
| UpdateSubscriptionsToEventBridge | Not implemented |