AWS RDS Migrations (Part 3)
Part 3: Configure AWS RDS Target Account Infrastructure
Resource Setup in the Target Account (DMS Account)
In part 2, we configured our resources for the RDS source account. Now we will set up the necessary infrastructure in the target account where the RDS instance will be migrated to. This will include setting up the DMS replication instance, target RDS instance, and the required security groups and subnet groups.
1. Database Migration Service (DMS) Resources
These components form the core of the data migration capability:
DMS Replication Instance (AWS::DMS::ReplicationInstance): A compute engine instance (in this use case, a dms.t3.medium instance, not publicly accessible) that performs the actual data migration. It resides in the private subnets specified in the input parameters.
DMS Replication Subnet Group (AWS::DMS::ReplicationSubnetGroup): Defines the subnets (SubnetIds parameter) within the VPC where the Replication Instance will be launched.
DmsVpcRole:
Type: AWS::IAM::Role
Properties:
RoleName: dms-vpc-role
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- dms.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonDmsVpcManagementRole
Path: /
DmsReplicationInstance:
Type: AWS::DMS::ReplicationInstance
Properties:
ReplicationInstanceIdentifier: !Sub "${AWS::StackName}-replication-instance"
ReplicationInstanceClass: dms.t3.medium
PubliclyAccessible: false
ReplicationSubnetGroupIdentifier: !Ref DmsReplicationSubnetGroup
VpcSecurityGroupIds:
- !GetAtt DmsSecurityGroup.GroupId
DmsSecurityGroup:
Type: AWS::EC2::SecurityGroup
Condition: CreateDmsSecurityGroup
Properties:
GroupName: !Sub "${AWS::StackName}-Dms-Rep-SG"
GroupDescription: Security Group for the Dms Replication Instance
VpcId: !Ref VpcId
Tags:
- Key: Name
Value: !Sub "${AWS::StackName}-Dms-Rep-SG"
DmsReplicationSubnetGroup:
Type: AWS::DMS::ReplicationSubnetGroup
DependsOn:
- DmsVpcRole
Properties:
ReplicationSubnetGroupIdentifier: !Sub "${AWS::StackName}-subnet-group"
ReplicationSubnetGroupDescription: Subnet group for Dms replication instance.
SubnetIds: !Ref SubnetIds
2. Database Migration Service (DMS) Endpoints
DMS Source Endpoint (AWS::DMS::Endpoint): Configured to connect to the RDS instance in the source Account. It uses a Secrets Manager ARN for credentials and relies on the VPC Interface Endpoint for network connectivity to the cross-account source database.
DMS Target Endpoint (AWS::DMS::Endpoint): Configured to connect to the RDS instance in the target Account, also using a Secrets Manager ARN.
DMS Replication Task (AWS::DMS::ReplicationTask): The actual job that orchestrates the data movement. It is set up for a Full Load and Change Data Capture (CDC) migration type, selecting all tables (”schema-name”: “%”, “table-name”: “%”).
DmsSourceEndpoint:
Type: AWS::DMS::Endpoint
Properties:
EndpointIdentifier: !Sub "${AWS::StackName}-source-endpoint"
EndpointType: source
EngineName: mysql
MySqlSettings:
SecretsManagerAccessRoleArn: !GetAtt SecretsManagerRole.Arn
SecretsManagerSecretId: !Ref SourceSecretArn
SslMode: none
Tags:
- Key: Name
Value: !Sub "${AWS::StackName}-source-endpoint"
DmsTargetEndpoint:
Type: AWS::DMS::Endpoint
Properties:
EndpointIdentifier: !Sub "${AWS::StackName}-target-endpoint"
EndpointType: target
EngineName: mysql
ExtraConnectionAttributes: Initstmt=SET FOREIGN_KEY_CHECKS=0;
MySqlSettings:
SecretsManagerAccessRoleArn: !GetAtt SecretsManagerRole.Arn
SecretsManagerSecretId: !Ref TargetSecretArn
SslMode: none
Tags:
- Key: Name
Value: !Sub "${AWS::StackName}-target-endpoint"
MySQLtoMySQLReplicationTask:
Type: 'AWS::DMS::ReplicationTask'
Properties:
ReplicationTaskIdentifier: rds-migration-task
MigrationType: full-load-and-cdc
ReplicationInstanceArn:
Fn::ImportValue: !Sub "${DmsInfraStackName}-DmsReplicationInstanceArn"
SourceEndpointArn: !Ref DmsSourceEndpoint
TargetEndpointArn: !Ref DmsTargetEndpoint
TableMappings: |
{
"rules": [
{
"rule-type": "selection",
"rule-id": "1",
"rule-name": "IncludeAllOthers",
"object-locator": {
"schema-name": "%",
"table-name": "%"
},
"rule-action": "include",
"filters": []
},
{
"rule-type": "selection",
"rule-id": "4",
"rule-name": "ExcludeMySQL",
"object-locator": {
"schema-name": "mysql",
"table-name": "%"
},
"rule-action": "exclude",
"filters": []
},
{
"rule-type": "selection",
"rule-id": "5",
"rule-name": "ExcludeInfoSchema",
"object-locator": { "schema-name": "information_schema",
"table-name": "%"
},
"rule-action": "exclude",
"filters": []
},
{
"rule-type": "selection",
"rule-id": "6",
"rule-name": "ExcludePerformanceSchema",
"object-locator": {
"schema-name": "performance_schema",
"table-name": "%"
},
"rule-action": "exclude",
"filters": []
},
{
"rule-type": "selection",
"rule-id": "7",
"rule-name": "ExcludeSys",
"object-locator": {
"schema-name": "sys",
"table-name": "%"
},
"rule-action": "exclude",
"filters": []
}
]
}
ReplicationTaskSettings: |
{
"Logging": {
"EnableLogging": true
},
"FullLoadSettings": {
"CreatePkAfterFullLoad": true,
"TargetTablePrepMode": "DO_NOTHING",
"ParallelLoadThreads": 2,
"MaxFullLoadSubtasks": 16,
"CommitRate": 10000,
"StopTaskCachedChangesApplied": true
},
"TargetMetadata": {
"TargetSchema": "",
"SupportLobs": true,
"LobChunkSize": 64,
"BatchApplyEnabled": false
}
}
3. Networking and Security
These resources ensure secure and private cross-account connectivity:
VPC Interface Endpoint (Implicit Connection): While we don’t explicitly define the AWS::EC2::VPCEndpoint resource, the functionality is enabled by referencing the RdsEndpointServiceName from the source account in the DMS source endpoint configuration.
Example of AWS Source Secret:
{
"username":"dms_user",
"password":"some-password",
"engine":"mysql",
"host":"vpce-012abcd0123456789-xy12zzyx.vpce-svc-0987a6b54b3c219d9.us-east-1.vpce.amazonaws.com",
"port":3306,
"dbname":"foo",
"dbInstanceIdentifier":"foo"
}
The DB host name is the VPC endpoint DNS name. This PrivateLink setup is the solution to the CIDR conflict.
DmsSecurityGroup (AWS::EC2::SecurityGroup) for the DMS Replication Instance.
This security group must be added as an ingress rule on the security group attached to the PrivateLink resource account.
4. IAM Roles and Permissions
Multiple IAM roles are created to grant necessary permissions to DMS:
DmsVpcRole and DmsCloudwatchRole are configured with standard AWS managed policies (AmazonDmsVpcManagementRole, AmazonDmsCloudWatchLogsRole) to allow DMS to manage network interfaces and push logs.
DmsVpcRole:
Type: AWS::IAM::Role
Properties:
RoleName: dms-vpc-role
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- dms.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonDmsVpcManagementRole
Path: /
DmsCloudwatchRole:
Type: AWS::IAM::Role
Properties:
RoleName: dms-cloudwatch-logs-role
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- dms.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonDmsCloudWatchLogsRole
Path: /
DmsSecretsManagerRole is a dedicated role that DMS assumes to access database credentials.
SecretsManagerRole:
Type: AWS::IAM::Role
Properties:
RoleName: dms-secret-manager-access-role
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- !Sub dms.${AWS::Region}.amazonaws.com
Action:
- sts:AssumeRole
Path: /
SecretsManagerAccessPolicyForDatabases is an IAM policy attached to the SecretsManagerRole, granting GetSecretValue and DescribeSecret permissions to the SourceSecretArn and TargetSecretArn.
SecretsManagerAccessPolicyForDatabases:
Type: AWS::IAM::Policy
Properties:
Roles:
- !Ref SecretsManagerRole
PolicyName: SecretManagerAccessForDmsPolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action:
- secretsmanager:GetSecretValue
- secretsmanager:DescribeSecret
Resource:
- !Ref SourceSecretArn
- !Ref TargetSecretArn
Data Migration Service (DMS) Final Thoughts:
The ExtraConnectionAttributes in the Target Endpoint disables foreign key checks during the initial load to avoid constraint violations. The replication task is configured to stop after the initial load so I can re-enable constraints before applying the Change Data Capture (CDC) to keep the target in sync.
The DMS Replication Table Mappings exclude system databases like mysql, information_schema, performance_schema, and sys to avoid unnecessary data migration.
Helpful Resources: