AWS RDS Migrations (Part 1)
Part 1: Assess Migration Strategies
Executing a database migration can be tricky and time-consuming, especially when dealing with large datasets, complex schemas, or network configurations. However, with the right approach and tools, you can streamline the process and minimize downtime. In this blog post, we’ll explore some best practices for executing database migrations using AWS RDS (Relational Database Service).
Our use case involves migrating a production database from one RDS instance to another with minimal downtime. The migration will take place across different AWS accounts and to make things even more difficult, the source and destination accounts have a CIDR overlap. Prior to building out the infrastructure, we need to assess which migration strategy is best suited for our use case.
Part 1: Assess Migration Strategies
There are several strategies for migrating databases in AWS RDS, including:
Snapshot and Restore: This involves taking a snapshot of the source database and restoring it to the target RDS instance. This method is straightforward but may result in significant downtime, especially for large databases.
Database Migration Service (DMS): AWS DMS allows for continuous data replication with minimal downtime. It supports homogeneous and heterogeneous migrations and can handle schema conversions.
A snapshot and restore method typically involves the following:
Create a Snapshot and enable Sharing
Create a manual snapshot: In the source AWS account, take a manual snapshot of your RDS instance. This is a point-in-time backup of your database.
Share the snapshot: Once the snapshot is complete, share it with the target AWS account using its account ID.
Encrypted snapshots: If your RDS instance is encrypted with a custom KMS key, you must also share the KMS key with the target account and grant it permission to use the key. If the snapshot is unencrypted, this step is not needed.
Restoration in Target Account
Copy the shared snapshot: Log in to the target AWS account. You’ll see the snapshot under the “Shared with Me” tab in the RDS console. You must copy the snapshot within the target account before you can restore it. This is a critical step, especially if the source snapshot is encrypted.
Restore the instance: From the copied snapshot in the target account, restore a new RDS instance. During this process, you will configure the new instance with the desired settings, including the new VPC, subnets, security groups, and parameter groups.
Post-Migration and Cleanup
Update application endpoints: After the new RDS instance is available, you’ll get a new endpoint. Update your application’s connection string to point to this new endpoint.
Testing and verification: Thoroughly test your application to ensure it’s functioning correctly and that all data has been migrated successfully and consistently.
Decommission the old instance: Once you’ve confirmed the migration is successful and the new instance is stable, you can decommission the original RDS instance in the source account to avoid unnecessary costs. This should include taking a final snapshot and then deleting the instance.
Clean up snapshots: Delete the shared snapshot in the source account and the copied snapshot in the target account if they are no longer needed.
Minimizing Downtime
Schedule a maintenance window: The application can’t write to the database while the snapshot is being created and restored. Perform the migration during a time of low traffic, like late at night or on a weekend.
AWS Database Migration Service (DMS) is a managed service that allows us to perform a one-time migration and then continuously replicate ongoing changes, allowing for a seamless cutover.
Prepare the Source Database Account
Create a Network Load Balancer (NLB): Place a Network Load Balancer (NLB) in the source account’s VPC. The NLB will be the entry point for traffic from the target account.
Create a Lambda Function: A function to register the IP addresses from the RDS endpoint in the NLB target group. This is required because the NLB must use the RDS IP address. Since IP addresses can change, we need to resolve the DNS endpoint for the RDS instance and update the NLB target group if the address has changed.
Create a VPC Endpoint Service: This service uses the NLB as its entry point.
Prepare the Target Account
Create a VPC Endpoint: In the target account, you create a VPC endpoint that connects to the VPC endpoint service you created in the source account.
Create the DMS Replication Instance and Endpoints: You then create your DMS replication instance and both a source and target endpoint.
Execution and Cutover
Configure the replication task, perform a full load, enable CDC, and finally, perform the cutover.
This approach is ideal for our use case, as it minimizes the disruption to our application service and handles the CIDR overlap. In Part 2, we will configure the necessary infrastructure for our RDS source account.