Simplified Database migration from RDS PostgreSQL to YugabyteDB using voyager

"YugabyteDB Voyager”  is a migration tool from Yugabyte that powers you to perform offline/online migration with fall-forward and fall-back options from your on-premise/public cloud to YugabyteDB. 

I have recently tested one of the scenario in my local environment, where source database is RDS PostgreSQL and target database is YugabyteDB which is running on ec2 cluster.  sharing step-by-step procedures which I have followed referring below document : https://docs.yugabyte.com/preview/yugabyte-voyager/migrate/live-fall-back/. 

Step -1. Install yugabytedb voyager on intermediate server which have connectivity from/to source and target databases

Step -2. Prepare Source and Target database for online migration

Step -3. Run Voyager Assessment to validate the configuration

Step -4. Run export schema from source database

Step -5. Analyze the schema

Step -6. Import the schema to target database

Step -7. Export the data from source with snapshot and changes enabled

Step -8. Import the data to target

Step 9. Generate the data migration report to validate the export/import count

Step 10. Cutover to Target

Step 11. Cutover to Source (If required)

Step 12. End the Migration.

Step -1. Please refer to the following document to install voyager. 

https://docs.yugabyte.com/preview/yugabyte-voyager/install-yb-voyager/ 

Step 2.  Source and Target Database Preparation

2.1 Prepare Source Database 

2.1.1 Enable rds.logical_replication parameter in RDS instance, you will need to create new parameter group with rds.logical_replication value as 1, and assign it to database instance from the modify option. 

 

2.1.2 Create Voyager user in source database. 

create user ybvoyager password 'password';

          2.1.3 Grant permission to user. 

Use the yb-voyager-pg-grant-migration-permissions.sql script (in /opt/yb-voyager/guardrails-scripts/ or, for brew, check in $(brew --cellar) / yb-voyager@<voyagerversion>/<voyagerversion>) to grant the required permissions as follows: 

psql -h <host>
-d <database>  \
-U <username> \ 
-v voyager_user='ybvoyager' \     
-v schema_list='<comma_separated_schema_list>' \     
-v is_live_migration=1 \     
-v is_live_migration_fall_back=1 \     
-v replication_group='<replication_group>' \     
-v original_owner_of_tables='<original_owner_of_tables>' \     
-f <path_to_the_script>

 2.2 Prepare Target Database

2.2.1 Add the following flags to the cluster before starting migration, and revert them after the migration is complete.

ysql_pg_conf_csv = yb_max_query_layer_retries=0  

ysql_max_read_restart_attempts = 0

ysql_max_write_restart_attempts = 0

2.2.2 Create target database

create user ybvoyager superuser password 'password';

2.2.3 Create user and grant superuser role

create user ybvoyager password 'password';

grant yb_superuser to ybvoyager;

2.3 Create Directory for export on Voyager server

Step -3. Run Voyager Assessment :

yb-voyager assess-migration --export-dir /home/ubuntu/export-dir \
    --source-db-type postgresql \
    --source-db-host postgresqlcluster1.c9wjgnaumka4.ap-south-1.rds.amazonaws.com  \
    --source-db-user ybvoyager \
    --source-db-password QQvoyager##123  \
    --source-db-name pgdb \
    --source-db-schema rdsschemapgdb

Step 4: Export Schema from Source

yb-voyager export schema --export-dir /home/ubuntu/export-dir/ \
        --source-db-type postgresql \
        --source-db-host postgresqlcluster1.c9xxxxxxxxxxxa4.ap-south-1.rds.amazonaws.com  \
        --source-db-user ybvoyager \
        --source-db-password QQutfyuligh##123 \
        --source-db-name pgdb \
        --source-db-schema rdsschemapgdb 

Step 5: Run analyze schema 

yb-voyager analyze-schema --export-dir /home/ubuntu/export-dir/   --output-format html

Step 6: Import the Schema into Target YugabyteDB

yb-voyager import schema --export-dir /home/ubuntu/export-dir \
    --target-db-host   10.98.41.199 \
    --target-db-user ybvoyager \
    --target-db-password  QQvoiygukhlger##123  \
    --target-db-name  ybdb

Step 7: Export Data from Source RDS PostgreSQL database

yb-voyager export data from source  --export-dir /home/ubuntu/export-dir \
    --source-db-type postgresql \
    --source-db-host postgresqlcluster1.c9xcvbnmxcvbka4.ap-south-1.rds.amazonaws.com  \
    --source-db-user ybvoyager \
    --source-db-password QQvoyagyvjhbkjlnr##123  \
    --source-db-name pgdb \
    --source-db-schema rdsschemapgdb \
    --export-type snapshot-and-changes

You can repeat export in regular interval until cutover , if you terminate it and start again it will start from the same point from where it was terminated. 

Step 8 : Import Data to Target YugabyetDB

yb-voyager import data --export-dir /home/ubuntu/export-dir \
    --target-db-host   10.98.41.199 \
    --target-db-user ybvoyager \
    --target-db-password  QQvoyager##123  \
    --target-db-name  ybdb \
    --parallel-jobs 2

You can keep importing data in regular interval into target until cutover , if you terminate it and start again it will start from the same point from where it was terminated.

Step 9: Generate data migration report , to validate export/import count.

yb-voyager get data-migration-report --export-dir /home/ubuntu/export-dir \
        --target-db-password QQvoyager##123 \
        --source-db-password QQvoyager##123

Step 10. Cutover to Target 

Before performing cutover to Target, quiesce your source database to stop all application writes and validate counts from source and target tables. 

One validated, you can start cutover as below :

yb-voyager initiate cutover to target --export-dir /home/ubuntu/export-dir --prepare-for-fall-back true

You can check cutover status as show below, In my case I have only initiated cutover to target and connect my application to target. 

Step 11. Cutover to Source (If fallback is required) 

In case of any issue at target I can fallback to my source by perforning cutover to Source. 

Before that quiesce your target database and stop all the writes.

yb-voyager initiate cutover to source --expor-dir /home/ubuntu/export-dir

Step 12. End the Migration 

Once Migration is successful, please end the migration to make it complete. 

yb-voyager end migration --export-dir <EXPORT_DIR> \
--backup-log-files <true, false, yes, no, 1, 0> \         
--backup-data-files <true, false, yes, no, 1, 0> \         
--backup-schema-files <true, false, yes, no, 1, 0> \         
--save-migration-reports <true, false, yes, no, 1, 0> \  # Set optional argument to store a back up of any of the above arguments.        
--backup-dir <BACKUP_DIR>