Migrate from self-managed Redis to Amazon ElastiCache

If you need to create a source database for this migration walkthrough, navigate to the Amazon EC2 console. Choose Launch instance to start the Amazon EC2 instance creation wizard.

The first step in the Amazon EC2 instance creation wizard is to choose your Amazon Machine Image (AMI). Use the Amazon Linux 2 AMI with the default x86 architecture and choose Select.

On the next page, choose the instance type for your Amazon EC2 instance. For this walkthrough, you can use the default t2.micro instance type, which falls in the AWS Free Tier. Choose Review and Launch to continue.

The next page shows the default options for the rest of your Amazon EC2 settings. The default options should be fine for this use case. Choose Launch to continue.

After you choose Launch, choose a key pair to allow SSH access to your new Amazon EC2 instance. From the dropdown, choose Create a new key pair to create a new key pair for this walkthrough. Then name your key pair elasticache-migration and choose Download Key Pair to download the key pair to your computer. Finally, choose Launch Instances to create your instance.

You should see a page confirming that your instance is launching. Choose View Instances to see your Amazon EC2 instance.

As your instance is initialized, it shows an Instance State of pending Wait until the Instance State shows running.

When the Instance State shows running, you can SSH into your instance.

Copy the IPv4 Public IP value for your instance, and then run the following commands in your terminal to SSH into your instance.

chmod 600 /path/to/elasticache-migration.pem
ssh -i /path/to/elasticache-migration.pem ec2-user@<IPv4PublicIP>

Be sure to use the proper values for the path to your elasticache-migration.pem file that you downloaded and use the correct IPv4 Public IP for your instance.

If you have difficulty connecting to your Amazon EC2 instance, see Connecting to your Linux instance using SSH.

If you connected successfully, your terminal should look like the following.

To install Redis, execute the following commands in your terminal.

sudo yum -y install gcc make
wget http://download.redis.io/releases/redis-5.0.6.tar.gz
tar xvzf redis-5.0.6.tar.gz
cd redis-5.0.6
make
sudo make install

Installing Redis takes a few minutes because the system downloads the Redis source and compiles Redis for your machine.

After Redis has been compiled, start your Redis cluster by running the following command in your terminal.

redis-server --daemonize yes --protected-mode no

Note: This is running with protected mode disabled only because it is for a demonstration. In a production use case, you should not disable protected mode.

You should see output indicating that the Redis cluster is starting. You can test your Redis cluster by running a ping command in your terminal.

redis-cli PING

This command should return a response of PONG.

If you get the PONG response, you have started your Redis cluster on your Amazon EC2 instance.

Finally, load some sample data into your Redis cluster to test the migration. In your terminal, run the following command to insert 10 keys into Redis.

redis-cli DEBUG POPULATE 10 test 10000

You should see a response of OK in your terminal. This loads 10 keys into your Redis cluster. Each key has a prefix of test.

To get started, navigate to the ElastiCache console. On the Redis page, choose Create to create a new cluster.

In the ElastiCache cluster creation wizard, choose Redis as your Cluster engine. Do not enable Cluster Mode because online migration does not work with Cluster Mode.

Give your Redis cluster a name and description, and choose the instance type for your cluster nodes.

In Advanced Redis settings, create a subnet group for your Redis cluster. A subnet group refers to a set of network subnets in your Amazon Virtual Private Cloud (Amazon VPC) where your Redis cluster is placed.

In the Security section, use the same security group that you used for your source Redis cluster on Amazon EC2. Do not enable encryption at rest or encryption in transit because these are not supported during online migrations.

The other default settings work for this migration. Scroll to the bottom and choose Create.

After you choose Create, you should see your cluster in the ElastiCache dashboard. It shows a Status of creating while it is initializing.

While you are waiting for your ElastiCache cluster, navigate to the Security Groups section of the Amazon EC2 console. Find the security group that you used for your Amazon EC2 instance and ElastiCache cluster, and choose it.

When looking at your security group details, choose Edit inbound rules.

Add an additional inbound rule to your security group that allows TCP traffic on port 6379 for all instances in your security group. This allows your ElastiCache cluster to access your self-managed Amazon EC2 instance.

Your screen should look like the following.

After you have added the rule, choose Save rules to confirm your changes.

Navigate back to the ElastiCache console. You should see the Redis cluster you created. When this cluster has a Status of available, you are ready to move on to the next module.

Before you start your migration, you need the private IP address of your self-managed Redis cluster on Amazon EC2. Navigate to the Amazon EC2 console and find the instance that is hosting your Redis cluster. Copy the Private IPs value and save it for the next step.

Navigate to the ElastiCache console to begin your online migration. Choose your ElastiCache cluster, and then choose Migrate Data from Endpoint in the Actions dropdown.

In the configuration window that is displayed, enter the private IP address of your Amazon EC2 instance as the Source Redis Endpoint. Then choose Start Migration.

If you prefer to use the AWS Command Line Interface (AWS CLI) to initiate an online migration, you can use the start-migration command.

aws elasticache start-migration \
    --replication-group-id <replicationGroupId> \
    --customer-node-endpoint-list ‘[
            {
                        “Address”: “<privateIPOfEC2Instance>”,
                        “Port”: 6379
            }
    ]’

Be sure to replace <replicationGroupId> and <privateIPOfEC2Instance> with the values for your configuration.

As your Redis cluster starts the migration, it shows a Status of modifying to indicate it is being added to your existing cluster.

After your ElastiCache cluster has successfully connected to your existing Redis cluster, you should see a Status of migrating.

There are a few ways you can check the status of your ElastiCache online migration.

First, you can look at the metrics for your primary node. Choose your ElastiCache cluster, which should bring up the nodes for your cluster. Choose the primary node to see the Amazon CloudWatch metrics for the node in the ElastiCache dashboard.

Scroll to the Master Link Health Status chart. When the value for this chart is at 1, your ElastiCache cluster is caught up with replicating items from your self-managed Redis cluster.

A second way you can check the status of your ElastiCache online migration is to connect to your ElastiCache cluster itself.

Find the primary endpoint of your ElastiCache cluster in the ElastiCache console. Copy the hostname but do not copy the port number at the end.

If you are still connected to your self-managed Amazon EC2 instance via SSH, you can use that instance to connect to your ElastiCache cluster.

Type the following into your terminal while connected to that cluster.

redis-cli -h <yourPrimaryEndpoint>

Be sure to replace <yourPrimaryEndpoint> with the hostname of your Redis cluster from the previous step. This connects you to your ElastiCache Redis cluster.

Then execute the following command while connected to the Redis cluster.

KEYS *

You should see the 10 test keys you inserted into your self-managed Redis cluster.

The following is an image that shows the terminal output after you connect to your Redis cluster and execute the KEYS command.

If the online migration is not copying the keys to your fully managed ElastiCache cluster, you can debug the process by choosing Events in the navigation pane.

The Event page shows you information about your migration, including when it started, when it completed, and if it failed. If your migration failed, this page includes information about the failure.

You can find the Primary Endpoint for your ElastiCache cluster in the ElastiCache console.

When you have switched the configuration to your new ElastiCache cluster and are confident in the results, you can stop the migration from your existing Redis cluster.

To do that, choose your ElastiCache cluster in the console, and then choose Stop Data Migration in the Actions dropdown.

This removes your ElastiCache node as a replica from your self-managed Redis cluster.

When your ElastiCache node has been removed as a replica, it shows a Status of available.

Next, you can delete your Amazon EC2 instance that is hosting your self-managed Redis cluster. You should do this only when you are comfortable that the migration has been successful and you no longer need your existing instance.

Navigate to the Amazon EC2 console. Choose your Amazon EC2 instance, and then choose Instance State > Terminate from the Actions dropdown.

If you were using the ElastiCache cluster only for this walkthrough, you can terminate that also. Navigate to the ElastiCache console. Choose your ElastiCache cluster, and then choose Delete in the Actions dropdown.

After a few minutes, your ElastiCache cluster should be deleted.