Scaling Memcached and Redis for Open edX

Summary

If you’re looking for a general overview of how to scale the Open edX platform then you should read this first, “Scaling Open edX“. You might also want to take a look at the automated tools included in Cookiecutter Open edX Devops as well as this article on Managing Your Open edX Backend With Terraform.

The Open edX platform persists data across five distinct subsystems: MySQL, MongoDB, the Ubuntu file system, Memcached, and since the Lilac release in May 2021, Redis. To scale the Open edX platform you first must physically separate the executable program code from the data that is managed by these four subsystems. This article explains how to migrate the Memcached service. By scaling, I mean migrating Open edX’s local Memcached service to its own independent Ubuntu EC2 instance.

This article describes my preferred approach, which is to use AWS EC2 console to create a new Ubuntu EC2 instance using the same version of Ubuntu on which your Open edX platform currently runs. As of this writing Open edX Lilac runs on Ubuntu 20.04 LTS. I install the latest versions of memcached and redis-server, regardless of the versions that the original Open edX server is using. When scaling Memcached and Redis I do not migrate data.

Note that you will only need to focus on vertical scaling for Memcached and Redis on Open edX. You can safely start with a t2.small or t2.medium EC2 instance size which will probably server your needs for the lifetime of your Open edX platform.

1. Launch a new EC2 instance from the AMI

Launch a new t2.medium EC2 instance from the AMI that you created. I re-use the existing SSH key that i used for the original Open edX EC2 instance. Note that you’d only need a different SSH key if for example, completely different teams manage the Open edX and Memcached environments.

2. Create a new EC2 Security Group for Memcached

You should create a separate EC2 Security Group for your new Memcached EC2 instance, as follows:

This firewall configuration limits remote access of the server to Memcached, regardless of whatever other services might still be installed and running internally on the server. Note that on the first row, SSH, you should try to limit access to your bastion server, if you use one.

3. Take Note of The Internal IP Address That is Assigned

You’ll access your new remote Memcached server via the internal IP address, which is automatically assigned by AWS when you create the new EC2 instance. Take note of this value, which will be titled, “Private IPv4 addresses”

1. Install memcached service

# First, make sure that your local package index is updated
sudo apt update

# install the official memcached package 
sudo apt install memcached

# libmemcached-tools is a library that provides several tools to work with your Memcached server.
sudo apt install libmemcached-tools

2. Configure memcached to accept remote connections

Edit the file /etc/memcached.conf on or around row 35 as follows:

# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
#
# change this from from '-l 127.0.0.1' to the following
-l 0.0.0.0

You can read more about the installation procedure here, “How To Install and Secure Memcached on Ubuntu 18.04“