Managing Your Open edX Backend With Terraform

Note: get the source code for this article at https://github.com/lpm0073/cookiecutter-openedx-devops. Follow the instructions in the README.

In all fairness, that question is prone to coming up for any successful, modern web platform that goes through a growth spurt. In this article we’ll explore how I manage not just one, but several very large Open edX installations. Here are what I consider to be the key success factors:

• Infrastructure as code. I use Terraform, but there are other good alternatives. Terraforms gives me the ability to version control my backend infrastructure service configurations so that I can safely fallback when I make a mistakes, and it gives me complete automation of the entire life cycle of each service which saves me lots of time.
• Dedicated VPC. I use a dedicated VPC for each Open edX installation, which helps to optimize the network for each installation as well as to keep systems from bleeding into each other, and, it also helps with tear-downs.
• Managed Services. All of my Open edX platforms run on AWS, and I’m biased towards using their managed services such as RDS for MySQL, DocumentDB for MongoDB, EKS for Kubernetes, and Elasticache for Redis. This dramatically reduces the number of failure points for which you are directly responsible.
• Kubernetes. Paradoxically, adding Kubernetes simplifies most aspects of system management
• Simple security policies. We’ll talk more below about firewall settings, user accounts, admin accounts, and exposing your backend services to the outside world.

Earlier this year I open-sourced my personal Terraform and Github Actions scripts in the form of a Cookiecutter template repository named Cookiecutter Openedx Devops. You can use this Cookiecutter to create your own Open edX devops repository, perfectly configured with your custom domain name and AWS account information. Cookiecutter Open edX Devops is a highly opinionated set of tools for creating and maintaining an AWS backend for Open edX that satisfies all five of these principals.

Cookiecutter Open edX Devops leverages Terraform and Github Actions to provide 1-click backend solutions incorporating the current best practices for each service with regard to feature set, configuration, maintainability and security. This is mostly achieved by restricting the Terraform modules that it leverages to those supported by Hashicorp directly and of Terraform AWS modules, which is a community of AWS service users spanning dozens of large organizations and thousands of individual contributors. For each backend service it:

• creates and configures the service
• stores admin account credentials in Kubernetes Secrets
• creates security groups, IAM policies and anything else that necessary for the service to work correctly with the Open edX applications
• creates Route53 DNS subdomain records
• reconfigures the Open edX applications to use the new remote service

• Kubernetes. Uses AWS Elastic Kubernetes Service to implement a Kubernetes cluster onto which all applications and scheduled jobs are deployed as pods. Tutor natively deploys Open edX applications as individual containers for LMS, CMS, Workers, Forum, etcetera. All backend service admin account credentials are automatically stored in Kubernetes Secrets. The Kubernetes configuration itself is intentionally as simple as possible. Simple is good.
• MySQL. uses AWS RDS for all MySQL data, accessible inside the VPC as mysql.yourdomain.edu:3306. Instance size settings are located in the environment configuration file, and other common configuration settings are located here. Passwords are stored in Kubernetes Secrets accessible from the EKS cluster.
• MongoDB. uses AWS DocumentDB for all MongoDB data, accessible inside the VPC as mongodb.master.yourdomain.edu:27017 and mongodb.reader.yourdomain.edu. Instance size settings are located in the environment configuration file, and other common configuration settings are located here. Passwords are stored in Kubernetes Secrets accessible from the EKS cluster.
• Redis. uses AWS ElastiCache for all Django application caches, accessible inside the VPC as cache.yourdomain.edu. Instance size settings are located in the environment configuration file. This is necessary in order to make the Open edX application layer completely ephemeral. Most importantly, user’s login session tokens are persisted in Redis and so these need to be accessible to all app containers from a single Redis cache. Common configuration settings are located here. Passwords are stored in Kubernetes Secrets accessible from the EKS cluster.
• Container Registry. uses this automated Github Actions workflow to build your tutor Open edX container and then register it in Amazon Elastic Container Registry (Amazon ECR). Uses this automated Github Actions workflow to deploy your container to AWS Amazon Elastic Kubernetes Service (EKS). EKS worker instance size settings are located in the environment configuration file. Note that tutor provides out-of-the-box support for Kubernetes. Terraform leverages Elastic Kubernetes Service to create a Kubernetes cluster onto which all services are deployed. Common configuration settings are located here.
• User Data. uses AWS S3 for storage of user data. This installation makes use of a Tutor plugin to offload object storage from the Ubuntu file system to AWS S3. It creates a public read-only bucket with write access provided to edxapp so that app-generated static content like user profile images, xblock-generated file content, application badges, e-commerce pdf receipts, instructor grades downloads and so on will be saved to this bucket. This is not only a necessary step for making your application layer ephemeral but it also facilitates the implementation of a CDN (which Terraform implements for you). Terraform additionally implements a completely separate, more secure S3 bucket for archiving your daily data backups of MySQL and MongoDB. Common configuration settings are located here.
• CDN. uses AWS Cloudfront as a CDN, publicly accessible as https://cdn.yourdomain.edu. Terraform creates Cloudfront distributions for each of your environments. These are linked to the respective public-facing S3 Bucket for each environment, and the requisite SSL/TLS ACM-issued certificate is linked. Terraform also automatically creates all Route53 DNS records of form cdn.yourdomain.edu. Common configuration settings are located here.
• Password & Secrets Management uses Kubernetes Secrets in the EKS cluster. Open edX software relies on many passwords and keys, collectively referred to in this documentation simply as, “secrets“. For all back services, including all Open edX applications, system account and root passwords are randomly and strongly generated during automated deployment and then archived in EKS’ secrets repository. This methodology facilitates routine updates to all of your passwords and other secrets, which is good practice these days. Common configuration settings are located here.
• SSL Certs. Uses