Continuous Integration (CI) With Tutor Open edX – Part II

This is part II of a two-part series on implementing CI/CD processes with Tutor Open edX. In part I of this series we learned how to automate the build process. In this article we’ll learn how to deploy the image that we built in part I. first we’ll look at a working, fully-automated Open edX deployment script using Github Actions, and then we’ll discuss how you can customize this work flow to suit your needs. we’ll be using this fully-functional Github Actions Deployment workflow that comes from the same repository as the workflow from part I.

Tutor provides two distinct means of modifying the default configuration of your Open edX instance. First, it gives you a way to modify any of the hundreds of Open edX application parameters found in the edx-platform environment configuration files such as edx-platform/lms/envs/common.py and production.py. Just follow these well-written instructions on how to use the Tutor command line to configure your Open edX platform. Additionally, it gives you a way to create your own custom Docker image containing for example, additional Xblocks, a custom theme, an Open edX plugin, or you can even choose your own fork of the edx-platform source code repository. Creating a custom Docker image is easier than it might seem, and the procedure is well-documented here. In this article we’ll look at some common use cases of both of these for customizing your Open edX platform configuration, and, we’ll leverage Github Actions to fully automate and properly document our steps.

Importantly, you’ll also see in this example that Tutor is a sophisticated deployment tool that provides out-of-the-box support for Docker and for Kubernetes, which is an amazing and under-hyped capability. For context, Tutor does not simply deploy a single Open edX container. In fact, it splits the Open edX application suite into separate pods for lms, cms, one of more worker threads for each application, e-commerce, Discovery service, and so on. And, you can then individually administer and optimize the behavior and performance of these pods in your production cloud infrastructure environment. Furthermore, Tutor configures and deploys all of the back end services such as MySQL, MongoDB, Redis, and Nginx (or Caddy). All told it potentially deploys upwards of a dozen different kinds of containers, either into Docker installed on an Ubuntu instance or into a Kubernetes cluster.

Now then, before we dive into our deployment, I want to digress for a moment on why we’re using Github Actions for this exercise. GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline. Github Actions can be triggered to run automatically upon, for example, every pull request to your repository. I became a fan of Github Actions about 18 months ago while working as part of a team on a large installation. It speeds up and simplifies the development pipeline for all of the team members by automating tasks such as kicking off unit tests each time code is pushed to a repository. It’s coded in yaml format and is very easy to learn and to read. It’s stored inside of your repository, right alongside your code and configuration data. It provides consistency in the build and deployment pipelines, especially when there are many steps to your build, like in the example we’re going to review below. It provides granular role-based permissions to your team and your systems user accounts allowing you to harden security around your deployment work flows. It provides a great set of tools for managing passwords and other sensitive data. and finally, it generates logs of each of your deployments which is enormously helpful when you need to trouble shoot something. So, in a few words, it’s valuable technology that you should consider adding to your repertoire.

The example Github Action workflow uses Tutor to deploy a custom Open edX Docker image that was previously uploaded to AWS Elastic Container Registry (ECR). Additionally, our backend has been horizontally scaled and leverages several AWS managed services like AWS Relational Database Service (RDS), AWS Document DB for MongoDB, and AWS Elasticache for Redis, and AWS Simple Email Service (SES) for SMTP, a Tutor plugin named hastexo/tutor-contrib-s3 that offloads file management from the Ubuntu file system to a secure AWS S3 bucket. Lastly, our example Github Action workflow deploys into AWS Elastic Kubernetes Service (EKS) which is also where all of our back end credentials are stored. Incidentally, for this example, all of these backend services were created using fully-automated Terraform modules that are included in Cookiecutter OpenedX Devops.

Keep in mind that all of these backend services are already up and running. During deployment we simply need to configure our Open edX applications to connect to these already-existing remote services rather than to the default “locally” hosted services. Also for the avoidance of any doubt, the practical theory surrounding how to scale Open edX’s backend services is substantially the same regardless of whether you’re running on Docker or using a native build.

We’re going to use this Github Actions workflow to automate the following operations

• setup our workflow environment: create a virtual instance of Ubuntu and then install Tutor, aws-cli, and kubectl
• authentication to the aws cli using a special AWS IAM user account named ci
• authentication to kubectl using credentials that we’ll retrieve with the aws cli
• retrieve connection parameters and account credentials from Kubernetes Secrets for all of the remote backend services, and then format these into valid Tutor Open edX parameters.
• format and merge all of our custom lms.env.json and cms.env.json parameter values
• configure hastexo/tutor-contrib-s3
• set our Open edX custom theme
• deploy our Open edX installation to a Kubernetes cluster

The workflow is pretty well documented, so I’m going to spend the remainder of this article explaining a few of the recurring patterns that you’ll encounter in this code.

The entire workflow is written in yaml using a limited set of commands that you can easily learn from this Getting Started guide. The workflow runs on a Github-hosted virtual server instance. Github gives you 2,000 minutes of server time for free each month which should be more than you need in most cases. The example workflow in this article consumer between 4 and 9 minutes each times it runs, and I usually run the work flow a few dozen times a month at most. The server instances are ephemeral and are destroy immediately upon completion of the workflow. You therefore have to build your entire deployment environment each time the workflow runs.

Per the screen shot below, this workflow runs on “workflow_dispatch” (row 18) which is a lofty way of saying that it runs when you click the “Run” button from the Github Actions console page of your repository on the github.com site.