Introducing Cookiecutter Open edX

Note: get the source code for this article at https://github.com/cookiecutter-openedx.

Several years ago but in recent memory Audrey and Daniel Roy Grenfeld published a fantastic book named Two Scoops of Django. Around that same time period, Audrey also created a wildly popular Python based Jinja templating tool named Cookiecutter. Later on Daniel used it to create an equally wildly popular templating tool for jumpstarting Django projects, aptly named Cookiecutter-Django. They’re both fantastic tools that i swear by, and after seeing Cookiecutter-Django for the first time I thought, “we need something like this for Open edX”.

Fast forward 6 years ….

The Cookiecutter Open edX project maintains an integrated suite of open source tools for deploying and managing Open edX at scale on Kubernetes using fully automated Build and Deploy CI workflows. As of this writing, Cookiecutter Open edX is a collection of a dozen git repositories totaling around 50,000 lines of code, plus another 35,000 lines of code in its sibling project openedx-actions on the Github Actions Marketplace. It’s what I use for building new Open edX sites for my clients. It is a combination of Jinja templating tools, code samples, and miscellaneous configuration-related code packages that I need for all of my Open edX projects. Basically, it’s all the stuff that all of my clients request. Thus, Cookiecutter Open edX is a pretty good statistical representation of what a typical project looks like.

So in the most general terms, Cookiecutter Open edX is 85,000 lines of devops, CI and scaffolding code that you definitely need, and that you definitely should try to avoid writing yourself. While I am the author and principal maintainer of both Cookiecutter Open edX as well as openedx-actions, you should know that credit for the genesis of the project really belongs to Miguel Afonso who wrote much of the original source code, and also to Pat McGuire and the UK Cabinet Office who facilitated the eventual open sourcing of the original source code.

For around $0.40 per hour you can create an entire Open edX cloud infrastructure platform and then build and deploy Open edX software onto it. It takes around 20 minutes to completely destroy.

Nearly all Open edX platforms share a common set of infrastructure needs, most of which are hard to do well, can be hazardous when not done well, are knowledge intensive, and more to the point, which provide negligible differentiation. Metaphorically, its like if your boss wants to meet you somewhere and assumes that you’ll simply take an Uber, whereas in reality you first have to design and build a car.

Ironically, transitioning to containerized workloads amplifies the complexity of build-deploy processes in Open edX because it adds several new skills-intensive technologies to a not-insignificant basket of know-how that’s already required for working with a large Python-Django codebase that also includes ReactJS and a bonkers amalgamation of other technologies.

Keeping in mind that typical project sponsors’ areas of focus are generally narrowly centered on things like configuration, site appearance, site marketing, e-commerce, systems integration, and legalese, it’s easy to imagine just how unsettling it is for them if you disappear for days on end to trouble shoot technical infrastructure problems. Worse still would be their reaction when, upon resurfacing, you spout endless Internet cloud technobabble while attempting to bring them up to speed on your activities of late. The apocalypse scenario would be you inadvertently exposing your Kubernetes compute plane to the open Internet, whereupon it is promptly conscripted by black hats who repurpose your infrastructure for a crypto mining operation. Yes, that is a possibility.

The fact is, it’s hard to personally guard against every hazard, and it’s easy to get lost in the weeds while trouble shooting anything that is new to you. It makes it all the more challenging to align your focus with that of your project sponsors, and this in turn puts you at risk of appearing to be over your head, or worse still, poorly focused in their eyes.

This is the scenario that Cookiecutter Open edX aspires to help you avoid. The cookiecutter project helps you to remain focused on the blue boxes in the diagram below — you know, the stuff that your project sponsors actually care about — by providing you with templated solutions for everything else.

My most recent articles, posted nearly a year ago, centered on “newish” tools for deploying and managing open edx installations, with Terraform and Kubernetes taking center stage. But lets backup for a moment. Several years prior, an epiphany moment came in 2019 at the annual Open edX conference at UC San Diego where Tutor was officially introduced. Tutor, another excellent example of a Python-Jinja templating tool, offers an easy on-ramp for using Docker to build and deploy Open edX software. Somewhat related, for several years prior to Tutor I’d been emulating a Docker-based deployment methodology by creating and deploying AWS AMI binaries, but I’ll be the first to recognize that this approach only leveraged a fractional set of the powerful Docker feature set. Tutor’s approach, which combines Python and Jinja to automate the heavy lifting of creating a working Dockerfile, is objectively superior, and it was a huge eye opener for me that led to a paradigm shift in my thought process.

Equally important, Tutor implements an extensibility architecture that encourages the development of community-supported plugins, each of which are intended to package the set of know-how required to accomplish some arbitrary but non-trivial configuration chore in Open edX software. hastexo/tutor-contrib-s3 is a great example, providing a much simplified way to configure your platform to offload file storage to AWS S3. Useful new plugins are regularly introduced to the Open edX community, and so that it doesn’t go unnoticed, even the Cookiecutter Open edX project has its own Tutor plugin, cookiecutter-openedx/tutor-contrib-k8s-deploy-tasks

Since it’s 2019 launch, Tutor has become the official Open edX installation and configuration tool, completely replacing Ansible beginning with the 2021 release of Maple. The advent of Tutor threw me into the fast lane for learning not only Docker but also container orchestration with Kubernetes, which in turn became a segway into learning more about sophisticated CI tools for building and deploying Open edX software.

Another epiphany moment came a few years later in 2022 while advising on an Open edX project for the UK government that runs on Kubernetes and leverages GitHub Actions to automate its Build and Deployment workflows. This was my first up-close opportunity to see both Kubernetes as well CI automation tools in use for an Open edX installation. Following the completion of this project the UK government generously open sourced all of these devops tools.

And that in a nutshell, is the origin story of Cookiecutter Open edX