Styling Open EdX Tips and Tricks

#######################
Styling in edx-platform
#######################

Over time, our Sass styling has become a little convoluted, with three major
reworkings often leading to confusion when a developer needs to style in
multiple locations across edx-platform. The main endeavors were v1, v2 (or
pattern library) and Bootstrap. We are trying to move away from using v2, as the
pattern library is deprecated, but there may still be locations in the code that
reference those styles (please remove them as you see fit).

Our platform uses a system of Sass partials that combine to compile into a
single large css file to be rendered on the page. From the Sass docs:

You can create partial Sass files that contain little snippets of CSS
that you can include in other Sass files. This is a great way to
modularize your CSS and help keep things easier to maintain. A partial
is simply a Sass file named with a leading underscore. You might name it
something like _partial.scss. The underscore lets Sass know that the
file is only a partial file and that it should not be generated into a
CSS file. Sass partials are used with the “@import“ directive.

This structure allows us to break up our styling into small pieces, making
readability and maintenance easier, while often at the expense of structural
complexity as a code base grows. Here is an example, directly from edx-platform,
for how the partials flow to a single scss file that compiles into CSS for the
page to use. This is a page that uses v1 styling.

.. figure:: v1_sass_pipeline.webp
:alt: Sass Compilation for v1 CSS

Sass Compilation for v1 CSS

Note that this only shows an example, there are far more partials that come
together to make the “lms-main-v1.css“ final file.

As you can see, the “lms-main-v1.scss“ file does not have a leading
underscore, telling the compilation to turn that scss file into an actual css
file to be rendered on the page. If you were to use the Chrome Inspector tool
and look at the styling, you will see that there is one unified “lms-
main-v1.css“ file that contains all the element styles.

The “lms-main-v1.scss“ file uses the “@import“ statement to pull in the
“_build- base-v1.scss“ and “build-lms-v1.scss“ files which in turn import
more partials down the road. It is critical to recognize that this tree works in
a depth-first, first-come-first-serve manner. That means that styles that are
imported early in the process cannot reference Sass variables imported later in
the process (ie: “bootstrap/variables“ cannot reference variables from
“shared/header“).

This diagram describes the process for the **v1 styles**, and there is a similar
setup for “lms-main-v2.scss“ as well as “bootstrap/lms-main.scss“. Each
individual HTML page on the edx-platform specifies which of the three that page
wants to use for styling. Please note that as an organization, we are slowly
trying to 1) move everything over to the bootstrap/lms-main.scss file and 2)
deprecate and stop using any v2 files. “lms-main-v1.scss“ can still be used,
but a migration over to “bootstrap/lms-main.scss“, and a migration of any
relevant partials from the v1 structure, would be a valuable endeavor.

Please note that as you add partials, make sure that they are not already being
imported in another file (ie: you add a node to the tree above that already
exists on the tree).

What theming does, and how to do it
***********************************

By thinking about the styling as a tree, theming becomes a lot simpler. All
theming means is that you can override one of the partials above by matching
the exact path in the “edx-platform/themes“ directory. So, for example, to
override the shared header file (located at “lms/static/sass/shared/_header“),
you would simply go into the “edx-platform/themes/[theme you want to
override]/lms/static/shared“ folder and add a _header.scss file. When django
compiles the assets, it will use this file as a replacement to the main
edx-platform implementation.

If you look at the actual code base, you will see that we have a standard of
using the partials directory for these overrides files. This keeps the specific
components isolated from the core styling to reduce errors when people override
files and forget to import other files that are needed elsewhere on the site.
This is a good practice that future development should adhere to.

Final Note: When dealing with front end changes, it is a good idea to also check
the edx-themes repo, that works exactly like our themes folder, but also
includes html templates that can further confuse things.

Bootstrap and edx-platform
**************************

In a month and a half endeavor in the Fall, Andy and I worked on integrating
Bootstrap into the platform for three main reasons.

1. **To unify our styling:** By specifying colors and variables that can be used
universally, reducing the ’50 shades of grey’ issue and inconsistencies in the
ways that we style components.

2. **To add a widely used component library:** Bootstrap has a robust community
of developers that contribute to their open source component library, allowing
for easier prototyping and create of front end experiences.

3. **To simplify theming:** Since we now have a single unified source of truth
for our variables, openedx instances and alternative themes can simply override
those files to customly style their sites.

Relating to the above styling conversation, we have pulled in the entire
bootstrap styling library into “lms-main.scss“, but only partially pulled it
into the v1 and v2 implementations due to naming conflicts. For example, we
could not pull in the bootstrap modal, due to conflicts with our own modal
styling (both use the generic “.modal“ class). We were, however, able **to
pull in the entire bootstrap javascript file**, since that file overrides jQuery
in ways that our current application does not (meaning no conflicts).

Therefore, any pages that use lms-main.scss can use any bootstrap component out
of the box. **To use bootstrap components in older, v1, v2 pages** we have to
manually specify the exact Sass partial from the “edx-
platform/node_modules/bootstrap“ that we want. For example, to add an alert to
a v1 page, you would add to “base/build“ the partial “bootstrap/scss/alert“.
Steps outlined below.

1. Determine from the “node_modules/bootstrap/scss“ directory which component
you want to import.

2. Add that import (i.e: “bootstrap/scss/alert“) somewhere in the Sass tree,
most likely to “base/build.scss“

3. NOTE: Run “paver compile_sass“ to make sure it works, if not, there is
likely a bootstrap mixin that you are missing. If so, search for the mixin in
“node_modules/bootstrap/scss/mixins“ and import it (i.e:
“bootstrap/scss/mixins/alert“)

4. You are done. Add the component using HTML or Javascript. Use the Bootstrap
Component Docs for examples.

FAQ
***

I want to add a new scss file for a feature on a particular page, how do I add it?
##################################################################################

First, you want to check which root file the page uses, whether it is
lms-main-v1, lms-main-v2 or lms-main. Then, go to that file and trace down the
tree to find a good spot for the new scss file to live. So, for example, if you
are adding an LMS feature and the page uses lms-main-v1, you can trace down lms-
main-v1 > _build_lms_v1 and see that there are plenty of standalone imports that
look like features. Simply add your file to the lms/static/sass directory in a
similar manner and add it as an @import to that page.

Why isn’t my bootstrap component styling like in the bootstrap docs?
####################################################################

In any pages that use the lms-main.css compiled file, the bootstrap component
will render. This case only arises when we are working with a legacy v1 or v2
page. As outlined earlier in this document, we cannot import the entire
bootstrap repository due to conflicts with old styling. For v1 and v2, we are
gradually pulling in styles for components, so this must mean that you are
building a component that has not yet been used in the LMS/Studio.

To add the styles, you first need to find them in the bootstrap package. To do
so, search at edx-platform/node_modules/bootstrap/scss for the file that you
need. Then add this to the v1 or v2 tree, most likely to the build/base.scss
file. Note that you may hit issues with mixins when you try to compile the SASS.
In this case, check the edx-platform/node_modules/bootstrap/scss/mixins folder
and import that into the lms or studio before trying to import the component
styling.

######################
Working with Bootstrap
######################

This topic describes the Bootstrap framework, and how it should be used to
build user interfaces within edX applications. Note that Bootstrap adoption
started after the Ginkgo release, and so this document applies to the edX
master branch and to the forthcoming Hawthorn release.

If you are interested in the rationale for edX choosing Bootstrap, you can
read about the decision in `OEP-16: Adopting Bootstrap
<https://open-edx-proposals.readthedocs.io/en/latest/oep-0016-bp-adopt-bootstrap.html>`_.

.. highlight:: none

***************
Getting Started
***************

Bootstrap is an open source front end component library that is used by many of
the world’s most popular applications. It allows for rapid assembly of front end
components using a responsive grid system, a robust component library and easy
to configure theming capabilities to ensure that new components are rendered
consistently. EdX is using `Bootstrap 4`_ which is a reimplemented version using
Sass and that is currently in beta release.

All edX applications should use the `edx-bootstrap`_ package that can be
installed via `npm`_. This package provides two themes (a default Open edX
theme, as well as an edX branded version), and eventually will provide custom
Open edX styles for common patterns.

.. _ui_bootstrap_custom_designs:

*************************
Developing Custom Designs
*************************

Bootstrap provides a large number of components and layouts out-of-the-box, but
there will always be a need to implement custom designs. There are a number of
considerations to take into account when implementing your designs using Sass.

The most important rule is to avoid hard-coding values such as colors and fonts.
Using hard-coded values means that Bootstrap themes will not be able to affect
your styles, and so your new elements will look out of place. Whenever possible
you should instead use the functions and variables provided by Bootstrap to
access theme colors or fonts. See `Bootstrap customization options`_ for more
details.

For example, here is an example of a hard-coded style::

.my-element {
font-family: “Open Sans”;
color: #0000ff;
}

The recommended alternative is as follows::

.my-element {
font-family: $font-family-sans-serif;
color: theme-color(“primary”);
}

If you do find the need for a custom color or font that isn’t provided by
the edX Bootstrap library, consider first whether it makes sense to contribute
it back so that other applications can use this value too. If you decide to
add a custom value, define it as a variable that can be overridden by a theme by
using the “!default“ flag. This allows themes to provide a different value
for this variable if they choose. See the Sass documentation for `default flag`_
for more details.

For example::

$my-custom-color: #0000ff !default;

.my-element {
font-family: $font-family-sans-serif;
color: $my-custom-color;
}

.. _Bootstrap 4: https://getbootstrap.com/docs/4.0/getting-started/introduction/
.. _Bootstrap customization options: https://getbootstrap.com/docs/4.0/getting-started/options/
.. _default flag: http://sass-lang.com/documentation/file.SASS_REFERENCE.html#Variable_Defaults___default
.. _edx-bootstrap: https://www.npmjs.com/package/@edx/edx-bootstrap
.. _npm: https://www.npmjs.com/

#######################################
edx-platform Static Asset Pipeline Plan
#######################################

Static asset handling in edx-platform has evolved in a messy way over the years.
This has led to a lot of complexity and inconsistencies. This is a proposal for
how we can move forward to a simpler system and more modern toolchain. Note that
this is not a detailed guide for how to write React or Bootstrap code. This is
instead going to talk about conventions for how we arrange, extract, and compile
static assets.

Big Open Questions (TODO)
*************************

This document is a work in progress, as the design for some of this is still in
flux, particularly around extensibility.

* Pluggable third party apps and Webpack packaging.
* Keep the Django i18n mechanism?
* Stance on HTTP/2 and bundling granularity.
* Optimizing theme assets.
* Tests

Requirements
************

Any proposed solution must support:

* Externally developed and installed Django apps.
* Theming.
* XBlock assets.
* Existing tests.
* Fast builds.
* An incremental implementation path.
* Other kinds of pluggability???

Assumptions
***********

Some assumptions/opinions that this proposal is based on:

* We want to shift as much as possible to Webpack and the JavaScript stack of
technologies, leaving the Python layer as thin as possible.
* While we will try to make theming upgrades straightforward, we will be moving
around where files are located and where they’re compiled out to.
* We will be pushing towards a world that is more Django app-centric than LMS
vs. Studio centric, to reduce duplication.
* At the same time, we want to consolidate assets far more efficiently than we
are doing today.
* Leaning towards more static front ends + API calls.
* However we still need to be compatible with Django’s asset system for things
like third party apps (e.g. Django Rest Framework browsing assets, Swagger,
etc.)
* It should be possible to pre-build static assets and deploy them onto S3 or
similar.

Where We Are Today
******************

We have a static asset pipeline that is mostly driven by Django’s built-in
staticfiles finders and the collectstatic process. We use the popular
“django-pipeline“ library, with UglifyJS as the JavaScript compressor (the
binary is installed via node into node_modules). We also use the less well known
“django-pipeline-forgiving“ extension to “django-pipeline“ so we don’t error
out when files are missing (added when we started dynamically scanning XBlocks
for assets).

The “django-pipeline“ config is aware of CSS files for the purposes of
concatenation, but it does *not* know about the source Sass files.
Those are processed with paver tasks before “django-pipeline“ ever sees them.

We also have the following custom extensions to Django’s builtin “STATICFILES“
mechanism:

“openedx.core.djangoapps.theming.finders.ThemeFilesFinder“
Custom finder that overrides any static asset with a version from the themes
directory (“COMPREHENSIVE_THEME_DIRS“ defined in “lms.envs.json“ and
“cms.envs.json“).

“openedx.core.lib.xblock_pipeline.finder.XBlockPipelineFinder“
Custom finder that accesses and extracts assets from pip-installed XBlocks via
“pkg_resources“.

“openedx.core.storage.DevelopmentStorage/ProductionStorage“
Custom “FileStorage“ classes that mostly exist for theme-awareness.

LMS and Studio/CMS Separation
—————————–

LMS and Studio have their own directories for source assets (“lms/static“ and
“cms/static“), and have symlinks to shared assets in “common/static“. We
treat the static asset compilation and collection phase for LMS and Studio as
separate projects that happen to share a lot of pieces. They output to different
places (typically “/edx/var/edxapp/staticfiles“ for LMS and
“/edx/var/edxapp/staticfiles/studio“ for Studio) and can be collected
separately. However in practice they’re always run together because we deploy
them from the same commits and to the same servers.

Django vs. Webpack Conventions
******************************

The Django convention for having an app with bundled assets is to namespace them
locally with the app name so that they get their own directories when they are
gathered together into a common static directory by collectstatic. For example,
the edx-enterprise app has a “static/enterprise“ folder, so its assets are
compiled to “/edx/var/edxapp/staticfiles/enterprise“ by edx-platform and will
not conflict with assets from any other Django app.

Webpack conventions would have us create a single set of configuration files at
the root of edx-platform, which would specify all bundles in the project.

TODO: The big, “pluggable Webpack components” question.

Proposed Repo Structure
***********************

All assets that are in common spaces like “common/static“, “lms/static“,
and “cms/static“ would be moved to be under the Django apps that they are a
part of and follow the Django naming convention (e.g.
“openedx/features/course_bookmarks/static/course_bookmarks“). An app’s
“templates/{appname}“ directory will only be for server side templates, and
any client-side templates will be put in “static/{appname}/templates“.

Proposed Compiled Structure
***************************

This is meant to be a sample of the different types of things we’d have, not a
full list:

::

# Webpack bundles/post-processed assets
/webpack/css
/fonts
/js
/vendor ?

# Django apps that are in the edx-platform repo
/course_bookmarks
/course_experience

# edX authored, installed via separate repo
/enterprise

# Entirely third party apps that we need to maintain compatiblity with.
/admin
/rest_framework

# Themes are part of the “theming” app
/theming/themes/open-edx
/red-theme
/edx.org

# XBlocks still collect their assets into a common space (/xmodule goes away)
# We consider this to be the XBlock Runtime’s app, and it collects static
# assets from installed XBlocks.
/xblock

Django vs. Webpack Roles
************************

Rule of thumb: Django/Python still serves static assets, Webpack processes and
optimizes them.

Webpack would be responsible for all Sass compilation in edx-platform. It would
also be responsible for the optimization/minification of JavaScript assets, but
those optimized assets would only appear under the “/webpack“ directory. Third
party assets that Webpack is not aware of may have hash suffixes applied to them
by the Django collectstatic layer, but will not otherwise be processed or
optimized in any way — so no sass compilation, no uglifyjs minification, etc.

The django-pipeline dependency should be removed altogether.

Themes
——

Theme handling is muddled. The fact that themes can override server-side
templates means that Python has to be aware of them. At the same time, we want
to shift over Sass compilation as a whole to Webpack, meaning that at least some
knowledge about where they are and how to compile them has to exist there. Also,
there are JS assets in some themes that provide additional functionality, and it
would be a performance degradation if those assets were no longer optimized.

What I do NOT want to happen:

* Significant end user performance degradation.
* Having an *additional* system in the asset pipeline (e.g. keeping
django-pipeline around while having additional systems).

I think that means that conceptually, there exists a larger Static Asset system
that exists and that we think of both Webpack and Django being consumers of its
configuration. This is also very fuzzy at the moment.

Asset Groups
————

There will be logical groupings of static assets. There should be uniformity and
no duplication within a group, but we would allow duplication between groups to
better facilitate independent deployment and isolation.

Example Groups:

* XBlock/XModule Assets
* LMS/Studio apps in edx-platform
* Third party app, such as edx-enterprise