Developer’s Guide to Opaque Keys

Though the Open edX course structure and its object key system, Opaque Keys are each quite complex, you’ll see in “Code Sample I” that we can create and traverse a course outline in only four lines of code. With a cursory understanding of edx-platform’s in-place application api’s you’ll be able to do amazing things with little code.

Our discussion necessarily begins with Opaque Keys as these are the code objects that describe individual chunks of course content. By way of example, each of the following is a string representation of some kind of Opaque Key:

Opaque Keys do the following:
1.) accurately describe any course object in terms of its structural location, which course run, which branch, and which version
2.) facilitate a map to-from the Open edX URL structures
3.) provide backward compatibility to older Open edX key and storage strategies

Opaque Keys are referred to generically in documentation as locators, keys, course keys, course id’s, asset keys, and usage keys. None of these is incorrect mind you. The three most common kinds of opaque keys are:

A CourseLocator (formerly CourseKey) contains many BlockUsageLocators (AssetKeys and UsageKeys), and it knows about these. An AssetKey and a UsageKey each knows which CourseKey it is associated with. DefinitonKey’s are context independent. The class hierarchy diagram below helps to make sense of each of these statements.

Opaque keys are fast and powerful. The keys are referenced everywhere in the code base and therefore need to instantiate and execute performantly. Incidentally, there’s a longer-term plan for Opaque Keys which will make them, well, more opaque. While I’m not privy to the salient details of this vision, I am aware that bright minds are hard at work at a brighter future for Opaque Keys (pun intended).

Modern Opaque Keys provides a way to embed additional distinguishing content attributes such as course run, branch (ie “Draft” or “Published”), and version in the object itself. Naturally, attributes of this nature have implications to the user experience and the overall behavior of the platform itself. Much of this is encapsulated in the Opaque Keys project.

In Open edX source code, whether this be in edx-platform or any other code repository in the Open edX ecosystem, the string representations of Opaque Keys such as the course_id slug in a course url are consumed by the pip-installed opaque-keys Python package which provides powerful introspection and utility capabilities. As the name “Opaque” implies, there is more to these keys than meets the eye. The Opaque Keys project was originally part of edx-platform, but it was spun off into a separate repository in 2014.

I am aware of only these three documentation sources for Opaque Keys, each located in the edx-platform repo Wiki: 1.) “Opaque Keys (Locators)“, 2.) “Locators: Developer Notes“, and 3.) “Split: the versioning, structure saving DAO“. When reading these documents it behooves you to keep in mind that they were written to inform an already-knowledgeable audience about noteworthy changes to Opaque Keys at a point in time in the past. It’s therefore pretty easy to draw incorrect conclusions from these documents. Conversely, this article is intended to illustrate in practical terms how to make use of these technologies at present based on my own personal experience.

Something else to keep in mind is that Opaque Keys have changed significantly over the life of the project (10+ years), and the Open edX code base provides backward compatibility to all of its incarnations. In many cases this complicates analyzing and understanding the source code. This is a direct effect of the double-edge sword of maturing open-source projects. it’s oftentimes best to not delve too deeply into the sausage-making of Opaque Keys’ inter-workings. Fortunately, the Opaque Key classes have been stable for more than seven years, and thus you have little if no need to burden yourself with its knotty history.

Lastly, the Opaque Keys project straddles large and gradual tectonic shifts in Open edX’s persistence strategy referred to, respectively, as “Old Mongo” and “Split Mongo”. This itself merits an eventual blog post as the changes were substantial and thus also impacted the complexity of the lower-levels of the Open edX code base. Vastly paraphrasing, the modern Open edX platform leverages XModule, which identifies courses with “Locators” (a kind of Opaque Key), and reusable program code with “Locations” (also a kind of Opaque Key).

From the extension engine blog, “XBlock is the SDK for the edX MOOC platform. It’s a framework that allows the open source software development community to extend and enhance edX to meet future needs and is written in Python2. XBlock is one of the aspects of edX that makes it such a powerful tool.” Except this is a little bit misleading in that all Open edX course content is created with XBlocks, also referred to more generically as “blocks”.

We could talk at length about blocks but we’ll save that for another blog post someday soon, hopefully.

All blocks are identifiable and addressable with a Locator (an Opaque Key class object instance), which in turn means that all blocks have a url, and, you can create an instance of any course block from XModule; even though blocks can vary significantly in terms of their UI, feature set and storage needs.

In practical terms, a Course Structure is the course outline of a given course run. It exists in a well-ordered tree-type organizational structure of various kinds of XBlocks. A Course Structure is built from blocks of discernible categories including “course”, “chapters” (Sections), “sequentials” (Subsections) and “verticals” (Units). Course Management Studio’s “Course Outline” view provides a technically accurate and complete representation of a Course Structure.

It turns out that every block in a course, including the block representing the course run itself, has a UsageKey. That’s because CourseLocator and BlockUsageLocator both inherit UsageKey. The UsageKey for the course itself is identifiable in code as the “root_block_usage_key”. All blocks individually can be traversed since all blocks are aware of their children. As per “Code Sample II” You can programmatically find any block’s parent vertical, sequential, chapter or course since blocks are aware of their parents,  these are generically inspectable categories of all blocks.

Persisting a data structure of this complexity requires both MySQL and Mongo, and potentially a file system or CDN as well. XModule exists to abstract these complexities from the basic CRUD of working with and organizing the blocks in a course run.

Instantiating and inspecting a Course Structure is actually a lot easier than it appears via inspection of the edx-platform code base, namely because you probably don’t need to worry about backwards compatibility and edge cases. The three code samples that follow provide examples of different ways to instantiate and traverse a course structure, and each of these only contains a few lines of salient code.

import string

from opaque_keys.edx.keys import CourseKey
from opaque_keys.edx.locator import BlockUsageLocator
from common.lib.xmodule.xmodule.modulestore.django import modulestore

# entry point to the block_structure api.
from openedx.core.djangoapps.content.block_structure.api import get_course_in_cache


def inspect_course_structure() -> None:
    """
    Create a block structure object for the demo course then
    traverse the blocks contained in the block structure.

    course_key:     opaque_keys.edx.keys.CourseKey
                    example course-v1:edX+DemoX+Demo_Course
    """
    def print_attribute(xblock: BlockUsageLocator, attr: string) -> None:
        print("XBlock {attr}: {val}".format(attr=xblock.display_name, val=xblock.__getattribute__(attr)))

    course_key = CourseKey().from_string("course-v1:edX+DemoX+Demo_Course")

    # get_course_in_cache() is part of the block_structure application api
    # which, when possible, is the preferred way to interact with course content.
    #
    # It is a higher order function implemented on top of the
    # block_structure.get_collected function that returns the block
    # structure in the cache for the given course_key.
    #
    # Returns:
    #   openedx.core.djangoapps.content.block_structure.block_structure.BlockStructureBlockData
    #   The collected block structure, starting at root_block_usage_key.
    collected_block_structure = get_course_in_cache(course_key)

    # topological_traversal() iterator returns all blocks
    # in the course structure, following the rules of a
    # topological tree traversal.
    # see https://en.wikipedia.org/wiki/Topological_sorting
    #
    # block_key is type opaque_keys.edx.locator.BlockUsageLocator
    for block_key in collected_block_structure.topological_traversal():

        # xblock is a BlockUsageLocator but it points to 
        # its fully-initialized XBlock content
        xblock = modulestore().get_item(block_key)

        # inspect some of the attributes of this xblock
        print_attribute(xblock=xblock, attr="display_name")
        print_attribute(xblock=xblock, attr="category")
        print_attribute(xblock=xblock, attr="location") # note: this is the block_key
        print_attribute(xblock=xblock, attr="start")
        print_attribute(xblock=xblock, attr="published_on")
        print_attribute(xblock=xblock, attr="published_by")
        print_attribute(xblock=xblock, attr="edited_on")
        print_attribute(xblock=xblock, attr="edited_by")

import string

from opaque_keys.edx.keys import UsageKey
from openedx.core.djangoapps.content.block_structure.block_structure import BlockStructureBlockData

from common.lib.xmodule.xmodule.modulestore.django import modulestore

def get_parent_location(category: string, block_key: UsageKey, block_structure: BlockStructureBlockData) -> UsageKey:
    """
    for the XBlock corresponding to the block_key, returns the UsageKey (location) for 
    its course, chapter, sequential, or vertical Block.

    Note: these equate to:
        course is the CourseSummary object
        chapter is a "Section" from Course Management Studio
        sequential is a "Subsection" from Course Management Studio
        vertical is a "Unit" from Course Management Studio

    Returns None if nothing is found.

    parameters:
    -------------
    category: xblock.fields.String or Python string
    block_key: opaque_keys.edx.keys.UsageKey, 
    block_structure: openedx.core.djangoapps.content.block_structure.block_structure.BlockStructureBlockData
    """

    xblock = modulestore().get_item(block_key)
    if xblock.category == category:
        return block_key

    for parent_key in block_structure.get_parents(block_key):
        parent_xblock = modulestore().get_item(parent_key)
        if parent_xblock.category == category:
            return parent_key

    return None

# python stuff
import string

# django stuff
from django.contrib.auth import get_user_model

# common stuff
from opaque_keys.edx.keys import CourseKey, UsageKey
from opaque_keys.edx.locator import BlockUsageLocator
from common.lib.xmodule.xmodule.modulestore.django import modulestore

# API entry point to the course_blocks app with top-level get_course_blocks function.
import lms.djangoapps.course_blocks.api as course_blocks_api

# XBlock transformers
# -------------------
from lms.djangoapps.course_blocks.transformers.access_denied_filter import AccessDeniedMessageFilterTransformer
from lms.djangoapps.course_blocks.transformers.hidden_content import HiddenContentTransformer
from lms.djangoapps.course_api.blocks.transformers.blocks_api import BlocksAPITransformer
from lms.djangoapps.course_api.blocks.transformers.milestones import MilestonesAndSpecialExamsTransformer
from openedx.core.djangoapps.content.block_structure.transformers import BlockStructureTransformers
from openedx.features.effort_estimation.api import EffortEstimationTransformer
# -------------------


def get_transformed_course_blocks():
    """
    Use the course_blocks api to create a list of the xblocks for 
    the demo course.

    Mutate the demo course's XBlocks using a list of
    commonly-used Transformers.
    """
    def print_attribute(xblock: BlockUsageLocator, attr: string) -> None:
        print("XBlock {attr}: {val}".format(attr=xblock.display_name, val=xblock.__getattribute__(attr)))

    # Note: refer to the class hierarchy above in this
    # blog post to better understand how these three
    # lines of code take us from a string value of a course key,
    # to a CourseKey, and finally to a UsageKey pointing to the 
    # root block of the course structure. 
    course_id = "course-v1:edX+DemoX+Demo_Course"

    # create an instance of opaque_keys.edx.keys.CourseKey
    course_key = CourseKey.from_string(course_id)

    # create an instance of opaque_keys.edx.keys.UsageKey
    # which is also the root usage key for the Course.
    course_usage_key = modulestore().make_course_usage_key(course_key)

    # any super user account w priveleges to see all content
    user = get_user_model().objects.filter(superuser=True, active=True).first()


    # BlockStructureTransformers encapsulates an ordered list of block
    # structure transformers. It uses the Transformer Registry
    # to collect their data.
    transformers = BlockStructureTransformers()

    # Transformers can add/remove xblock attribute fields, change 
    # the visibility settings of blocks and block attributes, as well as
    # change attribute values.
    #
    # We can add more transformers in order to continue to mutate
    # the xblock objects. 
    transformers += [
        # Default list of transformers for manipulating course block structures
        # based on the user's access to the course blocks.
        # these include:
        #    ContentLibraryTransformer(),
        #    ContentLibraryOrderTransformer(),
        #    StartDateTransformer(),
        #    ContentTypeGateTransformer(),
        #    UserPartitionTransformer(),
        #    VisibilityTransformer(),
        #    DateOverrideTransformer(user),
        course_blocks_api.get_course_block_access_transformers(user),

        # A transformer that handles both milestones and special (timed) exams.
        MilestonesAndSpecialExamsTransformer(),

        # A transformer that enforces the hide_after_due field on
        # blocks by removing children blocks from the block structure for
        # which the user does not have access. The hide_after_due
        # field on a block is percolated down to its descendants, so that
        # all blocks enforce the hidden content settings from their ancestors.
        HiddenContentTransformer(),

        # A transformer that removes any block from the course that has an
        # authorization_denial_reason or an authorization_denial_message.
        AccessDeniedMessageFilterTransformer(),

        # A transformer that adds effort estimation to the block tree.
        EffortEstimationTransformer(),

        # Umbrella transformer that contains all the transformers needed by the
        # Course Blocks API. includes the following transformers:
        #   StudentViewTransformer
        #   BlockCountsTransformer
        #   BlockDepthTransformer
        #   BlockNavigationTransformer
        #   ExtraFieldsTransformer
        BlocksAPITransformer()
    ]

    # create a BlockStructureBlockData object instance for 
    # the demo course.
    #
    # This is a transformed block structure,
    # starting at starting_block_usage_key, that has undergone the
    # transform methods for the given user and the course
    # associated with the block structure.
    #
    # If using the default transformers, the transformed block 
    # structure will be exactly equivalent to the blocks that
    # the given user has access.
    blocks = course_blocks_api.get_course_blocks(
        user,
        course_usage_key,
        transformers,
    )

    for xblock in blocks:
        # do stuff with the transformed xblocks
        print_attribute(xblock=xblock, attr="category")
        print_attribute(xblock=xblock, attr="display_name")
        print_attribute(xblock=xblock, attr="location")