Install Open edX Notes & Annotations

Get Notes & Annotations working on your Open edX native build. This post is a supplement to the official documentation, “How to Get edX Notes Running“, covering installation and configuration plus some pro tips on maintaining your environment afterwards.

Summary

Open edX Notes & Annotations is one of the coolest, most value-added upgrades that you can add to your Open edX Native build. Hopefully some day it will ship with the platform’s core functionality, but until then this post will help you get Notes & Annotations up and running on your Ginkgo installation (or later) in just a couple of hours. Notes itself is a Django app which largely follows the conventions of norms of the rest of the Open edX ecosystem. Technically speaking, it’s beautifully designed, works great, and really showcases the edX design team’s ability to thread the needle when it comes to enhancing the user experience while keeping the platform manageable.

Setup Procedure

We’ll use Ansible for most of the installation, leaving us with just a few chores beforehand and some cleanup activities afterwards. Here’s our installation and setup procedure:

1. Setup Oauth between LMS and the Notes API

First, we’ll use the Django admin console of the LMS to setup seamless Oauth authentication between the LMS and the Notes API. You’ll need to login using a super user account. You can review the “Managing Open EdX Tips and Tricks” page for instructions on how to convert your account. For the Python/Django uninitiated, Django apps like LMS and CMS come with a “back end” admin console where additional configuration parameters are available beyond what you’ll find in the four JSON files in /edx/app/edxapp/. Refer to this screen shot for the URL path and guidelines for creating your Oauth client.

2. Set Configuration Parameters

Take note that what follows is an unorthodox approach compared to most other configuration activities you’ll perform on your Open edX installation, but so far I haven’t come up with a better alternative.

A. Set Ansible Parameters

First, we’ll edit the Ansible Notes API defaults parameters file that is part of the edx “Configuration” github repository: /edx/app/edx_ansible/edx_ansible/playbooks/roles/edx_notes_api/defaults/main.yml. Following are the exact parameters that we’ll modify.

  • EDX_NOTES_API_MYSQL_DB_PASS – Choose a new strong password that the Notes user will pass to MySQL when connecting to the Notes database.
  • EDX_NOTES_API_MYSQL_HOST – This is “localhost” by default on Native installations unless you have migrated your MySQL environment elsewhere.
  • EDX_NOTES_API_ELASTICSEARCH_URL – set this to “localhost:9200”
  • EDX_NOTES_API_DATASTORE_NAME – This is the name of the MySQL database that Ansible will create for you. Use a simple, sensible name like “notes”
  • EDX_NOTES_API_SECRET_KEY – Choose a new strong password of at least 16 characters.
  • EDX_NOTES_API_CLIENT_ID – This is the “Client ID” value from the Oauth setup screen from the previous step
  • EDX_NOTES_API_CLIENT_SECRET – This is the “Client Secret” value from the Oauth setup screen from the previous step
  • EDX_NOTES_API_ALLOWED_HOSTS – add a new item to this list containing the fully qualified domain name of your LMS. See my example below (Row 50)

Following is an example showing how your Oauth configuration from step 1 should map to your Ansible parameter values in this step.

B. Set Application Parameters

Next we’ll edit the LMS environment configuration to enable Notes in the front end: /edx/app/edxapp/lms.env.json. There are three sets of changes within the Open edX platform json configuration files:

  1. EDXNOTES_INTERNAL_API & EDXNOTES_PUBLIC_API (row 154): The Notes application relies on a REST api that is authenticated by the LMS via Oauth. The API provides the annotation highlight meta data along with descriptive and tag data that the learner might have provided. These two parameter values by default are set to 127.0.0.1 (your local host). You’ll need to explicitly set both of these parameter to your fully qualified domain name. http[s]://[your-domain.com]:18120/api/v1
  2. JWT_ISSUER (rows 234 & 239): JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. These tokens are used by the Oauth authentication processes. Change each of these parameters from 127.0.0.1 to your fully qualified domain name, as you did in the previous step above.
  3. ENABLE_EDXNOTES (row 190): Set this to true.

NOTE: Running Ansible sometimes results in the configuration files in /edx/app/edxapp being overwritten.

3. Create Users with Ansible

Ok, we’re ready to put Ansible work, which is a two-step process for this installation. In this step we’ll use Ansible to create a new Linux user, “edx_notes_api”, plus create a MySQL user and then finally add permissions so that our new user can perform basic CRUD operations on the database. This is quick. This should take one or two minutes.

source /edx/app/edx_ansible/venvs/edx_ansible/bin/activate
cd /edx/app/edx_ansible/edx_ansible/playbooks
sudo ansible-playbook -i 'localhost,' -c local ./run_role.yml -e 'role=edxlocal' -e@roles/edx_notes_api/defaults/main.yml
4. Install Notes Software with Ansible

Now we’re ready to install the Notes Django app. This is also quick. This should take a minute or two to complete.

source /edx/app/edx_ansible/venvs/edx_ansible/bin/activate
cd /edx/app/edx_ansible/edx_ansible/playbooks/edx-east
sudo ansible-playbook -i 'localhost,' -c local ./notes.yml -e@/edx/app/edx_ansible/server-vars.yml
5. Run Database Migrations

For anyone new to Django, there’s a process that is generally referred to as “Database Migrations” whereby Django introspects it’s own code to deduce what database tables, fields and relationships are necessary to persist the data described in the various Python objects referred to in the code. This is one of the most compelling features of the Django framework but it can also make software installations seem buggy or at least problematic the first time you slug through the process.

Following are the commands to run database migrations for the edX Notes API, noting that DB_MIGRATION_USER refers to the MySQL user, which probably should be either “root” or “admin”, and DB_MIGRATION_PASS refers to the password for this user. Note also that the “admin” user is created as part of the Native build. You’ll find the password for this user in  /home/ubuntu/my-passwords.yml, stored as the parameter value COMMON_MYSQL_ADMIN_PASS on or around row 12 of this file.

export EDXNOTES_CONFIG_ROOT=/edx/etc/
export DB_MIGRATION_USER=root
export DB_MIGRATION_PASS=replacethisstringwithyoursuperduperaswesomepassword
/edx/bin/python.edx_notes_api /edx/bin/manage.edx_notes_api migrate --settings="notesserver.settings.yaml_config"

The first time your run this you should see a few screens worth of output. You can run migrations any time and as often as you want. It only modifies your database structures if/when its really needed.

6. Compile Assets & Restart edX Applications

The Open edX developer team created a defined process to sweep up and organize all static assets in the LMS and CMS, generically referred to as, “Compiling Assets”. Any time you add or modify either of these two applications you’ll need to run this process again. Otherwise you’re prone to getting some really quirky behavior in the UI. In the worst of cases the app will fail to start, leaving you with a stoic photo image of a sinking ship – yikes.

Following are the operating system commands to manually compile assets for both the LMS and CMS:

sudo -H -u edxapp bash
source /edx/app/edxapp/edxapp_env
cd /edx/app/edxapp/edx-platform
paver update_assets cms --settings=aws
paver update_assets lms --settings=aws

Note: this process takes up to 15 minutes to complete during which time the LMS and CMS will be unavailable to end user.

After this process finishes you should restart the two applications using these commands, and noting that the “:” symbols are intentional and should be included the commands:

/edx/bin/supervisorctl restart edxapp:
/edx/bin/supervisorctl restart edxapp_worker:
7. Enable Notes In One Or More Of Your Courses

For Notes to appear as an option to your online learners you first have to explicitly enable this feature on a course-by-course basis from the “Advanced Options” selection of the “Configuration Menu” in Studio for each course. You’ll find a true/false parameter option “Enable Notes” around halfway down this screen.

8. Verify The Installation

After logging in to the LMS as a student user and entering any course which you’ve enabled Notes you should see a new window tab, “Notes” where annotations will appear. Highlighting text anywhere in this course should result in a popup floating menu with Annotation options.

Trouble Shooting

  • Firewall. You should pay close attention to the ports used by the Notes API. You might need to open port 18120 on your firewall.
  • Nginx. If you’re using SSL on your site then you probably need to make adjustments to the Nginx virtual server configuration file /edx/app/nginx/sites-available/edx_notes_api. See screen shot below for an example of a site that uses a Letsencrypt SSL certificate.

I hope you found this helpful. Please help me improve this article by leaving a comment below. Thank you!

By |2018-08-01T10:05:10+00:00April 27th, 2018|Categories: Open edX|15 Comments

About the Author:

Lawrence is a full stack developer specializing in web and mobile development using AngularJS, Ionic, Wordpress and Amazon Web Services. He has worked as a freelance technology consultant since 1999. He earned a BS in computer science and mathematics with minors in physics and English from University of North Texas.

15 Comments

  1. gang wang August 31, 2018 at 5:26 am - Reply

    when i click `notes`, it shows error: `There has been a 500 error on the Your Platform Name Here servers`, check the log :`EdxNotesParseError: Invalid JSON response received from notes api.`?

    • admin August 31, 2018 at 7:31 am - Reply

      what version of Open edX are you running?

  2. Ashutosh August 30, 2018 at 9:59 am - Reply

    Hi Lawrence,

    I followed the steps and was able to set up edx notes but I am getting “Sorry we could not create this annotation” when I load any course page. This also occurs while saving any note.

  3. Ashutosh August 30, 2018 at 9:57 am - Reply

    Hi Lawrence,

    I was able to successfully set up edx notes. But on the course page, I get a 500 error “Sorry we could not create this annotation”. It also occurs while saving any note. Following is the log for the same: <> is my domain for noteserver

    Aug 30 15:32:45 ip-10-0-2-84 [service_variant=edx-notes-api][django.request][env:no_env] ERROR [ip-10-0-2-84 3249] [exception.py:135] – Internal Server Error: /api/v1/annotations/
    Traceback (most recent call last):
    File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/handlers/exception.py”, line 109, in get_exception_response
    response = callback(request, **dict(param_dict, exception=exception))
    File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/utils/decorators.py”, line 145, in _wrapped_view
    result = middleware.process_view(request, view_func, args, kwargs)
    File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/middleware/csrf.py”, line 277, in process_view
    good_referer = request.get_host()
    File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/http/request.py”, line 113, in get_host
    raise DisallowedHost(msg)
    DisallowedHost: Invalid HTTP_HOST header: ‘<>’. You may need to add u”<>’ to ALLOWED_HOSTS.

    • admin August 30, 2018 at 10:14 am - Reply

      hi Ashutosh, interesting error. a couple of questions: a) have you made any changes to your Nginx configuration? and b) where are you hosting your Open edX instance?

      • Ashutosh September 3, 2018 at 5:51 am - Reply

        Thank you for your quick reply.
        a) I have made no changes to the Nginx config.
        b) I have hosted the production stack on AWS.
        It is probably an issue with some settings file. So to at least make it work, I edited the request.py file and hardcoded my noteserver URL. Now I’m able to save notes in the database (notes are visible in the Notes tab). But it looks like it is not getting saved in elasticsearch. I’m still getting the “Sorry we could not create this annotation” error on top.

        Here is the edx-notes-api log for the same –

        Sep 3 11:33:25 ip-10-0-2-55 [service_variant=edx-notes-api][django.request][env:no_env] ERROR [ip-10-0-2-55 31684] [exception.py:135] – Internal Server Error: /api/v1/annotations/
        Traceback (most recent call last):
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/handlers/exception.py”, line 41, in inner
        response = get_response(request)
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/handlers/base.py”, line 249, in _legacy_get_response
        response = self._get_response(request)
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/handlers/base.py”, line 187, in _get_response
        response = self.process_exception_by_middleware(e, request)
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/core/handlers/base.py”, line 185, in _get_response
        response = wrapped_callback(request, *callback_args, **callback_kwargs)
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/views/decorators/csrf.py”, line 58, in wrapped_view
        return view_func(*args, **kwargs)
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/views/generic/base.py”, line 68, in view
        return self.dispatch(request, *args, **kwargs)
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/rest_framework/views.py”, line 489, in dispatch
        response = self.handle_exception(exc)
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/rest_framework/views.py”, line 449, in handle_exception
        self.raise_uncaught_exception(exc)
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/rest_framework/views.py”, line 486, in dispatch
        response = handler(request, *args, **kwargs)
        File “/data/edx/app/edx_notes_api/edx_notes_api/notesapi/v1/views.py”, line 367, in post
        note.save()
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/models/base.py”, line 808, in save
        force_update=force_update, update_fields=update_fields)
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/db/models/base.py”, line 848, in save_base
        update_fields=update_fields, raw=raw, using=using,
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py”, line 193, in send
        for receiver in self._live_receivers(sender)
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/haystack/signals.py”, line 52, in handle_save
        index.update_object(instance, using=using)
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/haystack/indexes.py”, line 284, in update_object
        backend.update(self, [instance])
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/haystack/backends/elasticsearch_backend.py”, line 190, in update
        bulk(self.conn, prepped_docs, index=self.index_name, doc_type=’modelresult’)
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py”, line 188, in bulk
        for ok, item in streaming_bulk(client, actions, **kwargs):
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py”, line 160, in streaming_bulk
        for result in _process_bulk_chunk(client, bulk_actions, raise_on_exception, raise_on_error, **kwargs):
        File “/edx/app/edx_notes_api/venvs/edx_notes_api/local/lib/python2.7/site-packages/elasticsearch/helpers/__init__.py”, line 132, in _process_bulk_chunk
        raise BulkIndexError(‘%i document(s) failed to index.’ % len(errors), errors)
        BulkIndexError: (u’1 document(s) failed to index.’, [{u’index’: {u’_type’: u’modelresult’, u’_id’: u’v1.note.64′, u’ok’: True, u’_version’: 1, u’_index’: u’edxnotes’}}])

        Elasticsearch logs have these errors which are not always thrown on requests –
        [2018-09-03 10:19:55,417][DEBUG][action.index ] [Arsenic] [edxnotes][2], node[XRX6RWoDR0uivEw64qHXMg], [P], s[STARTED]: Failed to execute [index {[edxnotes][_mapping][modelresult], source[{“modelresult”: {“properties”: {“ranges”: {“index”: “not_analyzed”, “type”: “string”}, “django_id”: {“include_in_all”: false, “index”: “not_analyzed”, “type”: “string”}, “created”: {“type”: “date”}, “quote”: {“type”: “string”, “analyzer”: “snowball”}, “tags”: {“type”: “string”, “analyzer”: “snowball”}, “updated”: {“type”: “date”}, “django_ct”: {“include_in_all”: false, “index”: “not_analyzed”, “type”: “string”}, “user”: {“index”: “not_analyzed”, “type”: “string”}, “text”: {“type”: “string”, “analyzer”: “snowball”}, “course_id”: {“index”: “not_analyzed”, “type”: “string”}, “data”: {“type”: “string”, “analyzer”: “snowball”}, “usage_id”: {“index”: “not_analyzed”, “type”: “string”}}}}]}]
        org.elasticsearch.indices.InvalidTypeNameException: mapping type name [_mapping] can’t start with ‘_’
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:212)
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:200)
        at org.elasticsearch.index.mapper.MapperService.documentMapperWithAutoCreate(MapperService.java:351)
        at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:373)
        at org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:205)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)

  4. Eric Mensah May 14, 2018 at 9:46 pm - Reply

    Ok, thank you!

  5. Eric May 8, 2018 at 6:16 pm - Reply

    Hi Lawrence,

    Thank you for the great work. In step 4, did you mean “-e@/edx/app/edx_ansible/edx_ansible/server-vars.yml”? I think your “-e@/edx/app/edx_ansible/server-vars.yml” ommitted one “edx_ansible” folder.

    I could be wrong though. Please confirm.

    • admin May 8, 2018 at 6:25 pm - Reply

      hi Eric, technically server-vars.yml can reside wherever you like. to my knowledge it’s only referenced once, in a single python script edx/bin/update, which itself is easy to modify. But at any rate this script looks for the file in this path /edx/app/edx_ansible/server-vars.yml. hope that helps! 🙂

      • Eric May 11, 2018 at 11:43 am - Reply

        Thanks for your response, Lawrence!

        I didn’t see server-vars.yml so I run main.yml and ended destroying my edx environment. Do you have any idea how I can create a server-vars.yml file for the Native Installation Ubuntu 16.04?

  6. Luis April 29, 2018 at 12:03 pm - Reply

    Great!! I’m eager to start!!

    • admin May 7, 2018 at 5:51 am - Reply

      hi luis, thank you for your help trouble-shooting this installation procedure. i’ve added additional configuration instructions to step 2 for the JWT Issuer and the EDX Notes API.

Leave A Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.