Open edX Troubleshooting Guide

When your Open edX installation is misbehaving and  you’re unsure of your next step, I always like to begin with a reboot. It can’t hurt, and occasionally you’re pleasantly surprised to find your instance in good health immediately afterwards. The commands are as follows:

• A cold boot (reboots the entire server)

  sudo reboot

• A warm boot (restarts Open edX)

  # These are the commands for restarting the LMS and CMS beginning with Gingko
  /edx/bin/supervisorctl restart lms
  /edx/bin/supervisorctl restart cms
  
  # If you're running a previous version of Open edX then use this command instead
  /edx/bin/supervisorctl restart edxapp:
  
  # This is the command to stop and restart the RabbitMQ Celery workers
  /edx/bin/supervisorctl restart edxapp_worker:
  

This is nothing more than a minor speed bump provided that you still have terminal access via SSH. Open edX relies on a single master user list that is stored in MySQL in the table “edxapp.auth_user”. You have a few options for how to regain administrative access to the LMS and CMS depending on your level of comfort with the different technologies.

1. Review the row-level contents of the MySQL table edxapp.auth_user. You can login to MySQL from a Linux terminal window using the command line. By default the root password of the local instance of MySQL is left blank. If you’re unable to login as root then Open edX also creates a MySQL user named edxapp, the password of which you’ll find in the master password file that is (hopefully) saved in /home/ubuntu/my-passwords.yml.
   – To login to MySQL from the command line: mysql -u root -p
   – To view all users: select * from auth_user;
   – To logout of MySQL: exit
   Once you find your user name and email address you can use the LMS login screen to request a password reset.
   
2. Create a new user with the Django Command Line Interface. If you’re more comfortable working with Django then you can create a completely new superuser account with the following command, substituting “mcdaniel” with your username and “lpm0073@gmail.com” with your email address:

   sudo su edxapp -s /bin/bash
   cd
   /edx/bin/python.edxapp /edx/bin/manage.edxapp lms lmcdaniel staff lpm0073@gmail.com --staff --superuser --settings=aws

   To change the password execute this line:

   sudo -u www-data /edx/bin/python.edxapp /edx/app/edxapp/edx-platform/manage.py lms --settings aws changepassword mcdaniel

3. Insert a new user record into the MySQL table “edxapp.auth_user”. If all else fails then you can force a new record into the user table. Take note however that password values are encrypted and thus you cannot simply insert a password value into this table. Instead you’ll need to either use the Django console command from option 2 above, or request a password reset from the LMS login screen. The insert statement will be of the form:

   INSERT edxapp.auth_user (is_superuser, username, first_name, last_name, email, is_staff, is_active) VALUES(1, 'mcdaniel', 'Lawrence', 'McDaniel', 'lpm0073@gmail.com', 1, 1);

If you recently installed Open edX and are just getting started setting up new users then you’ve probably seen that the “Account Activation” welcome emails that Open edX sends to newly registered users are almost always flagged as spam by users’ email management applications like MS Outlook and Mac Mail. In defense of the architects of Open edX, this is actually not a defect of the software but rather an indication that you still have more configuration and systems integration work to do.

To cure this problem you’ll need integrate your Open edX instance to an SMTP-compliant email server like MS Exchange or Gmail. I wrote a blog post, “Setup SMTP Email on Open edX“, that provides detailed step-by-step instructions.

Incidentally you might have also run into problems with the activation email itself providing a broken account activation link that points to “http://localhost”. This is a simpler problem which you can remedy by editing the configuration file /edx/app/edxapp/lms.env.json. Look for the following three parameter values:

"LMS_BASE": "localhost",
"LMS_INTERNAL_ROOT_URL": "localhost",
"LMS_ROOT_URL": "localhost",

and then replace the value “localhost” with your fully qualified domain name. These parameter values should be located contiguously beginning approximately on row 245 of this file.

If your Open edX server seems sluggish, and …

• your Open edx instance is running in AWS or Azure or GCC, and
• you have not modified the program code of your Open edX instance, and
• your Open edX instance is intended to support fewer than a few hundred concurrent users (ie users simultaneously logged in to your server)

then the short story is that you probably need more computing infrastructure. Save yourself time and brain damage by not attempting to “tune” your Open edX instance. It is already optimized, and frankly, you’ll do more harm than good if you begin tinkering with operating system and subsystem configurations. Instead, you can use the Linux command “top” to view real-time vital statistics of your Ubuntu server instance to begin to get a better understanding where performance bottlenecks problems might exist.

If you do not see problems with CPU and Memory then your problems might lie elsewhere with your IT infrastructure, beginning with network bandwidth and latency on both the server and client sides.

Open edX uses lots of http ports, and many of these are non-standard. If your server appears to be functional but you’re having trouble getting LMS or CMS pages to render in your browser then your firewall ports are a good place to begin troubleshooting. As of this writing you should ensure that the following ports are opened in your firewall:

If your LMS returns the dreaded “Uh oh, we are having some server issues..” message then you most likely are suffering from configuration errors that are directly related to the parameter values stored in any of the four JSON files saved in /edx/app/edxapp/.

These errors generally fit into one of the following cases:

• a JSON Syntax error in the file. This file has to syntactically conform to JSON standards. If there are ANY syntax violations, no matter how minor they might seem, this will break your Open edX installation. I prefer to edit these files on my laptop using a code editor like Atom or Sublime. Both of these editors provide native syntax highlighting for JSON files, which will help you to quickly identify any structural problems in the files.
• An invalid parameter value that causes Python/Django to throw an application exception. For example, if one of these parameters expects an integer but you provided a decimal number or an alpha character then this will break your Open edX instance. Frustratingly, in most cases you will not find ant useful diagnostics information in the logs files (example: /edx/var/log/lms/edx.log). Therefore, you should be methodical in modifying parameter values, testing each change individually.
• An invalid URL / URI. Keep in mind that the syntax of URL’s is evaluated at application startup whereas other problems like http return errors are evaluated at run-time. Thus if you’re certain that a recently-modified URL’s are causing problems then at the very least you can use this as a guideline for refining the nature of your problem.

Alternatively, if you’ve successfully modified one of these files and the platform still works, but the change doesn’t seem to have taken effect then you probably need to restart the applications. I suggest that you download the following script and edit the commands as indicated in the description in the script header: https://github.com/lpm0073/edx.scripts/blob/master/edx.platform-restart.sh

A common problem that you might encounter when you first begin to work with Open edX’s comprehensive theming system is that modifications to your theme are inconsistently rendered in the browser. Changes to Mako templates take effect immediately whereas changes to static assets like images, CSS and javascript might not appear, or at the very least these do not render as you’d intended.

Theme assets have to be “compiled”. That is, a set of subroutines on the server must first aggregate, pre-process and archive these files into a caching area on the server located at /edx/var/edxapp/staticfiles/. The following commands execute the static asset compilation process:

# update assets as edxapp user
sudo -H -u edxapp bash << EOF
source /edx/app/edxapp/edxapp_env
cd /edx/app/edxapp/edx-platform
paver update_assets lms --settings=aws
paver update_assets cms --settings=aws
EOF
# restart edx instances
sudo /edx/bin/supervisorctl restart lms
sudo /edx/bin/supervisorctl restart cms
#sudo /edx/bin/supervisorctl restart edxapp: #for versions prior to Ginkgo
/edx/bin/supervisorctl restart edxapp_worker:

For more information you might want to read my blog article, “Open edX Custom Theming Tutorial“

There are a couple of situations which can leave you with an Open edX codebase that is out of synch with the underlying MySQL database schemas. If you’ve upgraded your edx-platform code base recently, or if you recently restored one or more of the MySQL databases from another instance then you might experience sporadic errors when trying to save certain information.

The LMS and CMS are both written in Python/Django, which relies on a process called “database migrations” whereby Django source code is analyzed to infer what database structures must exist in order to persist the data from the Django models. The following series of commands will execute database migrations for both the LMS and CMS.

sudo -H -u edxapp -s bash << EOF
cd ~
source /edx/app/edxapp/edxapp_env python /edx/app/edxapp/edx-platform/manage.py lms makemigrations --settings=aws
python /edx/app/edxapp/edx-platform/manage.py lms migrate --settings=aws
python /edx/app/edxapp/edx-platform/manage.py cms makemigrations --settings=aws
python /edx/app/edxapp/edx-platform/manage.py cms migrate --settings=aws
EOF

Like all Linux software, file ownership and permissions play an integral role in the Open edX architecture. Modifying files permissions, broadly speaking, is a terrible idea. But, if you find yourself wishing that you’d read this post sooner then here are a couple of general suggestions for getting your Open edX instance back to “factory” settings.

• Re-install the edx-platform software code. You should definitely snapshot your instance before heading down this path. But this is otherwise a simpler procedure that it may seem. Ansible does everything for you. The commands are as follows:

  sudo rm -r /edx/app/edxapp/edx-platform/
  nohup wget https://raw.githubusercontent.com/edx/configuration/master/util/install/sandbox.sh -O - | bash > install.out &
  

  The installation will run as a background process which you can process by using the “tail” command on the the install.out output file. You can also use “top” to monitor the activity and progress real-time.

• Trying the following:

  sudo chown -R edxapp /edx/app/edxapp/edx-platform/
  sudo chgrp -R edxapp /edx/app/edxapp/edx-platform/
  sudo chmod 2755 /edx/app/edxapp/edx-platform/
  find /edx/app/edxapp/edx-platform -type d -exec sudo chmod 2755 {} \;

  As best I can tell, all file permissions appear to be 755 and the owner and group are both assigned to edxapp. This code snippet will set all files in the edx-platform codebase to these exact settings. While this APPEARS to be correct, I have no way to verify that this is unequivocally the case throughout the entire codebase.

During your original native build installation process you executed a really cool piece of code — https://github.com/edx/configuration/blob/master/util/install/generate-passwords.sh — that generates strong, randomized passwords for the couple of dozen system accounts that support your Open edX installation. The native build ansible installation scripts refer to the resulting file, /home/ubuntu/my-passwords.yml, to assign passwords for all of these system accounts.

If you deleted the file my-passwords.yml from your home directory then it might be game over for you. The only cure to not having these passwords available is the following:

• individually export your courses
• create a dump of the MySQL databases (assuming that the root password is still blank)
• build a new Open edX instance, import your courses, and restore the MySQL data
• run database migrations
• compile static assets

If you suspect that you’re having problems with Nginx then please note that I’ve never needed to modify the stock configurations created by Ansible other than to add SSL certificates, in which case I use the procedure described in my blog post, “Add SSL Encryption to Open edX“. That is, if I haven’t previously modified the Nginx configuration to add SSL then it definitely has never been a problem.

If you remain skeptical then you might try the following, which will restart the Nginx service and then provide a status report:

sudo systemctl restart nginx
sudo systemctl status nginx

Celery — also known as RabbitMQ — is a low-level subsystem that brokers tasks and messages between the LMS and CMS applications and the many underlying subsystems. On an aside, once I wrapped my head around what RabbitMQ does I was completely blown away by the elegance of it all. It’s a work horse, and for the most part its a trouble-free cornerstone of your platform. Occasionally however, you’ll do something like say, migrate your Open edX instance to a larger server, that unexpectedly causes Celery to break.

This topic is important enough that it got it’s own dedicated blog post, “Troubleshooting Celery & RabbitMQ For Open edX“.

If you’re not a Python Django developer and you haven’t previously supported a Django app then you’d have no reason to be aware that Django apps have their own back end that’s somewhat similar the WordPress admin console. From your web browser, open your LMS, login, and then navigate to /admin/.

I suggest that you spend some time to get acquainted with the parameters that are available to modify in the console. How and what parameters make their way onto the Django console screen are the stuff of books on learning to code in Django — and I shan’t steal their thunder here in my humble blog.

I hope you found this helpful. Please help me improve this article by leaving a comment below. Thank you!