Performance Tuning AWS EFS For WordPress

AWS’ Elastic File System (EFS) service can seem like the white whale of the arcane world of horizontal scaling for WordPress. It’s so promising and so easy to create and mount, but it delivers repugnant performance results; initially at least. If you’ve already tried migrating a WordPress site for a local file system to an AWS EFS volume then you likely saw the performance of your site dramatically degrade to the point of being unusable. But why? There are two reasons.

First, with NFS there are unavoidable increases in file access latency. This situation amplifies whenever your web app (more rather its underlying Php engine) accesses many small files over a short period of time. Unfortunately, that’s basically what Php is doing all the time, making local file caching critically important. Fortunately this is easy to remedy by installing cachefiles and memcached.

Second, your IO operations on EFS volumes are subject to limits based on a size tier system called Burst Credits, and it is often the case that your EFS volume initially resides within the pathetically slow bottom tier of this system. This is also easily remedied, as we’ll see below.

Some site management processes, namely virus scanning and site backups, will require re-thinking so as to avoid exhausting Burst Credits. We’ll look at the key factors below.

Let’s get that elephant out of the room right away. EFS is up to three orders of magnitude slower than the EBS counterpart from which you want to migrate, and the tips in this article are simply ways to work around this reality rather than change it. Php applications like WordPress are a worst-case scenario for this problem in that migrating to EFS will always result in some performance degradation in various parts of the application. Lets looks at why.

The charts below from Amazon EBS Volume Types indicate that EBS volumes of 100G or less allow throughput of 1,750mib / second whereas an NFS volume of the same size allows throughput of 500kib / per second (see table below for EFS Volume Sizing). That’s up to a 3,500x decrease in throughput! But this oversimplifies matters. You also need to look at what happens with individual disk fetches, or IOPS in AWS parlance. Generally speaking, any storage system will more efficiently retrieve a large stream of data like say, a video file, more efficiently that it can a random collection of small files like say, a bunch of Php files. That is to say that an EFS volume proportionately works harder to serve up a Php application, exacerbating an already-large reduction to data throughput. It behooves you to keep this in mind as we move on to my performance tuning tips.

Red Hat published an excellent file caching solution for NFS back around 2006, and which is installable on Amazon Linux as a yum package. It’s idiot proof and a highly effective way to minimize file reads from your NFS volume. Given the wretched performance all customers initially see from EFS I don’t know why AWS does not include this in their mounting instructions.

sudo yum install -y nfs-utils   # to ensure that NFS is installed on your Amazon Linux instance.
sudo yum install cachefilesd
sudo service cachefilesd start
sudo chkconfig cachefilesd on    #autostart cachefilesd

AWS provides precise EFS volume mounting instructions via a hyperlink inside the settings of your EFS volume within the AWS Console. However, you’ll need to make a small modification to these instructions so that your volume is aware of cachefiles. AWS’ mount instructions include a list of options which you’ll see immediately after the -o parameter.

Change this line: “nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2”
To the following: “nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,fsc”

The additional option “fsc” instructs NFS that we’ll be using File System Caching on the mounted volume. The complete set of instructions will look similar to the following:

sudo mkdir /efs
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,fsc [ADD YOUR FILE SYSTEM END POINT]:/ /efs
sudo service cachefilesd restart  #to ensure that our cachefiles service becomes aware of our newly mounted efs volume

Automatically mount volume on startup.

sudo vim /etc/fstab
[ADD YOUR FILE SYSTEM END POINT]:/ /efs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,fsc,_netdev 0 0

AWS governs IO operations on EFS volumes based on their size. If you’re migrating a single WordPress site to EFS then this will almost certainly cause problems for you. The workaround is to create a dummy file to force the size of your EFS volume into the performance tier that your site requires. Personally, I’d never given any thought to my site’s IO performance requirements and so I had no idea of my needs. If that’s your situation as well you should take a close look at AWS’ pricing and performance tier structure, also presented below. Keep in mind that artificially increasing the size of your EFS volume also makes it cost more.

Here is an example code snippet that you can use to create a dummy file of exactly 256G, which is exactly enough size to push your EFS volume up to the 2nd performance tier. Change the parameters to tailor the file size to your needs. Take note however that it takes around an 90 minutes to create a dummy file 256G in size. This snippet therefore wraps the command in a “nohup” command that runs the command on a background thread. You can monitor progress using the command line “ls -lh”.

cd /efs
sudo nohup dd if=/dev/urandom of=2048gb-b.img bs=1024k count=256000 status=progress &

Memcached is an industrial-grade, multi-purpose in-memory object cache. It’s easy to install and it provides immediate improvement to any Php web app. Take a look at my post on how to install memcached to get it up and running on your WordPress server in just a few minutes.

This is out of scope for this post but bears mentioning because serving static assets from your EFS volume will result in poor performance as well as prematurely exhausting your AWS Burst Credits. If you’re new to the topic then I suggest you get started by reading more about AWS’ Cloudfront Service which is easy to setup and inexpensive.

If you’re planning to migrate more than one WordPress site to an EFS volume then you should consider moving these individually, maximum one or two sites per day, so that you can monitor the impact of each site to the EFS IO operations. Additionally, you’ll observe unique behavior from each site for published content versus authoring activities. Authoring activities typically require more EFS IO and so you’ll need to observe each site for several days or weeks in order to develop a complete profile of EFS IO behavior.

Site backup and virus scanning plugins in particular can be problematic because they potentially will scan the entire file system for your site, and this consumes a lot of Burst Credit. For backups you should strongly consider implementing an EFS backup solution like AWS Glacier as a complete replacement to individual backups of site files. Ditto with virus scans. Personally, I’m still actively researching alternatives to common solutions like Wordfence that will play nicely with a EFS volume.

AWS Cloudwatch provides several performance metrics for EFS. The most important of these of course is your Burst Credit Balance. As you migrate sets of files to your EFS you’ll want to monitor this metric over the course of a day or so in order to fully understand it’s impact to your Burst Credit balance.

Additionally, you should monitor the impact of the EFS to your page download speed. In the worst of cases your site could come to a standstill, and in the best of cases the EFS will have no impact whatsoever. Many factors about your hosting environment come into play, making this aspect of setting up a EFS a very personal journey. Incidentally, this site runs on a AWS EFS.

Speaking of measuring page speed, an AWS business support tech from the EFS group sent the following snippet to me when I was in the early stages of trying to get an EFS volume to work for the first time. It provides an informative set of page retrieval metrics which I find incredibly useful for benchmarking purposes. Note that you’ll need to modify the snippet to replace your site name with that of this blog.

curl -v -s -w ‘\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nAppCon time:\t%{time_appconnect}\nRedirect time:\t%{time_redirect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n’ -o /dev/null https://blog.lawrencemcdaniel.com