A thanks to Adam Thurlow from SheepDog Inc. for providing this report for us!

Cherokee Webserver Report

Cherokee Logo

This report documents all of our findings testing the Cherokee Webserver for use with Django in production.

Cherokee is a very fast, flexible and easy to configure Web Server. It supports the widespread technologies nowadays: FastCGI, SCGI, PHP, CGI, uWSGI, SSI, TLS and SSL encrypted connections, Virtual hosts, Authentication, on the fly encoding, Load Balancing, Apache compatible log files, Data Base Balancing, Reverse HTTP Proxy, Traffic Shaper, Video Streaming and much more.

Detailed Overview of Cherokee

Installation


I documented all of the steps that I took in order to get the server working
on SpringPad.

Installation Docs on SpringPad

Benchmarking


The benchmarks were completed on all Dynamic Hosting virtualized hardware with varying purchased VPS packages. package list. Dynamic Hosting started us off with a standard Ubuntu 10.10 Bronze VPS Server, then as required, dynamically upgraded our package to Silver, then Gold. Here are the specs for each package that was used:

Bronze VPS

  • 128MB Guaranteed RAM
  • 256MB Burstable RAM
  • 1 CPU, 6 cores (limited)

Silver VPS

  • 384MB Guaranteed RAM
  • 512 Burstable RAM
  • 1 CPU, 6 cores (unlimited)

Gold VPS

  • 1GB Guaranteed RAM
  • 1.5GB Bustable RAM
  • 2 CPU, 6 cores each (unlimited)

Benchmark Results

All tests were performed using Apache Bench using multiple levels of concurrency from 25 to 200 from both an external server and on the local machine to test both physical throughput without the network latency and real life scenarios. The tests were all done on a non-static page (/login). No caching was enabled, and each test was comprised of 10000 requests.

$ ab -n 10000 -c 100 http://server/

Behold — A Table!

Platform                           MIN req/sec MAX req/sec  Stability
======================================================================
Platinum VPS /w Apache & WSGI            10          16        HIGH
----------------------------------------------------------------------
Bronze VPS /w Cherokee FastCGI           30          40        LOW
----------------------------------------------------------------------
Bronze VPS /w Cherokee uWSGI (1 proc)    45          65        HIGH
----------------------------------------------------------------------
Bronze VPS /w Cherokee uWSGI x 4 R.Rob.  62          70        MEDIUM
----------------------------------------------------------------------
Bronze VPS /w Cherokee uWSGI multiproc   78         105        HIGH
----------------------------------------------------------------------
Silver VPS /w Cherokee uWSGI multiproc  150         178        HIGH
----------------------------------------------------------------------
Gold VPS /w Cherokee uWSGI multiproc    225         263        HIGH
----------------------------------------------------------------------
Gold VPS Cherokee static file serving   N/A        5500        N/A

If you’re interested in how I got these numbers, keep reading :)

Findings

The Bronze package is known within the development team to perform miserably with both the Dashboard Framework and a standard Django installation on Apache using WSGI. As a base-line I tested a current beta site running in a VM on the regular Apache setup and the server maxed out at 10-16 req/sec. The server remained stable during the test (read: no crashing), however while using the site during the benchmark, the site was incredibly slow with page load time being from 15 to 30 seconds if they did not time out.

The Cherokee installation was progressively enhanced. I started with the Cherokee Django Wizard which uses FastCGI to serve the application and SQLite as the Database which required no configuration. This setup yielded a performance increase over the Apache server to between 30-40 req/sec. The server remained stable with a low amount of concurrent requests, however once ~3000 requests were served using $ ab -n 10000 -c 100 http://…, Cherokee ran out of memory and panicked. The fastCGI Handler kept spinning up new application threads, and this caused the whole serving process to quit and not re-spawn.

Dynamic Hosting explained why this happened:

“Once a VPS runs out of memory, the VM container starts killing large memory processes. The server process probably had the largest memory footprint and was removed from memory to preserve the stability of the other VMs hosted on the same machine.”

Optimizing

The next steps were spent trying to optimize the Cherokee server to keep stable under heavy load. Thinking that Cherokee’s spawning process simply had a higher minimum requirement than a Bronze VPS could handle; I requested Dynamic Hosting to upgrade the package to Silver to see if we could overcome this issue by simply adding additional resources. The same problem was observed on the Silver package as on the Bronze package. Once Cherokee spawned so many new processes to keep up with 100 concurrent requests, It simply ran out of memory and the VM container killed the process.

I/O Caching

I requested that the server be demoted to Bronze and started trying to find optimal settings to prevent the VM from panicking and killing the server process. I began by reading through the Cherokee documentation on optimization. They mentioned I/O Caching; which is supposed to “dramatically improve performance for serving files“, at the expense of putting the files in RAM. This feature is enabled by default, so I disabled it to preserve memory for the server to spawn new processes. Running the tests again yielded the same result as without I/O Caching.

SQLite

SQLite is known to be a lightweight, but slow performing database. The replacement I chose for the default SQLite backend was PostgreSQL using the psycopg2 connector. I won’t go into the installation process, as it was smooth and un-eventful.

Swapping database technologies made a noticeable difference in the stability of the server. Cherokee was able to fulfill a greater number of requests (~6000 / 10000) which is twice as much as the SQLite backed test. It was also observed that the PostgreSQL server caused the VPS to consume more RAM at idle. It went from 85MB to 125MB.

uWSGI

At the very top of the Django tutorial within the Cherokee documentation, It is said that there is no Wizard for Django with uWSGI yet, but it is a more modern way of serving pages than the default FastCGI.

I was able to find some examples of how to configure uWSGI to work in Cherokee online on the official uWSGI examples page. After creating the files that they suggest, and running the uWSGI Wizard to set up the new Virtual Server, I was quickly on my way.

Running Apache Bench against the uWSGI server produced VERY different results than the FastCGI server. The main difference: — it would not crash –, no matter how many concurrent requests were thrown at it. It was also observed that the more the concurrency of the requests increased, the more efficient the server appeared to run. For example: 25 concurrent would use an additional 50MB of RAM, but 100 concurrent requests only appears to use an additional 60MB RAM. The raw requests per second were higher as well. The uWSGI server was consistently serving the same page between 45 to 65 requests per second. Using a BRONZE VPS. It looks like we’re on the right track now.

A final observation about this setup was that running “$ ps aux” showed that only a single uwsgi process was spawning and handling this traffic. So the next step is to look at jumping that number up, since we have 6 cores to work with, even on the Bronze VPS package.

Concurrency # 1

The first steps at setting up the concurrent uWSGI requests was to use the Cherokee clone tool to make 3 copies of the uWSGI serving process. One of the fancy features in Cherokee is that it has a built in load balancer to bounce requests between processes. Using a “Round Robin” tactic, I registered the 3 additional uWSGI processes and ran the benchmarks again.

Before I mention the findings of this Concurrency strategy, It’s worth noting that the idle RAM usage for the server jumped from 72MB to 120MB with the new uWSGI processes started up, prepared to receive requests.

After the benchmarks ran, we saw another increase in performance. This time the average requests per second was between 62 and 70. Even more impressive for the lowly 256MB RAM the server was running on and that the RAM usage for the server did not surpass 140MB even under full load. Impressive.

Concurrency # 2

I dug into the uWSGI documentation some more and found some additional flags to put in the uwsgi.xml file. The option enabled was:

Processes = 4 – spawn [n] uwsgi worker processes

Using this uWSGI configuration, It made the local round-robin setup redundant. I updated the uwsgi.xml config file with the new values, and removed the additional uWSGI processes in the Round Robin pool. Feeling good about this setup, I worked tightly with Dynamic Hosting to test the new configuration at all available VPS levels to come up with the final performance report above.

Conclusion


This was an interesting venture into deploying a Django site. There were a lot of takeaways and learnings that will come into use whether we choose to use Cherokee in the future or not.

The List

Cherokee Web Server will:

  • Scale forever given a good configuration
  • Will scale well both horizontally (VPS Upgrade) and vertically (Multiple App Servers) and is easy to configure both.
  • Speed up Deployments
  • Enable soft server restarts
  • Have lower hardware requirements (CPU Speed makes the biggest difference)
  • Make life easier with the Cherokee-Admin Web UI

Cherokee Web Server won’t:

  • scale out of the box with Django Wizard (FastCGI)
  • Serve .htaccess files (apache only)
  • Have lots of external support (new-ish project)
  • Give you up
  • Let you down
  • Run around and desert you

Additional Thoughts

FastCGI is a bad idea, and we should not use it. Once uWSGI is enabled, the biggest impact on server performance was by increasing the CPU speed available. This would make using a service like RackSpace a great option where you can dynamically set the resources available for your server.

CPU is important in a modern web server, Even as VPS size increased, Cherokee held it’s memory usage below 200MB even if 1.5GB was available in Gold. This leaves a lot of room for in-memory caching on the Application server using tools such as Redis or Memcache.

Static File Servers Rock, Apache should never serve a static file, Cherokee doesn’t even allow you to serve the files with the same handler that serves the application. Cherokee ships with it’s own super fast I/O caching file server to serve up all the application resources. If you remember, the Gold VPS got 5500 req/sec using this strategy.

In the end, I fully support switching to Cherokee for deploying. It would even make sense to do all development on the same Gold VPS / Dedicated box and set up multiple Virtual Servers for each app to be hosted with such a small footprint that the server imposes. This project also convinced Dynamic Hosting to add Cherokee to their list of suggested technologies to their clients when purchasing new servers and VPS containers, as it gives limited hardware resources the ability to serve pages like a dedicated server.

« »