Litespeed Cache vs Varnish Cache – some thoughts

My main work pcs’ hard drive is failing with bad sectors, so while i run Spinrite (13+ hrs process) to try and fix them enough to be able to clone my hdd, I have some free time on my hands and thought I’d sit down and write up my experiences so far with two lovely web acceleration technologies – Litespeedtech.com web server’s Litespeed Cache and Varnish Cache.

 

Background

I work for vBulletin.com in a non-developer role which basically has me assisting customers optimising and tuning their server(s) to run vBulletin forums as best they can or advising them on hardware configurations to best serve their needs.  I also do this on a private consulting basis outside of vBulletin.  So any technology advances which can boost performance are of great interest to me. A big one at the moment is the transition from MySQL.com provided MySQL 5.x/5.1.x to performance improving drop-in replacement forks such as MariaDB 5.2.x/5.3.x MySQL and Percona 5.1.x/5.5.x MySQL. Personally, leaning towards MariaDB as it already includes the Percona XtraDB/Innodb plugin for Innodb performance improvements with the addition of the MySQL core changes and performance improvements which help folks using MyISAM tables still.

 

Varnish and I

It all started on October 20th, 2010, when a fellow vBulletin colleague , Mike Anders posted a vBulletin blog article on vB boost plugin which utilises Varnish to cache php end to boost performance by reducing the MySQL database end calls. Varnish helped with caching static files as well, but the focus was on alleviating the MySQL server load which basically is the bottleneck when trying to scale vBulletin for very large forums.

This would be my very first introduction into the world of Varnish  caching – prior to Mike’s article, I have never heard of Varnish!  More info on Varnish Cache can be found on their site and their wiki.  Unfortunately, I didn’t understand Varnish much so left the thoughts of testing out Varnish to linger for a few months.

 

Litespeed web server & Litespeed Cache

Litespeed web server from litespeedtech.com is a replacement web server for Apache which offers several folds better scalability than standalone Apache web servers. Litespeed web server works with various  web hosting control panels like WHM/Cpanel, Plesk, Directadmin as well as working standalone without any control panels.  More info can be found on litespeedtech.com site, wiki or their blog. Update: Litespeed web server v4.0.20 adds new response header for improved Litespeed caching.

 

2004

I first heard of Litespeed web server nearly 7yrs ago in 2004 through a vB client I consulted for  who switched from Apache to Litespeed web server.  Back then they were only barely coping handling a vBulletin forum with 6-8 million posts and 2,000-5,000+ vB users online when they were on Apache web server. In 2011, this vB client’s forum now has a staggering 38 million posts, 2.4 million threads, 333K registered members and last I heard 10,000+ vB users online at a time and yes they are still using Litespeed web server as an Apache web server replacement.

 

2009

It would be 5yrs later in 2009, that I would have my first hands on experience with Litespeed web server, when another private vB client I consulted for had to get off Apache web server to better scale their vBulletin + vBdrupal web site. Their site receives around 200,000+ unique visitors per day and about 65 million pageviews per month with around 2,000-8,000+ vB users online at a time.  New hardware technology improvements in the way of Intel Xeon Nehalem based processors + Litespeed web server and Memcached server caching for vB and vBdrupal elements made light work of serving the vB clients traffic needs.

Ever since then, I have opened my eyes to the benefits in replacing Apache web server with better scaling alternative web servers such as Nginx and Litespeed web server.  Nginx while great as a static file server, when it comes to php and rewrite rules etc it’s okay if you managing a few hosted sites, but once you’re managing several dozen or hundreds of sites each with different php/rewrite configurations, then the amount of extra manual configuration work can add up as I found for myself.

So these days I lean towards Litespeed web server as the big difference is it’s basically a drop replacement for Apache web server, so all htaccess, rewrite rules and even apache httpd.conf/virtualhost configs work.  For sites with many hosted sites or for administrators who want to save time, the licensing costs for Litespeed web server easily make up for the amount of time saved in managing and custom configuring Nginx config files etc to make them work. After all, time is money! And admins managing many sites, I hope for your sake, the extra time you  spent managing and configuring Nginx is worth more than the extra cost of Litespeed licensing!

 

2011 – Litespeed Cache

Unfortunately, it wasn’t until 2011 that Litespeed offered it’s caching feature. A few months had past since being exposed to Varnish caching, when a vB customer’s questions were answered by Litespeed on January 12, 2011 that there was indeed a Litespeed cache feature available to replicate Mike’s vB Boost Varnish caching plugin for vBulletin.

This was the news I was waiting for, my favourite web server, Litespeed web server now offering it’s own cache feature which can further boost vBulletin performance!  I had to be fair, so in order to test Litespeed cache’s performance, I had to sit down and figure out how to use Varnish as well to properly compare them both – the focus being on vBulletin performance (as well as wordpress performance as I personally use both for my own private sites).

Note Litespeed cache feature is only available in 2+ cpu licenses. VPS and 1 cpu licenses don’t support Litespeed cache.

 

2011 – Litespeed Cache vs Apache/Varnish Cache tests

Mid January to February, 2011 involved me setting up a local Virtualbox virtualized set of CentOS 5.5 64bit based servers for testing Apache, Nginx and Litespeed web servers with PHP 5.3.5 (mod_php, php-fpm, and LSAPI PHP respectively) with MariaDB 5.2.4 MySQL server Compared Varnish cache with Litespeed caching of which summary of the end result was posted on my vBulletin.com blog.

Through this testing experience, I gained a greater appreciation and understanding for both Litespeed and Varnish caching.  For Varnish caching the VCL flexibility reminds me of Nginx config wise. Through the VCL configurations there so much more you can do than just caching – it’s really growing on me. Varnish caching definitely boosts the performance of any web server it sits in front of but you’ll have to tweak the default VCL configuration in order to do so.

Basically, out of the box default Varnish VCL won’t cache any files which contain cookies of any sort, so even your site’s static files which have cookies tagged on won’t be cached. vBulletin, wordpress, and drupal all have their own sets of cookies, Google Analytics, stats counters, twitter, facebook widgets also have their own cookies added to the mix on the site’s domain. So end result is site and server performance and Varnish cache hit rate will be very low as most of the requests pass onto the backend server like Apache listening on a port other than port 80 which Varnish sits on.

There are various VCL configuration parameters to match site and file elements and strip cookies for caching by Varnish – the VCL is where all the magic happens in making Varnish cache the right content. Varnish VCL can also be configured to cache cookies tagged content which would end up in a hash forming a per user cache, meaning that cahe hit rate would be much lower and resources used will be higher as a cache set is kept for each user.

 

Mike Anders vB Boost Varnish plugin

Mike’s vB Boost Varnish plugin’s aim was to get around vBulletin cookies by introducing an additional cookie to differentiate through VCL logic, a logged in vB member versus a logged out or guest visitor. The aim is to strip out cookies for guest visitors only, so guest visitors hit the Varnish cache reducing the load on php files served which in return reduces calls to the MySQL server end. Thus improving vBulletin forum performance and scalability.  Members would still hit php and the MySQL server end so overalll performance would rely on inherent properties of the web server employed be it Apache, Nginx or Litespeed web server. Varnish caching would also cache static files for guests and members as VCL logic would configure such static files to be totally stripped of cookies, so Varnish can cache them.

With Varnish caching, you would think there would be dangers in caching content you don’t want cached. Out of the box Varnish has a default set of VCL rules and logic which would prevent some of the situations. You can set additional custom VCL rules yourself for your particular web application to excluding certain files and directories from being cached. So in vBulletin example, admincp, modcp, install.php, would be some directories and files excluded from caching. I also added upgrade.php to my own VCL rules for exclusion.

 

Litespeed web server + Litespeed cache

First, I’d like to thank George and Gary from Litespeedtech.com for all there assistance in getting me up to speed with Litespeed caching and throughout my tests so far.

Compared to Varnish which has been around since 2006, Litespeed cache is still fairly new and still being developed and improved. It’s current focus is on caching php pages. Why not cache static files you ask ? Well Litespeed web server already has a built in static file caching feature  which is configurable in the Litespeed admin console.  You can see the results of such static file cache feature in my Litespeed non-cache static file benchmark results here. The static file cache feature works out of the box and doesn’t need any user intervention to reap it’s benefits.

Litespeed cache feature for php files in vBulletin is similar to Varnish in that you can configure the cache feature to tell it what to cache by differentiating logged in user cookie versus guest visitor cookies. How this is done is by taking Mike’s vB Boost xml plugin and installing it like you would with vB Boost Varnish on your vB forum – this xml plugin adds an additional cookie to allow differentiation between a member logged in versus a guest. Then using rewrite logic you can tell Litespeed cache to cache all or certain php pages which don’t have the member logged in cookie which was added. So you end up with Litespeed cache only caching php pages for guest visitors, thus reducing php related load and MySQL server connection load for guest visitor traffic. Remember, though Litespeed web server itself has static file cache so static files are cached for both members logged as well as for guests. Just like Varnish’s VCL logic to stripe static files of all cookies so they can be served to both members and guests.

 

Varnish VCL vs Litespeed Cache rewrite rules

While Varnish cache has it’s VCL to define all the rules as to what is or isn’t cached, Litespeed cache currently does this through rewrite rules which can be implemented via .htaccess, httpd.conf or via Litespeed’s own admin console and native virtual host section’s rewrite area.   This is not to say in future Litespeed cache developments that this would change.

In the early stages of development this work through rewrite logic can potentially be problematic as potential conflicts with other script’s htaccess/rewrite rules can occur, i.e. with vBSEO for vBulletin. Litespeed has their own support forum section for Litespeed cache discussions including suggested workarounds for vBSEO compatibility with Litespeed cache feature.

So if you need to boost your vBulletin specific performance now and don’t use vBSEO, Litespeed web server + Litespeed cache is good, you get php cache boost along with the Litespeed web server’s already inbuilt static file caching. – thus boost scalability and performance for both members logged and as well as guests. This differs from Apache + Varnish configuration as Varnish would only boost guest visitor based performance, but members logged in for large vB forums won’t scale much due to the bottlenecked and resource hogging nature of Apache web server.

If you use vBSEO, you can still employ Litespeed web server + Varnish caching as well.  While Apache + Varnish needs mod_rpaf installed to take care of passing on real ip address from Varnish to Apache backend and the applications, Litespeed web server’s admin console has a setting to enable such equivalent feature.

Update: Looks like Litespeed cache feature finally works with vBSEO see these 2 threads here and here.

 

Unixy.net’s WHM / Cpanel Varnish Plugin

As mentioned before Litespeed web server comes as a plugin directly for WHM/Cpanel  so it works well so next chapter in my adventures, was figuring out how to get Varnish caching working with my favourite control panel which I have used for past 10yrs – WHM/Cpanel from cpanel.net.  Through the Virtualbox virtualized testing I did earlier I had developed my own custom setup of bash shell install and configuration scripts for such things as Apache, PHP, MariaDB MySQL and Varnish but all these were for standalone CentOS 5.5 64bit servers without any control panels.

WHM/Cpanel was too complex, but luckily Unixy.net’s Joe posted on webhostingtalk.com his new WHM/Cpanel Varnish plugin which basically compiles and installs Varnish v2.1.4 and mod_rpaf, loads and setups up their own custom set of VCL rules and logic, grabs all WHM/Cpanel hosted accounts ip addresses and sets them up in mod_rpaf and then switches all hosted accounts listening ports for apache to port 82 which Varnish sits and listens on front end port 80.  There’s alot other background processes involved. Unixy.net WHM/Cpanel Varnish plugin script has a small lease licensing fee but right now is offering a free 14 day trial. More info here.

 

Varnish default cache vs Unixy.net Varnish cache

Unixy.net took a different approach to Varnish caching as WHM/Cpanel users are predominantly web hosts with many shared or VPS hosting accounts, so per site/application configuration of VCL rules and logic would be a daunting and time consuming task.  So Unixy.net opted to change Varnish’s default VCL rules from don’t cache files with cookies, to hash and cache files with cookies – essentially turning Varnish into a per user/visitor cache.  This dramatically reduces the maximum potential cache hits ratio and adds a bit more resource usage but is countered by the default Unixy.net’s VCL rule set only setting a TTL of 20 seconds for Varnish cache, so a user whether logged in member or guest with cookies would only see a cached file for their session for 20 seconds but is adjustable via WHM Varnish plugin advanced configuration page.

But VCL is setup so static files like js, css and images all get cookies stripped and cached with additional TTL for browser caching which defaults to 120 minutes and you can adjust this directly in the default.vcl file. Certain cookies for authorisation and file such as install.php and cron.php are excluded by default in VCL.

I have made a few suggestions to Unixy.net to include upgrade.php as a default excluded file as well as improving the WHM Varnish plugin statistics page with additional stats available which make diagnosing Varnish performance much easier.  Another useful suggestion which made it’s way into v1.4.3b was being able to turn Varnish plugin on and off switch between plain default Apache and Varnish reverse cache proxy to Apache.  A few other features I’d like to see added include, ability via WHM Varnish plugin control panel:

1. Advance Varnish Configuration page to adjust various additional Varnish startup options such as thread_pools_min/max, thread_pools, listen_depth, thread_pool_add_delay, session_linger.
2. Be able to directly manipulate the Varnish VCL files – create, copy, backup and delete them.
3. Be able to start up a 2nd private Varnish server instance using -n named instance command on a different port and specify a different VCL startup file for testing additional custom VCL rule sets in a safe environment away from live running Varnish server
4. Additional statistics via varnishtop and varnishlog to display the top 5 or 10 most visited hosts, most requested urls and most frequent urls hitting the backend etc. Such stats would be useful for a web administrator in setting up opt-out url exclusion list for urls you don’t want cached etc.

 

Unixy.net WHM/Cpanel Varnish plugin results

As I don’t have a WHM/Cpanel license to test locally on virtualized server, I ended up tasking the risk of testing Unixy.net’s WHM/Cpanel Varnish plugin v1.4.2b / v1.4.3b on my own personal  VPS server at Wiredtree.com which runs WHM/Cpanel and CentOS 5.x 32bit. The end result was much snappier loading pages on my vBulletin and wordpress sites – I have 2 vB forums and 2 wordpress based sites which are live and active.  But cache hit rate for per user caching config is much lower than what I tested with on virtualized server running Varnish in it’s native default VCL state.  The ratio of Cache hit rate to requests served was between 0.10 to 0.25 average with peak of around 0.28 to 0.30 cache hit rate which means 70-80+ % of requests still hit the Apache backend server. My cache hit rate ratio might be lower than normal as 60% of my static files are offloaded to CDN service so aren’t handled by Varnish anyway. Luckily, my server isn’t high traffic so Apache backend can handle it still.  CPU peak and average loads are down with Unixy.net’s Varnish plugin installed – average and peak cpu loads fell from 1.8-2.2 average and 4.0-8.0 peaks to 0.50-1.8 average to 3.5-5.5 peaks. Some of my peaks due to other background processes and cronjobs and VPS environment I/O loads.

While others on webhostingtalk.com who are trialing or testing Unixy.net’s WHM/Cpanel Varnish plugin are reporting dramatic drops in cpu loads compared to straight Apache and claiming to switch from Litespeed web server to Varnish+Apache setup due to cost of Litespeed web server licensing, I am not sure how well that would fair for very high concurrency traffic sites or busy forums as you still have 70-80+ % of requests hitting the Apache backend with Unixy.net’s per user cache VCL rules setup.

 

Apache as Varnish backend, still a bottleneck ?

Apache being a resource hogging and scaling bottleneck it is, will still be the weak spot in an Apache+Varnish cache setup whether it be Varnish default VCL or Unixy Varnish VCL per user cache.  For software such as vBulletin in situations where high concurrency member logged in activity occurs, you are always at the mercy of the backend web server’s scalability and performance. This is where I think Litespeed web server as the backend paired with either Litespeed Cache or Varnish caching would excel best for such applications as vBulletin.

But in a WHM/Cpanel web hosting environment particularly shared/VPS hosting which Unixy.net is targeting, it has it’s benefits. In such shared/VPS hosting environments it would virtually be impossible to custom configure Varnish VCL rule sets for each and every application installed by each shared/VPS account on the server.  So a per user cache implementation makes for a nice work around.  Even a low 0.10 to 0.20 cache hit ratio would mean you’re getting a  minimum 10-20% boost and end users visiting such Varnish cached sites even with per user caching will experience very snappy loading pages. Note, outside of WHM/Cpanel environment, in a plain Varnish install you can configure per user caching via VCL as well without such plugins.

I’d like to thank Joe from Unixy.net for listening to and implementing some of my suggestions in his Varnish plugin script and for his continued assistance as I learn the ropes with his Varnish plugin for WHM/Cpanel.

 

Summary

While still very new to both Litespeed Cache and Varnish caching, I am very excited about the performance boosting possibilities they both bring to vBulletin and other web applications and scripts.  Things look even brighter once you throw into the mix other performance and scaling enhancing technologies like MariaDB/Percona MySQL, Memcached, Sphinx search, CDN offloading of static files, and PHP opcode caching from Xcache, Eaccelerator and APC .

Time permitting, I write up my future experiences with these awesome web accelerating technologies. If visitors have also tried Litespeed Cache and/or Varnish Cache for their web applications, please feel free to reply to this post with url linking to your experiences.  I’d be interested to read up on how other folks are using these technologies to improve performance and scale their web applications.

I’ll end this article with link and screenshot below to my results with Litespeed Cache, Litespeed web server + Varnish and Apache + Varnish comparisons.

Now back to getting some bad sectors fixed on my OS hard drive – enough to be able to clone and copy my disk data.  Fingers crossed!