gziptest.sh – parallel compression benchmark tests: gzip vs bzip2 vs pigz vs pbzip2 vs lzip vs plzip
A Cpanel feature request thread came up for inclusion for pigz a parallel multi-threaded implementation of gzip which dramatically speeds up compression. Update: Part 2 compression benchmarks have been posted here.
Pigz definitely does speed up compression times, I implemented parallel multi-threaded versions of gzip and bzip2 – pigz and pbzip2 in my mysqlmybackup.sh script I am writing so have seen first hand the speed up benefits.
I cleaned up my gziptest.sh script and decided to run some linux compression benchmark tests comparing the following compression methods:
- gzip v1.3.5 – http://www.gzip.org/
- bzip2 v1.03 – http://bzip.org/
- pigz v2.1.6 – multi-threaded version of gzip http://www.zlib.net/pigz
- pbzip2 v1.1.6 – multi-threaded version of bzip2 http://compression.ca/pbzip2
- lzip v1.1.3 rc1 – based on LZMA compression algorithm http://www.nongnu.org/lzip/lzip.html
- plzip v0.80 rc1 – multi-threaded version of lzip http://www.nongnu.org/lzip/plzip.html
Test Configuration:
VirtualBox CentOS server instance might not produce the most accurate timer stats for the tests, but it should give you an idea of speed of compression, cpu and memory usage for various compression methods outlined above.
- CentOS 5.7 64bit VirtualBOX
- Xeon W3540 @3.5Ghz – 4 core allocation (2 physical + 2 virtual)
- 2GB DDR3 memory allocated
- 20GB EXT3 VDI disk
- Original uncompressed file: MySQL backup sql file sized at 605,530 KB at /root/sqlbackup/vb387.sql
Findings:
From the below tests, you can see how much faster pigz is than gzip.
- For speed seems pigz is best up to 2x times faster than gzip. Probably limited by the fact that the test server has only 2 physical cpu cores. More physical cpu cores you’d have, the faster the speed up for pigz vs gzip. Tweaking the block size for compression can reduce the max memory usage while still retaining similar compression ratios and fast compression times. Updated: Check test #4 below for details or jump straight to results.
- For compression and smaller compressed file sizes, lzip seems to be best being able to compress MySQL sql backup file to 7% of original file size at level 3 compression. That’s nearly 1/3rd of the compressed size for gzip with 22% at level 3 and 19% at levels 6 to 9. But lzip takes much longer to run and consumes up to 40x times more memory.
- For speed + compression level, plzip – parallel lzip implementation seems best but memory consumption is out of this world at 188x to 270x times more memory used than gzip!
gziptest.sh options
[root@centos57a ~]# ./gziptest.sh install ./gziptest.sh at /usr/local/src chmod +x ./gziptest.sh ./gziptest.sh help ./gziptest.sh pigzinstall ./gziptest.sh pbzip2install ./gziptest.sh lzipinstall ./gziptest.sh plzipinstall ./gziptest.sh leveltest ./gziptest.sh threadtest ./gziptest.sh levelgzip ./gziptest.sh levelbzip2 ./gziptest.sh levelpigz ./gziptest.sh levelpbzip2 ./gziptest.sh levellzip ./gziptest.sh threadpigz ./gziptest.sh threadpbzip2 ./gziptest.sh threadplzip ./gziptest.sh helpfiles ./gziptest.sh dd |
Test 1:
To give you a base line, here’s gzip compression level 1 to 9 tests showing the time it took to complete (real), the percentage of CPU utilisation, max memory usage as well as the level of compression as a percentage of the original uncompressed file. Gzip uniformly consumes 2,624KB to 2,704KB of memory while compressing utilising between 60-95% CPU. Compressed file sizes start at 24% of original file size at level 1 and reduce to 19% from levels 6 to 9.
[root@centos57a ~]# ./gziptest.sh levelgzip ------------------------------------------------------------ gziptest.sh 0.07 by George Liu (eva2000) http://vbtechsupport.com ------------------------------------------------------------ ------------------------------------------------ gzip -1 /root/sqlbackup/vb387.sql real: 15.47 seconds cpu: 60% max mem: 2704KB compression level: 1 compress: 24% 147230/605530 KB ------------------------------------------------ gzip -2 /root/sqlbackup/vb387.sql real: 13.06 seconds cpu: 75% max mem: 2704KB compression level: 2 compress: 23% 142356/605530 KB ------------------------------------------------ gzip -3 /root/sqlbackup/vb387.sql real: 12.71 seconds cpu: 85% max mem: 2704KB compression level: 3 compress: 22% 138441/605530 KB ------------------------------------------------ gzip -4 /root/sqlbackup/vb387.sql real: 15.52 seconds cpu: 81% max mem: 2688KB compression level: 4 compress: 21% 127571/605530 KB ------------------------------------------------ gzip -5 /root/sqlbackup/vb387.sql real: 19.54 seconds cpu: 80% max mem: 2656KB compression level: 5 compress: 20% 122217/605530 KB ------------------------------------------------ gzip -6 /root/sqlbackup/vb387.sql real: 23.54 seconds cpu: 85% max mem: 2624KB compression level: 6 compress: 19% 120392/605530 KB ------------------------------------------------ gzip -7 /root/sqlbackup/vb387.sql real: 32.24 seconds cpu: 72% max mem: 2640KB compression level: 7 compress: 19% 119340/605530 KB ------------------------------------------------ gzip -8 /root/sqlbackup/vb387.sql real: 43.68 seconds cpu: 96% max mem: 2624KB compression level: 8 compress: 19% 118054/605530 KB ------------------------------------------------ gzip -9 /root/sqlbackup/vb387.sql real: 53.93 seconds cpu: 93% max mem: 2640KB compression level: 9 compress: 19% 117995/605530 KB ------------------------------------------------ gzip -d /root/sqlbackup/vb387.sql.gz real: 4.19 seconds cpu: 99% max mem: 2256KB |
Test 2:
Compression level 1 to 3 tests + decompression test comparising gzip vs bzip2 vs pigz vs pbzip2 vs lzip vs plzip. By default, pigz, pbzip2 and plzip will try to detect number of cpu threads the system has and run that many threads for compression tests if no option is specified for number of threads. In this case it will be 4 cpu threads with exception of pigz seems it is more likely detecting physical cpu cores rather than the additional virtual cpu threads from Intel Hyperthreading or somewhere between 2 and 3 cpu threads.
You’ll notice the parallel versions use much more memory and cpu time. Compared to gzip, pigz uses around 5-6x times more memory, pbzip2 uses up to 40x times more memory, and plzip uses 188x to 270x times more memory but much smaller file sizes. Compression level wise, gzip and pigz ended up last with 22-24% compared to bzip2 / pbzip2 at 14-17% and lzip at 7-9% and plzip at 8-11% of original file size. Not sure why but pigz didn’t fully utilise all cpu threads hitting around 211% CPU utilisation at level 3.
Decompression times were pretty close for all but bzip2 which was 4x times slower. Memory consumption wise, pbzip2 and plzip consumed the most memory 443,888KB and 405,920KB respectively which was much higher than gzip’s 2,240KB, bzip2’s 6,880KB and pigz’s 2,976KB. While plzip was the only one to close to utilising all cpu threads at 369% CPU utilisation for decompression operations and with pbzip2 following at 278% CPU utilisation. While pigz decompression managed only 127% and gzip and bzip2 didn’t go past 99% CPU utilisation.
[root@centos57a ~]# ./gziptest.sh leveltest ------------------------------------------------------------ gziptest.sh 0.07 by George Liu (eva2000) http://vbtechsupport.com ------------------------------------------------------------ ------------------------------------------------ gzip -1 /root/sqlbackup/vb387.sql real: 22.19 seconds cpu: 43% max mem: 2688KB compression level: 1 compress: 24% 147230/605530 KB ------------------------------------------------ gzip -2 /root/sqlbackup/vb387.sql real: 12.99 seconds cpu: 76% max mem: 2704KB compression level: 2 compress: 23% 142356/605530 KB ------------------------------------------------ gzip -3 /root/sqlbackup/vb387.sql real: 14.45 seconds cpu: 76% max mem: 2688KB compression level: 3 compress: 22% 138441/605530 KB ------------------------------------------------ gzip -d /root/sqlbackup/vb387.sql.gz real: 4.39 seconds cpu: 99% max mem: 2240KB ------------------------------------------------ bzip2 -1 /root/sqlbackup/vb387.sql real: 84.90 seconds cpu: 97% max mem: 6400KB compression level: 1 compress: 17% 106378/605530 KB ------------------------------------------------ bzip2 -2 /root/sqlbackup/vb387.sql real: 93.05 seconds cpu: 95% max mem: 9552KB compression level: 2 compress: 15% 92240/605530 KB ------------------------------------------------ bzip2 -3 /root/sqlbackup/vb387.sql real: 102.50 seconds cpu: 92% max mem: 12656KB compression level: 3 compress: 14% 85639/605530 KB ------------------------------------------------ bzip2 -d /root/sqlbackup/vb387.sql.bz2 real: 17.78 seconds cpu: 99% max mem: 6880KB ------------------------------------------------ pigz -1 /root/sqlbackup/vb387.sql real: 13.87 seconds cpu: 85% max mem: 14320KB compression level: 1 compress: 24% 146626/605530 KB ------------------------------------------------ pigz -2 /root/sqlbackup/vb387.sql real: 6.27 seconds cpu: 197% max mem: 14064KB compression level: 2 compress: 23% 141627/605530 KB ------------------------------------------------ pigz -3 /root/sqlbackup/vb387.sql real: 6.39 seconds cpu: 211% max mem: 14256KB compression level: 3 compress: 22% 137511/605530 KB ------------------------------------------------ pigz -d /root/sqlbackup/vb387.sql.gz real: 3.02 seconds cpu: 127% max mem: 2976KB ------------------------------------------------ pbzip2 -1 /root/sqlbackup/vb387.sql real: 24.12 seconds cpu: 363% max mem: 93456KB compression level: 1 compress: 17% 106356/605530 KB ------------------------------------------------ pbzip2 -2 /root/sqlbackup/vb387.sql real: 24.64 seconds cpu: 387% max mem: 96064KB compression level: 2 compress: 15% 93768/605530 KB ------------------------------------------------ pbzip2 -3 /root/sqlbackup/vb387.sql real: 27.84 seconds cpu: 369% max mem: 107472KB compression level: 3 compress: 14% 85683/605530 KB ------------------------------------------------ pbzip2 -d /root/sqlbackup/vb387.sql.bz2 real: 6.77 seconds cpu: 278% max mem: 443888KB ------------------------------------------------ lzip -1 /root/sqlbackup/vb387.sql real: 55.51 seconds cpu: 94% max mem: 67456KB compression level: 1 compress: 9% 57413/605530 KB ------------------------------------------------ lzip -2 /root/sqlbackup/vb387.sql real: 66.81 seconds cpu: 95% max mem: 87952KB compression level: 2 compress: 8% 50561/605530 KB ------------------------------------------------ lzip -3 /root/sqlbackup/vb387.sql real: 84.24 seconds cpu: 95% max mem: 108432KB compression level: 3 compress: 7% 45938/605530 KB ------------------------------------------------ lzip -d /root/sqlbackup/vb387.sql.lz real: 6.25 seconds cpu: 99% max mem: 12272KB ------------------------------------------------ plzip -1 /root/sqlbackup/vb387.sql real: 16.15 seconds cpu: 337% max mem: 496992KB compression level: 1 compress: 11% 66885/605530 KB ------------------------------------------------ plzip -2 /root/sqlbackup/vb387.sql real: 18.16 seconds cpu: 360% max mem: 593776KB compression level: 2 compress: 9% 58445/605530 KB ------------------------------------------------ plzip -3 /root/sqlbackup/vb387.sql real: 22.64 seconds cpu: 375% max mem: 712720KB compression level: 3 compress: 8% 52533/605530 KB ------------------------------------------------ plzip -d /root/sqlbackup/vb387.sql.lz real: 2.06 seconds cpu: 369% max mem: 405920KB |
