pigz parallel gzip compression revisited

Today, revisiting pigz – a parallel gzip compression alternative to gzip. I had a client who’s MySQL sql database backup came in at 9.9 gigabytes in size and he was running a level 9 gzip compression on the file which took ~48 minutes to run on his WHM/Cpanel powered dual Intel Xeon E5520, 12GB memory based server.

Cpanel folks have a feature request thread on their forums to replace gzip with pigz to speed up Cpanel backup process. It hasn’t been implemented yet.

I installed pigz on client’s server and recommended he switch to using pigz at level 4 compression instead of gzip and here is the results below. Very dramatic differences in compression times, pigz was 33.3x times faster than gzip !

backup_070712.sql = 9.9GB sized

gzip -9 compress backup took 48 mins and 28.85 seconds

gzip -9 backup_070712.sql
real: 2909.05s cpu: 98% maxmem: 2640 KB cswaits: 3785

gzip -d uncompress took 171.08 seconds

gzip -d backup_070712.sql.gz
real: 171.08s cpu: 62% maxmem: 2224 KB cswaits: 1053

pigz took 87.23 seconds compared to 2,909.05 seconds which is 33.3x times faster to compress your database backup. If you did daily backups, you would save 2,821.82 seconds per day = over 30 days save you 1,410.91 mins or 23.51 hrs per month in backup time !

pigz -4 backup_070712.sql
real: 87.23s cpu: 694% maxmem: 52288 KB cswaits: 554067

ls -lk
total 2535264
-rw-r--r--  1 root root 2532772 Jul  7 02:37 backup_070712.sql.gz

pigz -d uncompress took 148.42 seconds compared to gzip’s 171.08 seconds so 13.24% faster

pigz -d backup_070712.sql.gz
real: 148.42s cpu: 56% maxmem: 3072 KB cswaits: 752334

Combine pigz multi-threaded parallel compression speed up with mydumper multi-threaded MySQL backup tool (3x to 10x times faster than mysqldump), and you’ll have sped up MySQL database backup and compression times ALOT !