gziptest.sh – parallel compression benchmark tests: gzip vs bzip2 vs pigz vs pbzip2 vs lzip vs plzip
Test 4:
Seeing as pigz is the leading candidate in terms of compression speed and decent memory usage, I thought I’d play with the block size parameter
Here’s full options for pigz which is basically same as for gzip.
[root@centos57a ~]# ./gziptest.sh install ./gziptest.sh at /usr/local/src chmod +x ./gziptest.sh ./gziptest.sh help ./gziptest.sh pigzinstall ./gziptest.sh pbzip2install ./gziptest.sh lzipinstall ./gziptest.sh plzipinstall ./gziptest.sh leveltest ./gziptest.sh threadtest ./gziptest.sh levelgzip ./gziptest.sh levelbzip2 ./gziptest.sh levelpigz ./gziptest.sh levelpbzip2 ./gziptest.sh levellzip ./gziptest.sh threadpigz ./gziptest.sh threadpbzip2 ./gziptest.sh threadplzip ./gziptest.sh helpfiles ./gziptest.sh dd |
Comparing pigz default block size of 128KB versus tested minimum allowed block size of 32KB shows a 40% reduction in max memory usage with the lowest block size supported of 32KB. But with 32MB block size, pigz still uses around 3.2x times more memory than standard gzip. Compression ratios and compression speed are fairly similar with 32KB block size – producing 1.4-2.2% smaller compressed files and only being 5.8% slower on average than the default 128KB block size for compression times. However, decompression times were nearly 29% faster with 32KB block size versus default 128KB block size.
However, CPU utilisation was higher with 32KB block size particularly at higher level compression tests. With 32KB block size, pigz had 9.5% higher CPU utilisation for compression and 34% higher CPU utilisation for decompression tests compared to default 128KB block size.
default 128KB block size
[root@centos57a ~]# ./gziptest.sh levelgzip ------------------------------------------------------------ gziptest.sh 0.07 by George Liu (eva2000) http://vbtechsupport.com ------------------------------------------------------------ ------------------------------------------------ gzip -1 /root/sqlbackup/vb387.sql real: 15.47 seconds cpu: 60% max mem: 2704KB compression level: 1 compress: 24% 147230/605530 KB ------------------------------------------------ gzip -2 /root/sqlbackup/vb387.sql real: 13.06 seconds cpu: 75% max mem: 2704KB compression level: 2 compress: 23% 142356/605530 KB ------------------------------------------------ gzip -3 /root/sqlbackup/vb387.sql real: 12.71 seconds cpu: 85% max mem: 2704KB compression level: 3 compress: 22% 138441/605530 KB ------------------------------------------------ gzip -4 /root/sqlbackup/vb387.sql real: 15.52 seconds cpu: 81% max mem: 2688KB compression level: 4 compress: 21% 127571/605530 KB ------------------------------------------------ gzip -5 /root/sqlbackup/vb387.sql real: 19.54 seconds cpu: 80% max mem: 2656KB compression level: 5 compress: 20% 122217/605530 KB ------------------------------------------------ gzip -6 /root/sqlbackup/vb387.sql real: 23.54 seconds cpu: 85% max mem: 2624KB compression level: 6 compress: 19% 120392/605530 KB ------------------------------------------------ gzip -7 /root/sqlbackup/vb387.sql real: 32.24 seconds cpu: 72% max mem: 2640KB compression level: 7 compress: 19% 119340/605530 KB ------------------------------------------------ gzip -8 /root/sqlbackup/vb387.sql real: 43.68 seconds cpu: 96% max mem: 2624KB compression level: 8 compress: 19% 118054/605530 KB ------------------------------------------------ gzip -9 /root/sqlbackup/vb387.sql real: 53.93 seconds cpu: 93% max mem: 2640KB compression level: 9 compress: 19% 117995/605530 KB ------------------------------------------------ gzip -d /root/sqlbackup/vb387.sql.gz real: 4.19 seconds cpu: 99% max mem: 2256KB |
reduced 32KB block size
[root@centos57a ~]# ./gziptest.sh leveltest ------------------------------------------------------------ gziptest.sh 0.07 by George Liu (eva2000) http://vbtechsupport.com ------------------------------------------------------------ ------------------------------------------------ gzip -1 /root/sqlbackup/vb387.sql real: 22.19 seconds cpu: 43% max mem: 2688KB compression level: 1 compress: 24% 147230/605530 KB ------------------------------------------------ gzip -2 /root/sqlbackup/vb387.sql real: 12.99 seconds cpu: 76% max mem: 2704KB compression level: 2 compress: 23% 142356/605530 KB ------------------------------------------------ gzip -3 /root/sqlbackup/vb387.sql real: 14.45 seconds cpu: 76% max mem: 2688KB compression level: 3 compress: 22% 138441/605530 KB ------------------------------------------------ gzip -d /root/sqlbackup/vb387.sql.gz real: 4.39 seconds cpu: 99% max mem: 2240KB ------------------------------------------------ bzip2 -1 /root/sqlbackup/vb387.sql real: 84.90 seconds cpu: 97% max mem: 6400KB compression level: 1 compress: 17% 106378/605530 KB ------------------------------------------------ bzip2 -2 /root/sqlbackup/vb387.sql real: 93.05 seconds cpu: 95% max mem: 9552KB compression level: 2 compress: 15% 92240/605530 KB ------------------------------------------------ bzip2 -3 /root/sqlbackup/vb387.sql real: 102.50 seconds cpu: 92% max mem: 12656KB compression level: 3 compress: 14% 85639/605530 KB ------------------------------------------------ bzip2 -d /root/sqlbackup/vb387.sql.bz2 real: 17.78 seconds cpu: 99% max mem: 6880KB ------------------------------------------------ pigz -1 /root/sqlbackup/vb387.sql real: 13.87 seconds cpu: 85% max mem: 14320KB compression level: 1 compress: 24% 146626/605530 KB ------------------------------------------------ pigz -2 /root/sqlbackup/vb387.sql real: 6.27 seconds cpu: 197% max mem: 14064KB compression level: 2 compress: 23% 141627/605530 KB ------------------------------------------------ pigz -3 /root/sqlbackup/vb387.sql real: 6.39 seconds cpu: 211% max mem: 14256KB compression level: 3 compress: 22% 137511/605530 KB ------------------------------------------------ pigz -d /root/sqlbackup/vb387.sql.gz real: 3.02 seconds cpu: 127% max mem: 2976KB ------------------------------------------------ pbzip2 -1 /root/sqlbackup/vb387.sql real: 24.12 seconds cpu: 363% max mem: 93456KB compression level: 1 compress: 17% 106356/605530 KB ------------------------------------------------ pbzip2 -2 /root/sqlbackup/vb387.sql real: 24.64 seconds cpu: 387% max mem: 96064KB compression level: 2 compress: 15% 93768/605530 KB ------------------------------------------------ pbzip2 -3 /root/sqlbackup/vb387.sql real: 27.84 seconds cpu: 369% max mem: 107472KB compression level: 3 compress: 14% 85683/605530 KB ------------------------------------------------ pbzip2 -d /root/sqlbackup/vb387.sql.bz2 real: 6.77 seconds cpu: 278% max mem: 443888KB ------------------------------------------------ lzip -1 /root/sqlbackup/vb387.sql real: 55.51 seconds cpu: 94% max mem: 67456KB compression level: 1 compress: 9% 57413/605530 KB ------------------------------------------------ lzip -2 /root/sqlbackup/vb387.sql real: 66.81 seconds cpu: 95% max mem: 87952KB compression level: 2 compress: 8% 50561/605530 KB ------------------------------------------------ lzip -3 /root/sqlbackup/vb387.sql real: 84.24 seconds cpu: 95% max mem: 108432KB compression level: 3 compress: 7% 45938/605530 KB ------------------------------------------------ lzip -d /root/sqlbackup/vb387.sql.lz real: 6.25 seconds cpu: 99% max mem: 12272KB ------------------------------------------------ plzip -1 /root/sqlbackup/vb387.sql real: 16.15 seconds cpu: 337% max mem: 496992KB compression level: 1 compress: 11% 66885/605530 KB ------------------------------------------------ plzip -2 /root/sqlbackup/vb387.sql real: 18.16 seconds cpu: 360% max mem: 593776KB compression level: 2 compress: 9% 58445/605530 KB ------------------------------------------------ plzip -3 /root/sqlbackup/vb387.sql real: 22.64 seconds cpu: 375% max mem: 712720KB compression level: 3 compress: 8% 52533/605530 KB ------------------------------------------------ plzip -d /root/sqlbackup/vb387.sql.lz real: 2.06 seconds cpu: 369% max mem: 405920KB |
