A quite efficient compression algorithm: Blosc

  • [アーティクル]

Recently I just learnt about a new compression algorithm Blosc (https://www.blosc.org/trac). It has amazing performance, but with poor compression ratio (by design).

I have done a performance comparison between the Blosc and the .Net built in Deflate compression algorithm.

Here is my PC environment:

  • CPU: Intel Xeon CPU E5-1620 3.6 GHz
  • RAM: 16.0 GB

For Blosc, I use the following settings:

  • Number of threads: 1

    • Set as single thread for fair comparison. Using multi-threads will improve the throughput a lot.

  • Compression level: 9

    • It is the desired compression level and must be a number between 0 (no
      compression) and 9 (maximum compression).

  • Whether do shuffle: No

  • Type size: 8

 

DataSet

Blosc  

Compression

Throughput (MB/S)  

Compression  

Ratio

Deflate  

Compression

Throughput (MB/S)  

Compression

 Ratio

[advwks_cust.dat]

 

220.3

2.8

 

33.7

4.4

[advwks_fact.dat]

 

451.1

4.9

 

58.4

14.5

[experian_fact.dat]

 

285.0

3.0

 

26.7

6.2

[jitb_fact.dat]

 

290.7

2.2

 

16.9

3.9

[mssales_fact.dat]

 

501.9

6.0

 

41.7

10.4

[mssales_prod.dat]

 

530.0

5.8

 

50.5

10.1

[skype_fact.dat]

 

213.0

2.1

 

15.9

4.0

[synthetic_int.dat]

 

359.4

3.9

 

15.7

5.7

Comments

  • Anonymous
    June 05, 2014
    An interesting property of the Blosc algorithm is that it supports a built-in multithread compression. However, its exposed compression API is not thread safe, which is quite different from the existing other compression algorithms (there is no built-in multithread, but the API is thread safe). This may make it harder to integrate with your existing framework if using multi-thread compression.

  • Anonymous
    December 11, 2014
    Adding multithreading to compression libraries is a trivial task. Blosc is no more than Shuffle+Compression. You can add shuffle+Multithreading to any compression library.