bzip3

kosolam

Impressive compression benchmark. Four times smaller than z standard.

sylware

Isn't that XZ?

altairprime

Previously: “Hi, tool author here.” A useful explanation of Burrows-Wheelers transform as used by bzip3: https://news.ycombinator.com/item?id=42902407 “bzip3 is not yet listed on the large text compression benchmark” It is now: https://mattmahoney.net/dc/text.html (2 years ago, 176 comments) https://news.ycombinator.com/item?id=42899713 (4 years ago, 104 comments) https://news.ycombinator.com/item?id=31324439

sedatk

The latest release is a year ago, the last commit is two months ago, and the build is failing. The claim “stronger than bzip2” is strange. What does it even mean? Also, comparing parallel decompression benchmarks with bzip2 instead of pbzip2 seems unfair.

ot

The benchmarks are disingenuous, to the point of looking cherry-picked. The block size for bzip3 is set to 512MB, but the window size for zstd is left to its default (8MB I believe for high levels). So in this corpus, which is made up of all versions of Perl source code concatenated, the window is too small to see all the identical files and just match them. Also corpora made out of very long repetitions are pretty much the best case scenario for BWT-based compressors. If we match the window size of zstd to that of bzip3 we get dramatically different results: % gzcat *.gz | time zstd -T8 -16 | wc -c # baseline 2819113884 zstd -T8 -16 2054.50s user 3.47s system 783% cpu 4:22.80 total % gzcat *.gz | time zstd -T8 -16 --long=29 | wc -c 196405076 zstd -T8 -16 --long=29 1083.06s user 2.41s system 783% cpu 2:18.55 total Almost 15x smaller than the baseline, and more than 2x smaller than bzip3, also CPU time halves (since long matches are found earlier, so there's less work to do). (the baseline number is slightly different because I don't have the exact Perl version set used by the author) Also, in the benchmarks using lrzip, which would make the window size less relevant, zstd is not even compared.

CodesInChaos

I think this should include benchmarks zstd with larger windows and long range mode. I wouldn't be surprised if the window they used is smaller than an individual version tar file, which would prevent useful compression.

amelius

> However, the complexity of the algorithms, and, in particular, the presence of various special cases in the code which occur with very low but non-zero probability make it impossible to rule out the possibility of bugs remaining in the program. Sounds like perhaps a nice testcase for formalization + AI?

JdeBP

An interesting unintentional benchmark is to go to https://github.com/iczelia/bzip3/releases and see to what degree bzip3 compresses its own release archives; and go to https://github.com/iczelia/bzip3/blob/master/.github/workflo... to see what options have been chosen for the other compressors here.

8organicbits

I was processing compressed .jsonl files recently (JSON lines format). I found that lzma gave a much better compression than gzip or bzip2, which helps for archival costs, but it's challenging to work with as software support is lacking. I do duckdb processing which supports gzip transparently. There's an extension for bzip2, but not for lzma or bzip3. I ended up using gzip because it's best supported by the software I use and most likely to have support in software I adopt. But it gave the worst compression results of the options I tried. These bzip3 numbers certainly give me FOMO...

self_awareness

Any relation to Iczelion from the masm32 tutorial?

sergiotapia

"DO NOT COMPRESS ANY DATA WITH THIS PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER SMALL, THAT THE DATA WILL NOT BE RECOVERABLE." So why would anybody, hobbyist or enterprise, use this? Or does something older like 7zip also have this caveat that I've never experienced.

roschdal

Imagine bzip4

algorithm314

Bro just decompile agiannis_text

ThiraSoft

I would be interrested in a comparison with openzl

red_admiral

Why LGPL when the original license is more permissive? ( https://sourceware.org/bzip2/manual/manual.html )

charcircuit

It's distasteful to use such a name when it's not created by the bzip authors. For some reason open source developers love using would be trademark infringing names instead of coming up with something unique.

sehw

For data recovery of archival stuff it's better to keep it uncompressed, no?

whatever1

I think compression algorithms are ripe for significant improvement with LLMs. It’s an ideal candidate problem you can have in a closed loop evaluation, and you can just let agent try things.

blobbers

Not sure why this is being posted today or getting comments. bzip3 not a terribly impressive or useful compression algo. benchmark on enwik: https://www.mattmahoney.net/dc/text.html#1703

BorisMelnik

faster than gzip now or still slower? (sorry I did not read readme)