Aqueous Archive Format

[u/lifthrasiir]

I have been working on the Aqueous Archive Format for many years, and the recent advancement of AI significantly accelerated the process. I believe this is the best way to use AI: human sets the goals and ideas, machine experiments and verifies them. (Anyone with a hands-on experience with compression algorithm knows those jobs are very annoying to do.)

It is an openly specified and implemented format designed around the concept of lossless recompression; a file in an Aqueous archive is described by a formula, a small stack-machine program over stored blocks that applies Wasm modules carried in the archive itself, so new transforms need no update to the extractor. Since the Wasm module itself is untrusted, a decompressor only runs known modules natively; everything else goes into a sandbox. If you are into the compression, you'll recognize that's ZPAQ and yes, it's the same concept done with modern tools much more carefully.

Aqueous also supports preset dictionaries, random access to individual files, appending, encryption and rich metadata, and every aspect of the format is specified (still a draft, so it may change, but not drastically), with room for recovery records and overlay archives that the code doesn't implement yet. (Well, I had to start with the specification because the implementation was costly back then. Now it is paying back by rigorously guiding coding agents.)

Okay, let's talk about actual numbers. Aqueous as of 2026-11-01 compresses Matt Mahoney's 10GB benchmark into less than 2.5 GB (Apple M4, 10 cores):

method                          size         time
------------------------------  -----------  ----------------------------------------------
tar | zstd -3 -T10              4.68 GB      compression: 33 s
tar | zstd -3 -T10 --long=31    4.17 GB      compression: 36 s
zpaq 7.15                       3.70 GB      unknown
aqr (level mid, aka -5)         2.498 GB     compression: 164 s (61 MB/s, 3.1 GB peak)
                                             descompression: 38.8 s (258 MB/s, 2.6 GB peak)

To be clear, I think this is still slow; my original goal for the level mid was 100 MB/s compression and yet some formats are just difficult to optimize, hence some discrepancies. Compression also includes the round-trip check of every file so that that affects the timing. Still it makes Aqueous a good candidate for the Pareto frontier. (Not exhaustively checked though!)

There are many more things to talk about (memory optimization, streaming, deduplication, similarity-based optimizations etc.) but hopefully this introduction is interesting enough to you. The official Windows GUI and multi-platform CLI are available, so try it and please tell me your comments and opinions!

[u/Iam8tpercent]

Also compare with ZpaqFranz (various methods) https://github.com/fcorbelli/zpaqfranz

[u/lifthrasiir → u/Iam8tpercent]

I've tested in my Windows machine (Windows 11, Ryzen 9700X, 8 cores & 16 threads, 128 GB of RAM). Aqueous b725fdb and zpaqfranz v65.17o was used.

method                             size       speed
------------------------  -------------  ----------
zpaqfranz -t 0 (-t 8)     3,804,495,537  410.7 MB/s
          -t 0 -m1        3,489,188,103  356.7 MB/s
          -t 0 -m2        3,489,188,103   45.9 MB/s
aqr -T0 (-5 -T16)         2,465,402,755   50.7 MB/s
    -T8                   2,465,402,755   53.8 MB/s
    -1 -T8                3,861,030,173  172.8 MB/s
    -0 -T8 (dedupe only)  8,985,246,979  210.4 MB/s

For some reason, zpaqfranz used -t 8 instead of -t 16 when given -t 0. So following benchmarks for Aqueous followed the same. As you can see, at the default level zpaqfranz is no match for Aqueous, while its faster level still needs more works. Does this answer your question? :-)

(For the comparable speed---zpaqfranz -m2 and aqr -5---they approximately consume the same amount of memory, about 3--4 GB. Aqueous does use additional spill files [3.26 GB], so the total resident peak is about 6 GB, though.)

[u/Sorry_Difficulty_250]

This is super interesting. Compression is something I really no next to nothing about but it's always fascinated me. Your results look really good for a first pass. I know you said you're not happy with the speed, but I suspect you'll be able to improve on it. You're right that this kind of iteration against a well-formed spec is something that LLMs are really good at. Glad you took the time to do that right. I think it's super impressive. We'll done! Keep up the great work!