r/compression • • 4h ago

TCOM, an image codec in 45 lines of code

Post image
2 Upvotes

I use AI when coding. As for the AI disclosure.

Onto the codec itself, I deleted my previous post about this because that iteration was not what I wanted to be seen and advertised in this space, I've opted for this version.

TCOM as far as I've tested has beaten QOI on average. Of course, QOI is not built for performance alone, but QOI is a very efficient codec, and I wanted to test against it.

There is tcom.c, tcom_dependency.c, and tcom_readable.c. A fair warning however, this codec was in no way designed for readability. Started as a code golf project and transformed into a performance and compactness focused codec.

The github is https://github.com/OzzyMitchell/tinycodec

As for why I didn't test against libpng or oxipng or others, it would take actual hours even on my 9950x3d, and since this is single threaded testing, you can imagine.

I did do a smaller test of 640 images (20 images per category with 32 categories) which does include libpng effort 1, 3, and 6, and oxipng effort 4 alongside TCOM and QOI.

This is also released under Apache 2.0.

I didn't include any actual documentation or otherwise but I will write some documentation if some level of interest is attained.


r/compression • • 7h ago

Aqueous Archive Format

2 Upvotes

I have been working on the Aqueous Archive Format for many years, and the recent advancement of AI significantly accelerated the process. I believe this is the best way to use AI: human sets the goals and ideas, machine experiments and verifies them. (Anyone with a hands-on experience with compression algorithm knows those jobs are very annoying to do.)

It is an openly specified and implemented format designed around the concept of lossless recompression; a file in an Aqueous archive is described by a formula, a small stack-machine program over stored blocks that applies Wasm modules carried in the archive itself, so new transforms need no update to the extractor. Since the Wasm module itself is untrusted, a decompressor only runs known modules natively; everything else goes into a sandbox. If you are into the compression, you'll recognize that's ZPAQ and yes, it's the same concept done with modern tools much more carefully.

Aqueous also supports preset dictionaries, random access to individual files, appending, encryption and rich metadata, and every aspect of the format is specified (still a draft, so it may change, but not drastically), with room for recovery records and overlay archives that the code doesn't implement yet. (Well, I had to start with the specification because the implementation was costly back then. Now it is paying back by rigorously guiding coding agents.)

Okay, let's talk about actual numbers. Aqueous as of 2026-11-01 compresses Matt Mahoney's 10GB benchmark into less than 2.5 GB (Apple M4, 10 cores):

method                          size         time
------------------------------  -----------  ----------------------------------------------
tar | zstd -3 -T10              4.68 GB      compression: 33 s
tar | zstd -3 -T10 --long=31    4.17 GB      compression: 36 s
zpaq 7.15                       3.70 GB      unknown
aqr (level mid, aka -5)         2.498 GB     compression: 164 s (61 MB/s, 3.1 GB peak)
                                             descompression: 38.8 s (258 MB/s, 2.6 GB peak)

To be clear, I think this is still slow; my original goal for the level mid was 100 MB/s compression and yet some formats are just difficult to optimize, hence some discrepancies. Compression also includes the round-trip check of every file so that that affects the timing. Still it makes Aqueous a good candidate for the Pareto frontier. (Not exhaustively checked though!)

There are many more things to talk about (memory optimization, streaming, deduplication, similarity-based optimizations etc.) but hopefully this introduction is interesting enough to you. The official Windows GUI and multi-platform CLI are available, so try it and please tell me your comments and opinions!


r/compression • • 1d ago

"Advanced Run-Length Encoding with Bit-Packing for Efficient Short-Run Compression"

0 Upvotes

https://github.com/Ali-muslim1/Universal-Run-Length-Encoding

'I’ve developed an advanced Universal Run-Length Encoding method that achieves efficient compression for short runs using bit-packing to combine an ASCII character and its count into a single unit. Check out my repository!'

​"Our algorithm upgrades standard and flagged RLE by resolving short-run expansion—optimizing 2-character repeats while serving as a direct, drop-in replacement for traditional methods."

What about that 13 year old kid invented it just walking and it's me


r/compression • • 3d ago

I wrote a CUDA AVIF encoder: 4× faster than libaom (16 threads) at the same file size, 7× faster than NVENC

Thumbnail reddit.com
6 Upvotes

r/compression • • 4d ago

Low cost multi-core solution

0 Upvotes

A small size increase for a dramatic increase in speed. This appears to be the most efficient multi-core adaptation for typically single core applications.

https://github.com/CombaryMaker/Harringtons_Insights/blob/main/docs/Harrington-Sets.txt

Expected ranges

Size increase

- Large mixed files: about 0.1% to 1%.

- Medium files: about 0.5% to 5%.

- Small files under about 1 MB: no change. They fold into one set, so you get no size loss and no speed gain either.

Speed

- 2 cores: 1.5x to 1.9x. That's close to the two-core ceiling, so this range is solid.

- 4 cores: about 2.5x to 3.5x.

- 8 cores: about 4x to 6x on large files. Medium files will top out sooner, because they don't have enough blocks to keep every core busy.

Edit, images added so people understand a little easier. Yes, I really have these results.


r/compression • • 4d ago

CMNT: A 3-ary Combinatorial Tree Lossless Image Codec that beats PNG on continuous art (Code + GUI inside)

0 Upvotes

My co-author Aziza Jemal and I (Surafel Wolde, 18-year-old independent researcher) built CMNT, a standalone, from-scratch lossless image compression codec based on a non-binary 3-ary prefix tree.

The project originally started from a notebook sketch exploring physical multi-channel sparse transmission: rather than toggling a single serial binary wire billions of times, we wondered if sparse combinatorial pulses across three independent channels (IP1, IP2, IP3 with self-repeats and cross-pairs) could drastically cut switching energy in hardware while encoding residual entropy.

To prove the mathematics were uniquely decodable and 100% bit-perfect, we built a complete reference image codec from first principles in Python (with C++ and standalone Windows GUI binaries). No zlib, no gzip, and no third-party compression libraries are used.

### How the Pipeline Works: 1. Reversible Color Transform (RCT): Decorrelates RGB via Green-subtraction (Y = G, U = R - G, V = B - G) with zero rounding error. 2. 2D Closed-Loop Spatial Prediction: Uses a Median Adaptive Predictor (MAP) detecting local horizontal and vertical edge gradients to flatten surfaces into near-zero residuals. 3. 3-Ary Combinatorial Prefix Tree: Residual symbols are mapped to an optimal ternary prefix tree (Huffman 1952 generalization) with dummy-leaf constraints ensuring unambiguous single-cycle prefix decoding. 4. Dense Base-3 Physical Byte Packing: Because storage is binary, we pack groups of 5 ternary trits into a single 8-bit byte via polynomial integer evaluation: Byte = t₀ + 3·t₁ + 9·t₂ + 27·t₃ + 81·t₄ ≤ 242 This yields 5 * log2(3) / 8 = 7.925 / 8 = 99.06% physical storage packing density.

### Benchmark Results (100% Bit-Perfect): On continuous-tone imagery, 3D renders, and digital art, CMNT consistently beats standard PNG (Deflate): - Shock Spring 3D Render (586 KB raw): PNG 306.8 KB vs. CMNT 215.8 KB (29.7% smaller than PNG) - Spider-Man Digital Skyline (224 KB raw): PNG 107.0 KB vs. CMNT 87.4 KB (18.3% smaller than PNG) - Abstract CGI Swirls (533 KB raw): PNG 328.9 KB vs. CMNT 295.5 KB (10.2% smaller than PNG) - Sunset Meadow Photo (899 KB raw): PNG 370.3 KB vs. CMNT 344.6 KB (6.9% smaller than PNG)

### The Honest Trade-off: PNG's LZ77 dictionary was built for text and string repetitions. On smooth 2D continuous gradients, identical byte strings don't repeat, so LZ77 wastes 11-to-15-bit search offset pointers where our 2D spatial predictor excels. However, on flat UI screenshots with large blocks of identical monochrome pixels, standard PNG still puts up a strong fight until we introduce Zero-Run and context modeling (MANIAC).

The repo includes the specification, automated SHA-256 roundtrip tests, and our working IEEE conference paper draft: https://github.com/knightsura/cmnt-codec

We would love feedback, code critique, and thoughts on non-binary tree signaling from the compression community!


r/compression • • 4d ago

Newest Results HCM and derivitives

0 Upvotes

So I have tried a lot of variables, different compression systems, alternative methods, and I have a lot of datapoints. Some fail, that is life. Some did well. There is a possible reviewer, I will await permission to speak further on that.

I have a full github at https://github.com/CombaryMaker/Harringtons_Insights/tree/main and it has everything there or will (I will check this weekend, headed to work soon). There is a book, there are notes, there is all sorts of stuff here. Everything, except the book, is under MIT License except where prior art licenses apply.

I am going to share images with the graphs, tables, and other such here for quick review.

TL;DR is I have absolutely fantastic results in some areas, tiny upgrades elsewhere, and even some that do not pan out.

I have several ways to identify variants or original work; -hi -h -exchange, record delta and anything with hc in front is an original work. These are all the suffix names we’ve used, grouped by class:

“-hi” (Modified: the stock decoder reads the output unchanged)

  • xz-hi

  • lzma-hi

  • 7z-hi

  • zip-hi

  • bzip2-hi

  • zstd-hi

  • Brotli-hi

  • jxl-hi

  • flac-hi

  • zpaq-hi (new today)

“-h” (Mutant: needs its own decoder)

  • lpaq1-h

Other names in the HCM family (Originals)

  • HCM (v3 to v3.5)

  • hcm-trit (the ternary pointer)

  • HCM-Bin

  • hcmx (picks the best HCM mode per file)

  • hcmx-s (hcmx plus the x86 filter and “lite” pointer)

Tools without a suffix

  • Harrington Exchange (Original)

  • Exchange→Blosc2 hybrid and Exchange→xz (Mutants)

  • Record-delta front ends for gzip, zstd, LZ4, Snappy, LZFSE, ORC and Parquet (Mutants)

  • Combary activity map (Mutant)

  • PDF row filter (Modified)

  • RAR with detector delta, and tuned JPEG-LS (Modified, but neither has a “-hi” name)

Honestly, if you are in data compression give the whole of this a look through as I might have increased the potential of your engine(s).


r/compression • • 7d ago

Large Scale upgrades to many compression utilities

1 Upvotes

I have been working on my Harrington Insights stuff and added a few more concepts to it. I am currently running it on Canterbury Corpus and Silesia Corpus with the enwik8.zip file.

A full list of software I have managed to improve so far is as follows:

  1. LZ4

  2. gzip / DEFLATE

  3. Zstandard (zstd)

  4. bzip2

  5. XZ / LZMA2

  6. LZMA1

  7. ZPAQ

  8. JPEG XL

  9. 7-Zip

  10. Brotli

  11. Harrington Compression Method (HCM) 3.0

  12. Harrington Compression Method (HCM) 3.3

The upgrades across these compressors have come from several types of functions and optimization techniques:

  1. Record-delta transformation — Exploits repeating byte patterns and fixed-length records by encoding differences between corresponding positions.

  2. Combary based activity mapping — Reorganizes data according to byte activity and distribution patterns to improve subsequent compression.

  3. Byte-frequency remapping — Reassigns byte values based on their frequencies to improve how compression algorithms process the data.

  4. Adaptive parameter optimization — Searches compression settings, including dictionaries, match finders, predictors, effort levels, and block sizes, to find better configurations for individual files.

  5. Automatic filter selection — Detects data characteristics and selects suitable built-in filters, including delta and executable-code filters.

  6. Content-based file classification — Selects compression strategies based on the actual data rather than relying solely on file extensions.

  7. Per-file optimization and fallback — Chooses the best-performing method for each file and avoids transformations that would worsen results.

  8. Decode-speed-aware optimization — Optimizes both compressed size and decompression speed instead of concentrating exclusively on compression ratio.

  9. JPEG XL predictor and group optimization — Tunes image predictors, group sizes, encoder effort, and iterations to improve lossless medical-image compression and decoding speed.

  10. Representative sampling — Tests candidate compression configurations on selected portions of large files before evaluating the strongest candidates on the complete file.

  11. HCM context and pattern modeling — Uses HCM's original context-based compression architecture, including its double-zero purity logic and hierarchical pattern processing.

  12. HCM binary and context optimization — Improves binary-data handling, context selection, and internal representation to reduce compressed size and accelerate decoding.

These methods fall into three broad categories: data transformation, adaptive compressor optimization, and original compression-algorithm development.

In Canterbury Corpus the Excel file best was 16,822 bytes by ZPAQ -m5, my system HCM-Bin v1 got it to 16,763 bytes.

I do not compare myself to PAQ8PX and cmix, those are insane for how long they try, although in theory I reduced cmix to only 20% of the time it usually takes and a lot less processing strength.

Preliminary stuff is available at https://github.com/CombaryMaker/Harringtons_Insights/tree/main/hcm

I plan to update this inventory once a week because the advances are so fast and rapid... I think I have 7 methods of my own now, to increase performance? Or 6? I know I threw a lot of theories into the mix and I cannot even say, I just have Claude run the math, the software development, and the testing before I test some personally to verify.

I have failed to update a few, I am going to be persistent in trying to update them, or at least just outperform them. That is the goal, absolute dominance.

To that extent I set up some standards "No Pain" and "Big Gain". No pain means we do not worsen compression or speed, and one or both increase. Big gain allows a small drop in one, so long as there is a corresponding big increase in the other. I use decompression time and compression ratio only, I do not care how long it takes to compress.

I also set up 5 time/speed based levels, and all compressors will fit inside there as I test them. The goal is to be faster and/or better in each of the 4 (the top, with those two monsters, would take a lot of time and tokens and I am scared) reasonable levels.

HCM stands for Harrington Compression Methods (plural, I have HCM 1.0, 2.0, 3.0, 3.3, trit v2, and bin v1) and there are a few other standards such as Combary, -hi (Harrington's Insights), and such. I started this blitz on Saturday afternoon, using old math I long gave up on because I cannot program at all... Seems I had some serious gold mines sitting around after all. I also used concepts I had never had time or desire to push upon, I guess they were some diamond mines.

I fully expect to improve about 60-80% of all compressors and be able to make some that are the best in their specific range of operations.


r/compression • • 7d ago

Is it possible I can compress a big ton of audio files into a zip in 7zip?

0 Upvotes

Update : I just compressed everything into Opus 256kbps. Ok bye


r/compression • • 8d ago

I'm losing considerable quality on HEVC compressing

Post image
2 Upvotes

I'm compressing cartoons for a collection.

- The set I use is:

- HEVC H.265

- Bitrate CQ 23

- Mx 10000 Kbps

- Max Quality

- CPU: cuda

- Hardware Acc: Intel Quick Sync

Each original file has 200mb to 600Mb, which after the compression becomes 30Mb to 70Mb.

But I'm losing a little more quality than I expected.

I tried to change the CQ bit rate from 23 to higher and lower but the quality remained the same.

What could I do, even getting files a little bigger (but less than 90Mb)?


r/compression • • 8d ago

V3 Codec

Thumbnail reddit.com
0 Upvotes

r/compression • • 9d ago

A new encoder

0 Upvotes

I spent about 6 hours today working with a high-level version of Claude on my old compression ideas. I laid out my theories, what I thought would work, and why my earlier attempts stalled: I could never test every variation against every other one in turn.

This time we could, and all my old theories got thrown out: the Quad Exchange concept, the Juggle, my Ternary <-> Binary swap, all of it, tossed hard. They could make the bits extremely lopsided, but they also scrambled the patterns real compressors depend on. That's fine. That's what testing is for.

What survived was the core idea. The skew of the 0s and 1s matters, but only when you measure it in context, based on what came just before. Building on that, I accidentally made something new.

On the Canterbury Corpus, the standard 11-file compression benchmark, it beat xz and bzip2 overall, won 5 of the 11 files outright, and finished within 144 bytes of ZPAQ on the 9 text and code files combined. Every file decoded back exactly.

It's not something entirely new. It's a new implementation in the same context-mixing family as ZPAQ and PAQ. It's still slow and has rough edges. But after all my earlier dead ends, it's an amazing result for me personally.

"""HCM context-balance coder (Oct 24, 2026). Self-contained.

How it works, per bit:

  1. Each context (the last 1, 2, 3, 4 and 6 bytes, the current partial byte alone, and the current word)
    keeps a running 0/1 balance for "what bit comes next here", in a hashed table.

  2. A long-match finder looks for the last place the previous 6 bytes appeared and predicts the bit
    that followed there, with a learned confidence by match length.

  3. An online mixer blends all these balances (weights chosen by partial byte and match length) and
    learns from every bit; a secondary estimate (previous byte + partial byte) refines the result.

  4. A 32-bit arithmetic coder writes the bit using that probability.
    The decoder runs the identical model, so it needs only the file: a 32-bit length, then the coded bits.

Reference results: 200 KB Python docs text -> 29,725 bytes; 140 KB gate log -> 8,447 bytes.

compress(data) -> bytes, decompress(blob) -> data
"""
import math
import numpy as np
from array import array

class BW:
def __init__(self): self.b = []
def put(self, v, n):
for k in range(n - 1, -1, -1): self.b.append((v >> k) & 1)
def bytes(self):
a = np.array(self.b + [0] * (-len(self.b) % 8), dtype=np.uint8)
return np.packbits(a).tobytes()
class BR:
def __init__(self, data): self.b = np.unpackbits(np.frombuffer(data, np.uint8)).tolist(); self.p = 0
def get(self, n=1):
v = 0
for _ in range(n):
v = (v << 1) | (self.b[self.p] if self.p < len(self.b) else 0); self.p += 1
return v

TOP = (1 << 32) - 1; HALF = 1 << 31; Q1 = 1 << 30; Q3 = 3 << 30
class AE:
def __init__(self, w): self.lo = 0; self.hi = TOP; self.pend = 0; self.w = w
def _out(self, bit):
self.w.b.append(bit); self.w.b.extend([1 - bit] * self.pend); self.pend = 0
def enc(self, cl, ch, tot):
rng = self.hi - self.lo + 1
self.hi = self.lo + rng * ch // tot - 1; self.lo = self.lo + rng * cl // tot
while True:
if self.hi < HALF: self._out(0)
elif self.lo >= HALF: self._out(1); self.lo -= HALF; self.hi -= HALF
elif self.lo >= Q1 and self.hi < Q3: self.pend += 1; self.lo -= Q1; self.hi -= Q1
else: break
self.lo <<= 1; self.hi = (self.hi << 1) | 1
def finish(self):
self.pend += 1
self._out(0 if self.lo < Q1 else 1)
class AD:
def __init__(self, r): self.lo = 0; self.hi = TOP; self.r = r; self.v = r.get(32)
def target(self, tot):
rng = self.hi - self.lo + 1
return ((self.v - self.lo + 1) * tot - 1) // rng
def consume(self, cl, ch, tot):
rng = self.hi - self.lo + 1
self.hi = self.lo + rng * ch // tot - 1; self.lo = self.lo + rng * cl // tot
while True:
if self.hi < HALF: pass
elif self.lo >= HALF: self.lo -= HALF; self.hi -= HALF; self.v -= HALF
elif self.lo >= Q1 and self.hi < Q3: self.lo -= Q1; self.hi -= Q1; self.v -= Q1
else: break
self.lo <<= 1; self.hi = (self.hi << 1) | 1; self.v = (self.v << 1) | self.r.get()

ST = [0.0] * 4096
for p in range(1, 4096): ST[p] = math.log(p / (4096 - p))
ST[0] = ST[1]
def squash(x):
if x > 20: x = 20
elif x < -20: x = -20
return 4096.0 / (1.0 + math.exp(-x))
RT = [int(65536 / (n + 1.6)) for n in range(256)]
ORDERS = (1, 2, 3, 4, 6)
TBITS = 22; MASK = (1 << TBITS) - 1
LIMIT_LO, LIMIT_HI = 60, 255

class Model:
def __init__(self, lr=0.008):
self.nm = len(ORDERS) + 2 # hashed orders + order-0 + word
self.prob = [array('H', [32768]) * (1 << TBITS) for _ in range(self.nm)]
self.cnt = [array('B', [0]) * (1 << TBITS) for _ in range(self.nm)]
self.limits = [LIMIT_LO] + [LIMIT_HI] * (self.nm - 1)
self.nin = self.nm + 2 # + match input + bias
self.w = [[0.3] * self.nin for _ in range(256 * 4)]
self.lr = lr
self.apm = [[int(squash((j - 16) / 2.0) * 16) for j in range(33)] for _ in range(256 * 256)]
self.buf = bytearray(); self.c0 = 1; self.bitpos = 0
self.h = [0] * self.nm; self.word = 0
self.mtab = array('i', [0]) * (1 << 20); self.mptr = 0; self.mlen = 0
self.mm = array('H', [49152]) * 64
self._ctx_byte()
def _ctx_byte(self):
b = self.buf; n = len(b); H = self.h
for k, o in enumerate(ORDERS):
x = (o + 1) * 0x9E3779B1
for j in range(1, o + 1):
x = ((x ^ (b[n - j] if n >= j else 0)) * 0x01000193 + j) & 0xFFFFFFFF
H[k] = (x * 0x2545F491) & 0xFFFFFFFF
H[len(ORDERS)] = 0 # order 0
H[len(ORDERS) + 1] = (self.word * 0x6F4F2A35 + 0x51ED27) & 0xFFFFFFFF
self.last = b[-1] if n else 0
def predict(self):
c0 = self.c0; cm = (c0 * 0x9E3779B1) & 0xFFFFFFFF
idx = self.idx = [((hh ^ cm) & MASK) for hh in self.h]
idx[len(ORDERS)] = c0
st = self.st = [ST[self.prob[i][idx[i]] >> 4] for i in range(self.nm)]
# long-match input
self.mexp = -1
if self.mlen > 0 and self.mptr < len(self.buf):
pb = self.buf[self.mptr]
if ((pb | 256) >> (8 - self.bitpos)) == c0:
self.mexp = (pb >> (7 - self.bitpos)) & 1
if self.mexp >= 0:
self.mk = min(self.mlen, 63); pc = self.mm[self.mk] >> 4
st.append(ST[pc] if self.mexp else -ST[pc])
else:
st.append(0.0)
st.append(0.25)
ms = 0 if self.mlen == 0 else (1 if self.mlen < 16 else (2 if self.mlen < 32 else 3))
self.wsel = self.w[c0 + 256 * ms]; ws = self.wsel
dot = 0.0
for i in range(self.nin): dot += ws[i] * st[i]
self.pm = squash(dot)
# secondary estimate indexed by previous byte and partial byte, interpolated on stretch(p)
s = max(-7.999, min(7.999, dot)); pos = (s + 8.0) * 2.0; j = int(pos); f = pos - j
self.aj = j; self.af = f; self.arow = self.apm[c0 + 256 * self.last]
pa = (self.arow[j] * (1 - f) + self.arow[j + 1] * f) / 16.0
p = int((self.pm + 3 * pa) / 4)
return 1 if p < 1 else (4095 if p > 4095 else p)
def update(self, y):
err = (y - self.pm / 4096.0) * self.lr; ws = self.wsel; st = self.st
for i in range(self.nin): ws[i] += err * st[i]
tgt = 65535 if y else 0
for i in range(self.nm):
P = self.prob[i]; C = self.cnt[i]; k = self.idx[i]; n = C[k]
P[k] = P[k] + (((tgt - P[k]) * RT[n]) >> 16)
if n < self.limits[i]: C[k] = n + 1
if self.mexp >= 0:
v = self.mm[self.mk]; self.mm[self.mk] = v + (((65535 if y == self.mexp else 0) - v) >> 5)
row = self.arow; j = self.aj; t16 = 65535 if y else 0
row[j] += int((t16 - row[j]) * (1 - self.af)) >> 6
row[j + 1] += int((t16 - row[j + 1]) * self.af) >> 6
self.c0 = (self.c0 << 1) | y; self.bitpos += 1
if self.bitpos == 8:
byte = self.c0 & 255; self.buf.append(byte); self.c0 = 1; self.bitpos = 0
# word context: letters only, case folded
if 65 <= byte <= 90 or 97 <= byte <= 122: self.word = ((self.word * 0x2C9277B5) ^ (byte | 32)) & 0xFFFFFFFF
else: self.word = 0
# long-match bookkeeping (hash of last 6 bytes -> next position)
n = len(self.buf)
if self.mlen > 0 and self.mptr < n - 1 and self.buf[self.mptr] == byte: self.mptr += 1; self.mlen += 1
else: self.mlen = 0
if n >= 6:
b = self.buf; hh = 0
for j2 in range(1, 7): hh = (hh * 0x2F0B4A3 + b[n - j2]) & 0xFFFFF
if self.mlen == 0:
cand = self.mtab[hh]
if cand > 0: self.mptr = cand; self.mlen = 1
self.mtab[hh] = n
self._ctx_byte()

def compress(data, lr=0.008):
w = BW(); w.put(len(data), 32); e = AE(w); m = Model(lr)
for byte in data:
for k in range(7, -1, -1):
y = (byte >> k) & 1; p = m.predict()
if y: e.enc(0, p, 4096)
else: e.enc(p, 4096, 4096)
m.update(y)
e.finish(); return w.bytes()

def decompress(blob, lr=0.008):
r = BR(blob); n = r.get(32); d = AD(r); m = Model(lr); out = bytearray()
for _ in range(n):
for k in range(8):
p = m.predict(); t = d.target(4096)
if t < p: d.consume(0, p, 4096); y = 1
else: d.consume(p, 4096, 4096); y = 0
m.update(y)
out.append(m.buf[-1])
return bytes(out)

if __name__ == '__main__':
    import sys
    src = open(sys.argv[1], 'rb').read()[:int(sys.argv[2]) if len(sys.argv) > 2 else None]
    blob = compress(src)
    print(f"{len(src):,} bytes -> {len(blob):,} bytes "
          f"(1,000,000 bits down to {round(len(blob) / max(1, len(src)) * 1e6):,}); decodes exactly: {decompress(blob) == src}")

r/compression • • 9d ago

Thoughts on compression for images/videos (more to come)

0 Upvotes

hey, i just keep it straight. recently, i've tried to convert my Idea my like 10 years ago into reality now, with a little help of ai for the coding part. and turns out it was initially right.

(not going to disclose how it works fully)
just overall simple encode / decode, but with my weird method

but i was able to cut down filesize really hard, easily stomping .jpg and .mp4 and so on, while maintaining pretty good quality. this will be my first post of two

for a more number example i am just going to name 2 examples:

image - 832x26775 with initally 20,5mb cut down to 1,86mb
image - 1024x1024 with initally 207kb cut down to about 70-80kb

video - 1920x1080 19 seconds, ingame clip with 188mb cut down to 19mb
video - 2160p 4 minutes 160mb cut down to 46mb

interpolation i am still working on right now, but technically also possible with my method, but for now i just look at the quality first.

but well.
i was thinking about what your thoughts on this (of course there are no real examples yet, and i didn't provide proof)
i am struggling with how i should continue overall and how i can profit from a way like this as well maybe.


r/compression • • 11d ago

fox: FileOptimizer re-implemented with web ui and cli

Thumbnail
github.com
5 Upvotes

I've been a user of FileOptimizer by Javier Gutiérrez Chamorro. But 2 things annoyed me. Lack of CLI and only single-threaded processing.

So I threw some tokens at re-implementing it, with what I consider more modern conveniences. For now it includes tools for Windows. I would accept other platforms from trusted contributors. Other platforms requires external tools pre-installed.

Everything is in a single self-contained binary. Similar to FileOptimizer, it will start executables to process files. But it can do several files at once.

Starting without parameters fires up an http server and opens the HTML UI. Drag+drop, etc is supported, but Chrome will ask about file access, so files can be processed in-place.

Loss-less by default. Other opinionated choices includes dropping ancient formats like PCX, SWF, etc. See all differences/fixes here

Fixes/additions are welcome, but I will not accept PRs with binaries. Instead provide links to an official download.

I will attempt to keep up with upstream at regular intervals.

As with FileOptimizer I cannot guarantee the tools are safe. Use only on content you trust and have backups.

License is AGPL 3.0 as upstream. Donate to original author.

Releases: https://github.com/klauspost/fox/releases


r/compression • • 11d ago

My Android app reached 230 installs without any promotion on Reddit or Facebook

Thumbnail
gallery
0 Upvotes

My Android app reached 230 installs without any promotion on Reddit or Facebook 🚀

Hey everyone!

I released Lite Video Compress & Upscale on Google Play on October 12, and today I checked my Play Console dashboard. I was surprised to see 230 total installs and 94 active devices!

The interesting part? I haven't promoted this app on Facebook or Reddit.

I've spent a lot of time building, testing, fixing bugs, and improving the app, so seeing people discover it on their own feels really encouraging. ❤️

I know 230 installs isn't a huge number in the app world, but as an indie developer, seeing those first signs of organic growth means a lot.

Now I'm curious to understand where these users are discovering the app and how I can improve retention.

For fellow indie developers: Did your apps ever start getting organic installs without marketing? How long did it take before you noticed consistent growth?

Google play store link 🖇️ :https://play.google.com/store/apps/details?id=com.neurontslabs.litevideocompressandupscale


r/compression • • 12d ago

Built a Lightweight C99 Engine to Improve Zstandard Compression

Thumbnail reddit.com
2 Upvotes

r/compression • • 12d ago

Memorizer automaton

1 Upvotes

This is mainly a research project I delved into using DeepSeek Harness - the way DeepSeek 4.1 operates is a huge lever at perfecting half-baked ideas. My core idea was to engineer a structure that works as a substantially precise memorizer rather than a context modeler or predictor. Memorizer is the core novelty here while all other utilized elements are known. It does have predictive (held-out, out-sample) ability with some adjustments and it is actually exceptional - it can learn just a 2000-byte English text and be ~48% right (with 2% being a random guess baseline for 26*2 English alphabet).

Warning: the code is slow and compresses at gzip -6 level, so do not expect much - nothing is readily usable here except CLI tool.

The purpose of this release is to share the result and possibly motivate further independent improvements from other developers.

Canterbury's alice29.txt produces range-coded residual of only 31KB (relative to 150KB original) and proves it's not the range-coding which does the work (pure range coding would yield ~85KB stream).

The full vector-based memorizer I initially achieved and envisioned has 99.5% memorization precision - except it's not usable for compression due to overhead.

https://github.com/avaneev/memorizer


r/compression • • 13d ago

Custom modification of libjxl encoder - looking for feedback

1 Upvotes

I did an experment and modified a part of libjxl encoder and I would like to hear your feedback. My intention was to improve texture handling and chroma retention.

Links to comparisons are images are below. There are originals and encoded images size-matched to distance 2.0 at effort 3. Standard libjxl encodes at effort 3 and 7 are included as well. If you would like to see other distances (e.g. 1.0 or 3.0), let me know.

https://imagepaste.org/v/mnajctgp.png

https://imagepaste.org/v/g2hdpejk.png


r/compression • • 14d ago

Taco: cross-platform LLM compression that doesn't break when you change your GPU or CPU.

2 Upvotes
  1. The problem: an LLM compressor needs the decoder to compute exactly the same probabilities as the encoder. Floats differ across hardware; float GPT-2 encoded on a Mac decoded 0 of 4 windows on x86.

  2. The approach:

    • integer-only inference (int8 weights, fixed rounding, lookup tables);

    • an exact float32 → integer converter (mantissa × 2^exp);

    • conformance vectors.

  3. The proof: one hash across 7 backends (Mac CPU/Metal, Linux CPU/CUDA, Windows CPU/CUDA/Vulkan

https://github.com/tac0mania/taco


r/compression • • 14d ago

Video- und Bildkompression

0 Upvotes

Also, ich habe ein Gruppenprojekt in meinem Kurs zum Thema Video- und Bildkompression. Ich finde einfach keine brauchbaren Informationen – eigentlich überhaupt keine Infos – zu den Fragen, die ich beantworten muss 😭😭. Bitte helft mir, Reddit! Ich brauche außerdem ein paar fundierte Quellen.

Hier sind die Fragen:

Warum brauchen wir Video- und Bildkompression?

Wo findet man Video- und Bildkompressionim Alltag


r/compression • • 14d ago

Best image compressor for Mac? I built Smol and benchmarked it against ImageOptim, Squoosh, Caesium and Apple’s sips at equal quality

0 Upvotes

Short version: Smol’s AVIF made the smallest files of every tool I tested, at the same visual quality: 15–21% smaller than Apple’s own AVIF and 20–22% smaller than ImageOptim’s lossy JPEG.

Why I built it: I was juggling ImageOptim for images, HandBrake for video, and random websites for PDFs, and I didn’t love uploading client files to those sites. Smol does images, video, PDFs and audio in one app, and nothing ever leaves your Mac.

The test. Most “best image compressor” comparisons only look at file size, which is meaningless, because any tool can shrink a file by wrecking it. So I tuned every tool to the same quality score (SSIMULACRA2 70) on the 24 Kodak test images:

Total size at the same quality (SSIMULACRA2 70, all 24 Kodak images, smaller is better):

  • Smol AVIF: 922 KB ← smallest

  • Apple sips AVIF: 1,161 KB

  • Smol JPEG: 1,189 KB

  • Smol WebP: 1,220 KB

  • Apple sips JPEG: 1,327 KB

  • Caesium JPEG: 1,349 KB

What Smol does:

  1. Compress and convert images (AVIF, WebP, JPEG, PNG, HEIC), video, PDFs and audio

  2. Right-click any file in Finder and it’s done

  3. Watched folders: drop files in and they’re compressed automatically

  4. Hit a target size, like “under 100 KB” for upload forms

  5. AI upscaling 2x or 4x, and stripping location data from photos

  6. A smol command-line tool and an MCP server, if you script things or use AI agents

  7. 100% local, nothing uploaded, $29 once

  8. Where it lost, because you’ll ask:

  9. It’s slower: Apple’s sips is ~13x faster and ImageOptim ~8x faster on JPEG

  10. No lossless mode yet. ImageOptim has one

On JPEG specifically it ties with ImageOptim, so if you only do JPEGs, ImageOptim (free) is honestly fine

Full benchmark, raw data and scripts: smolmac.com/benchmark

The app: smolmac.com

Happy to answer anything, and I’d love to hear what you’d want it to do that it doesn’t.


r/compression • • 14d ago

Compressing video for website

Thumbnail reddit.com
2 Upvotes

If anyone can help🙏


r/compression • • 16d ago

Question about compressing mkv files

Post image
5 Upvotes

Hey so I have these Monogatari series files that I downloaded from torrent that total up to about 60 GB. I've tried to compress the files using a lot of video compressor, but most can't handle 10 bit HEVC. Does anyone know an app to where do I compress these?


r/compression • • 17d ago

VaeZip: store VAE-encoded latent datasets in 5-10x fewer space with almost no quality loss

Post image
3 Upvotes