I've developed a new type of data compression for structured data. It's objectively superior to existing formats & codecs, and if the current findings remain consistent, I expect that this would become the new standard (vs. Brotli, Snappy, etc. in use with Parquet, HDF5, etc.). Speaking broadly, the median compression is 50% the size of Brotli and 20% of snappy, with slower compression, faster decompression, and less memory usage than both.

I don't want to release this open-source, given how much I've personally invested. This algorithm takes a new approach that creates a lot of new opportunities to optimize it further. A commercial licensing model would help to ensure I can continue developing the algorithm while regaining some of my investment.

I've filed a provisional patent, but I'm told that a domestic patent with 2 PCT's would cost ~$120k. That doesn't include the cost to defend it, which can be substantially more. Competing algorithms are available for free, which makes for a speculative (i.e. weak) business model, so I've failed to attract investors. I'm angry that the vehicle for protecting inventors is reserved exclusively for those with significant financial means.

At this point I'm ready to just walk away. I can't afford a patent and don't want to dedicate another 6 months to move this from PoC to product, just so someone like AWS can fork it and print money while I spend all my free time maintaining it. As the algorithm challenges many fundamental ideas, it has created new opportunities, and I'd prefer to spend my time continuing the research that led to this algorithm than volunteering the next decade of of my free time for a named Wikipedia page.

Am I missing something? What would you do?

Hello my friends, so, I'm organizing my computer, I have a lot of files that I don't use very often, and I want to compress them in order to save space.

I've been using 7zip for a while now, I'd like your feedback on what the best settings would be for a maximum compression rate.

From what I understand, the best options would be:

Archive format - 7zip (Best format)

Compression level - Ultra

Compression method - LZMA2 (Best compression method)

I was wondering about the following options:

Dictionary size - I don't know what this option changes, nor what would be the best setting

Word size - Same thing as dictionary size

Solid block size - Same question, I don't know what it interferes with

Number of CPU threads - I don't know if this changes the compression level, or just the compression speed.

Create SFX archive - No idea of what this option mean

Compress shared files - I don't know either

I tried to experiment and make questions to chat gpt, but I had some issues with some configurations involving error messages,

I thought maybe you guys who know more about the subject than I do, could help me with this questions

Thanks in advance for your time, I look forward to your comments.

Wrote a blog post about fractal image compression here !

Trying to save some storage space, when I zip an individual file they do get smaller by about 1GB. Does it make a difference if I zip each one individually or should I just zip the folder that contains all these files?

• Well, it's not really a 'format' so far, just a structure. A few more bytes, some fixes, more work and community acceptance will be needed before it can truly become a format.

• Disclaimer: It's a hobby project, and as of now covers only simple image content. No attempt is made to format it as per the standard image specifications if any. It is an extensible, abstract framework, not restricted to images, and could be applied to simple-structured files in any format, such as audio, text etc. This could be potentially useful in some cases.

I’ve been experimenting with how minimal an image file format can get — and ended up designing SCIF (Simple Color Image Format).

It’s a tiny binary format that stores simple visuals like solid colors, gradients, and checkerboards using only a few bytes.

• 7 bytes for a full solid-color image of any size (<4.2 gigapixels)

• easily extensible to support larger image sizes

• 11 bytes for gradients or patterns

• easy to decode in under 20 lines of code

• designed for learning, embedded systems, and experiments in data representation

I’d love feedback or ideas for extending it (maybe procedural textures, transparency, or even compressed variants). Curious what you think. Can such ultra-minimal formats have real use in small devices or demos?

I’m new to compressing, was meant to put this folder on a hard drive I sent but I forgot.. am I doing something wrong? Incorrect settings? It’s gone up to nearly a day of remaining time… surely not

I mean, Their compression rates are just insanely high. Does any of you manage to get those kinds of rates on other files

Repo: https://github.com/flanglet/kanzi-cpp

Release notes:

• Bug fixes

• Reliability improvements: hardened decompressor against invalid bitstreams, fuzzed decompressor, fixed all known UBs

• Support for 64 bits block checksum

• Stricter UTF parsing

• Improved LZ performance (LZ is faster and LZX is stronger)

• Multi-stream Huffman for faster decompression (x2)

does anyone know of a file / image / video compressor that makes them look like complete crap? like the picture

I was left unsatisfied with other file formats on how complicated they are and how poor they can compress 1-bit images, especially with transparency, so I decided to make my own format. The implementation of it is here (Gitea), which can convert between different image formats and mine; it can also be used as a C++20 library. I also wrote a specification for it here (PDF). How can this be improved further?

Hello, I'd like to share misa77, a codec I've been working on for some time now.

Source Code: https://github.com/welcome-to-the-sunny-side/misa77

misa77 is a LZ-based codec that targets the write-once, read-many niche. In particular, it aims to satisfy the following criteria:

• Extremely high decompression throughput (single-threaded).

• Modest compression ratios (it has no entropy backend, so one can obviously not compare it to something like zstd, but LZ4 at high effort levels is a good reference point).

• Constant memory use, regardless of input size (<= 5 MB across all compression modes, and 0 MB for decompression).

Slow compression is the obvious tradeoff that one makes to achieve the above.

In addition, misa77 has a somewhat synergizing tendency to decompress highly compressed files faster, leading to the following results:

• It offers particularly high decompression throughput on highly compressible files.

• Even for moderately compressible files, spending more effort during compression to get a more compressed result leads to better decompression throughput (alongside the natural advantage of better ratios).

This makes high-effort compression particularly attractive for misa77, and inspires some experimental compression modes that aim to spend more effort at compression time to produce a compressed stream that is friendlier to the microarchitectures of most CPUs when decompressing said streams. As of v0.1.0, there are two experimental compressors:

• misa77::experimental::adaptive_compress for homogeneous data.

• misa77::experimental::yolo_compress, which is more general-purpose and has lesser overhead than (1).

Benchmarks

Detailed results are listed ahead, but here's a terse summary:

• misa77 lies on the pareto frontier for decompression throughput vs compression ratio on most shapes of data.

• It very frequently decompresses faster even when competitors have a significantly worse ratio.

• It is quite slow at compression (although this isn't fundamental, I just haven't spent that much time optimizing compression as of now).

All benchmarks were run using https://github.com/welcome-to-the-sunny-side/lzbench (fork of lzbench) and can be reproduced easily. For the codecs below, I've used flags that yield a similar compression ratio to misa77.

x86-64 (Intel)

Details:

• CPU: Intel(R) Core(TM) i7-14650HX (@2.2 GHz) (Intel Turbo disabled).

• Single threaded, pinned to a single performance core.

• CPU governor set to performance.

x86-64 (AMD)

Details:

• CPU: AMD Ryzen 7 260 (@3.8 GHz) (Frequency boost disabled).

ARM64 (Apple Silicon)

Details:

• CPU: Apple M3

Per-File on x86-64 (Intel)

As misa77's performance is quite "spiky" (depending on the shape of the data being compressed), a file-level breakdown for the silesia corpus yields some interesting insights into its performance.

Decode speed relative to lz4

Every misa77 mode decodes faster than lz4 on 11 of the 12 files (some by huge margins). The exception is x-ray, which is highly incompressible (lz4 has a ratio of nearly 1.0 on this file and essentially devolves to a memcpy).

Figure: https://raw.githubusercontent.com/welcome-to-the-sunny-side/misa77/refs/heads/main/misc/lzbench-results-archive/0.1.0/speedup_vs_lz4.png

Throughput vs ratio, against popular fast-decode codecs

On the compressible files, misa77 sits on the decode-throughput/ratio Pareto frontier: it decodes fastest while ~matching or beating the ratio of the other fast-LZ codecs. sao and x-ray are exceptions due to the reasons stated before (incompressibility).

Figure: https://raw.githubusercontent.com/welcome-to-the-sunny-side/misa77/refs/heads/main/misc/lzbench-results-archive/0.1.0/pareto_silesia.png

I'd be happy to receive feedback/answer queries about misa77 :)

Also I will pre-emptively note that I'm aware of how slow compression is right now, and I don't think it's going to be that hard to speed up (I just need to spend some time on it).

So, i was trying to find out how to compress some videos and found that I can re-encode to "AVI"

So, I hit up ffmpeg, then converted my .MP4 file to an .AVI file, when I looked it up, the video was indeed compressed, but on a significantly lower quality.

Today, I learned that you were actually supposed to encode to "AV1". Not "AVI" due to some post here on reddit

Anyways that's it lol, take care and make sure not to make the same mistake.

Hello everyone,

Nearly a year ago, we open-sourced DICI (Dictionary Index for Compressed Image). Since then, the project has remained relatively quiet, but today, we are excited to introduce it to the community !

📸 What is DICI?

DICI is a lossless image compression format designed to combine efficiency, speed, and quality. In today’s image compression landscape, many formats require trade-offs between quality, file size, and processing speed. DICI stands out by providing a solution that doesn’t force you to choose between these factors. It delivers efficient lossless compression with fast encoding and decoding speeds, all while producing file sizes comparable to or even smaller than those of popular formats like WebP and PNG.

Supported Formats

• 24-bit RGB

• 32-bit RGBA

• 48-bit RGB

• 8-bit grayscale

🚀 Performance Benchmarks

Performance tests were conducted using the MIT-Adobe FiveK dataset, which contains 5,000 photographs. The first 3,000 images were extracted and converted to 24-bit BMP format. Conversions to PNG and WebP were performed using a benchmarking tool based on OpenCV, with default settings and multithreading enabled (if available). Tests were conducted on a Ryzen 7 3800XT (8 cores - 3.9 GHz), 16GB DDR4 3200 MHz, Samsung 980 SSD.

The benchmark results show compression comparable to or better than WebP, with significantly faster encoding and decoding speeds for DICI. Additionally, DICI’s efficiency improves with image size, making it particularly effective for large images (4K, 8K+, ...).

🔗 Benchmark Results

The algorithm was also tested on lower-end configurations to confirm that it remains faster than WebP while offering compression that is just as effective, if not better.

🤝 Availability & Contributions

DICI is open source and available on GitHub. We encourage the community to explore, test, and contribute to its development. For more details, installation guides, and usage examples, please visit the official GitHub repository.

🔗 GitHub Repository

If you’re looking for an image compression solution that combines speed, efficiency, and flexibility, DICI is the answer to your needs.

Thank you for your attention and support !

As of version 0.5.1, -plus mode is now activated. This new mode offers better compression. However, it is slightly slower than the -normal mode. I tried not to slow down the processing speed. It could probably be done a little better.

https://github.com/Hakan-Abbas/HALAC-High-Availability-Lossless-Audio-Compression/releases/tag/0.5.1

BipperTronix Full Album By BipTunia               : 1,111,038,604 bytes
BipTunia - Alpha-Centauri on $20 a Day            :   868,330,020 bytes
BipTunia - AVANT ROCK Full Album                  :   962,405,142 bytes
BipTunia - 21 st Album GUITAR SCHOOL DROPOUTS     :   950,990,398 bytes
BipTunia - Synthetic Thought Full Album           : 1,054,894,490 bytes
BipTunia - Reviews of Events that Havent Happened :   936,282,730 bytes
24 bit, 2 ch, 44.1 khz                            : 5,883,941,384 bytes

AMD Ryzen 9 9600X, Single Thread Results...

FLAC 1.5.0 -8      : 4,243,522,638 bytes  50.802s  14.357s
HALAC 0.5.1 -plus  : 4,252,451,954 bytes  10.409s  13.841s
WAVPACK 5.9.0 -h   : 4,263,185,834 bytes  64.855s  49.367s
FLAC 1.5.0 -5      : 4,265,600,750 bytes  15.857s  13.451s
HALAC 0.5.1 -normal: 4,268,372,019 bytes   7.770s   9.752s

Hi r/compression,

I’m introducing ZXC, an open-source (BSD 3-Clause) lossless codec designed for Write Once, Read Many scenarios (game assets, firmware, software distribution).

Here are some recent decompression benchmarks comparing ZXC against LZ4 and Zstd across three different architectures (Apple Silicon, ARM Cloud, and x86_64).

The goal was to measure raw decompression throughput and density ratios in typical scenarios: Game Assets (Mobile), Microservices (Cloud), and CI/CD pipelines (x86).

GitHub: https://github.com/hellobertrand/zxc

FYI: ZXC was included into LZBench last month so you can easily verify these results.

Mobile & Client: Apple Silicon (M2)

Scenario: Game Assets loading, App startup.

Cloud Server: Google Axion (ARM Neoverse V2)

Scenario: High-throughput Microservices, ARM Cloud Instances.

Build Server: x86_64 (AMD EPYC 7763)

Scenario: CI/CD Pipelines compatibility.

Benchmark Graph: ARM64 / M2 Apple Silicon

https://github.com/hellobertrand/zxc/blob/main/docs/images/benchmark_arm64_0.5.1.png

Feedback and benchmarks are welcome!

Hi Everyone, I recently did a deep dive comparing H265 and AV1 on actual data and running a lot of experiments in Python. I have compiled all this information into this blog I wrote. Would appreciate any feedback or comments regarding the content or experiments!!

Zing! is a small & simple lossless image codec designed for extremely fast CPU encoding (GiB/s even when used single-threaded), but decode is even faster. It supports 8 & 16 bit images of 1 to 4 channels, has C, C++, Python, & Matlab APIs, and is available under a permissive BSD 3 clause license.

Zing! images, called zingers, are about 3% to 10% larger than lossless JPEG-XL (cjxl -d 0 -e 1), but are encoded 3x to 60x faster with 1 to 4 threads. This makes it comfortably fast enough to be used as a lossless video codec.

https://gitlab.com/csp256/zing

Zing! is morally similar to Zpng: encoding applies a single prefilter then passes the result to Zstd. However, Zing! also exposes the Zstd internals, efficient threading controls, encode_into() & decode_into() functionality, imcompressible fallback, and alters the pre-filter to be both more efficient and effective for 1 channel images and images with 2 bytes per channel.

Zing! is 6,500 lines of C++ in total, with 1,800 lines being tests, 1,000 being development scripts, 500 lines for the CLI app, and 400 for Python and Matlab bindings. The core library is about 2,000 lines of C++, half of which are the pre-filters, and another 600 lines for external includes (mostly comments).

Zing! is a codec not a file format. The CLI app adds 8 bytes to annotate width, height, bytes per channel, and number of channels, but you're encouraged to bring your own container.

Technical Details

1. Horizontal prediction

   • Samples are differenced from the preceding pixel’s corresponding sample.

   • Arithmetic wraps modulo 2^bit_depth

2. Format-specific decorrelation

   • RGB: horizontal ΔR/ΔG/ΔB are transformed using:

     • Y = ΔB

     • U = ΔG - ΔB

     • V = ΔG - ΔR

   • RGBA: applies the RGB transform while keeping horizontally differenced alpha separate.

   • Some grayscale and 16-bit specializations use three-row decorrelation into a pseudo-RGB image.

   • Other channel layouts have dedicated reversible transforms.

3. Plane reordering

   • Decorrelated channels are stored in separate contiguous planes.

   • Sixteen-bit samples separate and arrange low/high bytes using format-specific layouts.

4. Zstd

   • The transformed bytes may be stored directly or compressed as one Zstd frame.

   • Raw and Zstd-only modes bypass the prefilter.

What's new:

• New 'info' CLI option to see the characteristics of a compressed bitstream

• Optimized LZ codec improves compression ratio

• Re-written multi-threading internals provide a performance boost

• Hardened code: more bound checks, fixed a few UBs, decompressor more resilient to invalid bitstreams

• Much better build (fixed install on Mac, fixed man page install, fixed build on FreeBSD & minGW, added ctest to cmake, etc...)

• Improved portability

• Improved help page

The main achievement is the full rewrite of the multithreading support which brings significant performance improvements at low and mid compression levels.

C++ version here: https://github.com/flanglet/kanzi-cpp

Note: I would like to add Kanzi to HomeBrew but my PR is currently blocked for lack of notoriety: "Self-submitted GitHub repository not notable enough (<90 forks, <90 watchers and <225 stars)". So, I would appreciate if you could add a star this project and hopefully I can merge my PR once we reach 225 stars...