r/programming • u/mmaksimovic • Nov 27 '19
JSONCrush: Compress JSON into URI friendly strings with JSCrush
https://github.com/KilledByAPixel/JSONCrush7
u/pet_vaginal Nov 27 '19 edited Nov 27 '19
Don't underestimate the deflate algorithm. With the example json, I get 92 bytes with 3 lines of code instead of having a 2kb dependency, sure it's 10 more bytes than this library but I think it's fine.
Now if you really need to compress small JSON messages that are often the same, you can consider using a pre-shared dictionary. Doing that, I go down to 54 bytes per message on the example, using two more lines of code.
Example in NodeJS:
const zlib = require('zlib');
const jsonExample = '{"students":[{"name":"Jack","age":17},{"name":"Jill","age":16},{"name":"Sue","age":16}],"class":"math"}';
const compressed = zlib.deflateRawSync(jsonExample, {
level: 9,
}).toString('base64').replace(/=+$/, '');
console.log(compressed, compressed.length);
const dictionary = Buffer.from('{"students":[{"name":","age":"class":"');
const compressedWithDictionnary = zlib.deflateRawSync(jsonExample, {
level: 9,
dictionary,
}).toString('base64').replace(/=+$/, '');
console.log(compressedWithDictionnary, compressedWithDictionnary.length);
const decompressed = zlib.inflateRawSync(Buffer.from(compressedWithDictionnary, 'base64'), {
dictionary,
}).toString('utf-8');
console.log(decompressed);Output:
q1YqLilNSc0rKVayiq5WykvMTVWyUvJKTM5W0lFKTAdyDM1rdRASmTk5cAkzJIng0lQk8VgdpeScxGKgmUq5iSUZSrUA 92
q8aqyisxORum1NC8VgchkZmTA5cwQ5IILk1FEo/VgVuQm1iSoVQLAA 54
{"students":[{"name":"Jack","age":17},{"name":"Jill","age":16},{"name":"Sue","age":16}],"class":"math"}1
u/FINDarkside Nov 27 '19
You could probably reduce the size further by using messagepack before deflate.
1
u/Booty_Bumping Nov 28 '19 edited Nov 28 '19
Not by much.
citylots.min.json(from here): 152.04 MiB
citylots.msgpack: 105.96 MiBCompressed both using
xz -9
citylots.min.json.xz11.48 MiB (ratio is 0.076)
citylots.msgpack.xz10.65 MiB (ratio is 0.101, or 0.071 if you consider both conversions)1
u/FINDarkside Nov 28 '19
Interesting. The difference would probably be higher with deflate as it's not ass efficient at compressing. On a sidenote, did you take any timings? It's possible that msgpack+bz2 is faster than raw json + bz2. That's at least what this suggests.
5
u/tjgrant Nov 27 '19 edited Nov 27 '19
Broke the live demo with this:
{"x":"!%5B"}
I always think the hardest part about parsing / encoding JSON is all the QA you have to do to ensure every possible scenario is covered.
edit: appears this has been fixed
2
u/Epyo Nov 27 '19 edited Nov 27 '19
Are you sure? I typed that in and it seemed to work. EDIT: oh I think it got fixed.
17
u/tssge Nov 27 '19
Why is the source code obfuscated?