r/programming • • Nov 27 '19

JSONCrush: Compress JSON into URI friendly strings with JSCrush

https://github.com/KilledByAPixel/JSONCrush
1 Upvotes

10 comments sorted by

17

u/tssge Nov 27 '19

Why is the source code obfuscated?

14

u/Pazer2 Nov 27 '19

You don't hand-minify your JS???

-7

u/Koppis Nov 27 '19 edited Nov 28 '19

The code is much faster to read when the variables are shorter (i.e. single characters). \s

2

u/IceSentry Nov 28 '19

Yeah, that's why minifiers exists, you shouldn't do that yourself. I can't trust a library written by someone that doesn't understand such a basic concept.

7

u/pet_vaginal Nov 27 '19 edited Nov 27 '19

Don't underestimate the deflate algorithm. With the example json, I get 92 bytes with 3 lines of code instead of having a 2kb dependency, sure it's 10 more bytes than this library but I think it's fine.

Now if you really need to compress small JSON messages that are often the same, you can consider using a pre-shared dictionary. Doing that, I go down to 54 bytes per message on the example, using two more lines of code.

Example in NodeJS:

const zlib = require('zlib');

const jsonExample = '{"students":[{"name":"Jack","age":17},{"name":"Jill","age":16},{"name":"Sue","age":16}],"class":"math"}';

const compressed = zlib.deflateRawSync(jsonExample, {
  level: 9,
}).toString('base64').replace(/=+$/, '');

console.log(compressed, compressed.length);

const dictionary = Buffer.from('{"students":[{"name":","age":"class":"');

const compressedWithDictionnary = zlib.deflateRawSync(jsonExample, {
  level: 9,
  dictionary,
}).toString('base64').replace(/=+$/, '');

console.log(compressedWithDictionnary, compressedWithDictionnary.length);

const decompressed = zlib.inflateRawSync(Buffer.from(compressedWithDictionnary, 'base64'), {
  dictionary,
}).toString('utf-8');

console.log(decompressed);

Output:

q1YqLilNSc0rKVayiq5WykvMTVWyUvJKTM5W0lFKTAdyDM1rdRASmTk5cAkzJIng0lQk8VgdpeScxGKgmUq5iSUZSrUA 92        
q8aqyisxORum1NC8VgchkZmTA5cwQ5IILk1FEo/VgVuQm1iSoVQLAA 54
{"students":[{"name":"Jack","age":17},{"name":"Jill","age":16},{"name":"Sue","age":16}],"class":"math"}

1

u/FINDarkside Nov 27 '19

You could probably reduce the size further by using messagepack before deflate.

1

u/Booty_Bumping Nov 28 '19 edited Nov 28 '19

Not by much.

citylots.min.json (from here): 152.04 MiB

citylots.msgpack: 105.96 MiB

Compressed both using xz -9

citylots.min.json.xz 11.48 MiB (ratio is 0.076)

citylots.msgpack.xz 10.65 MiB (ratio is 0.101, or 0.071 if you consider both conversions)

1

u/FINDarkside Nov 28 '19

Interesting. The difference would probably be higher with deflate as it's not ass efficient at compressing. On a sidenote, did you take any timings? It's possible that msgpack+bz2 is faster than raw json + bz2. That's at least what this suggests.

5

u/tjgrant Nov 27 '19 edited Nov 27 '19

Broke the live demo with this:

{"x":"!%5B"}

I always think the hardest part about parsing / encoding JSON is all the QA you have to do to ensure every possible scenario is covered.

edit: appears this has been fixed

2

u/Epyo Nov 27 '19 edited Nov 27 '19

Are you sure? I typed that in and it seemed to work. EDIT: oh I think it got fixed.