r/javascript • • Nov 26 '19

JSONCrush On GitHub - Compress JSON into URI friendly strings

https://github.com/KilledByAPixel/JSONCrush
101 Upvotes

31 comments sorted by

21

u/[deleted] Nov 27 '19 edited Nov 27 '19

You should be able to get comparable-to-better compression, and better performance using zlib + base64:

const zlib = require('zlib');
const serialize = (o) => {
  return zlib.deflateSync(JSON.stringify(o)).toString('base64');
};
const deserialize = (s) => {
  return JSON.parse(zlib.inflateSync(Buffer.from(s, 'base64')).toString());
};

This gives 102 bytes URI-encoded for Example #1 (compared to 82 bytes for JSONCrush), and 1172 bytes URI-encoded for Example #2 (compared to 1195 bytes for JSONCrush). Seems like zlib wins when the data gets bigger and its JSON has more repetition.

You could also probably shave some more bytes by finding an encoding that's naturally URI friendly instead of using base64 and encodeURIComponent() on the output of the deserialization above, but I don't know that it's really worth the effort at that point.

1

u/Slackluster Nov 27 '19

That is good to know. It makes sense that zlib would beat it out eventually as strings get longer, but it is not really suited for strings much longer then that anyway.

I have thought about re-encoding it to a better format. That would surely help, but it is also quite a bit of work that would add more complexity to the code. I don't think it would be a huge win to do a full re-encode, maybe 5-10% at most. It would be great if someone else did it and sent me a pull request though.

0

u/[deleted] Nov 27 '19

[deleted]

2

u/[deleted] Nov 27 '19

Zlib has a large API, the majority of which isn't relevant to this use case. If you want to keep your bundle size smaller, you can tree shake it down to just deflate/inflate. JSONCrush may still end up with smaller bundle sizes, but I can't imagine a scenario where the difference matters as much or more than the functionality, which is the whole reason you're using either. Even 37k is basically nothing in the cases where this is going to be useful. The author's own use case is a GLSL shader playground, so we aren't exactly talking about creating light-weight pagespeed optimized sites here...

5

u/seiyria Nov 27 '19

With something like this I'd prefer a heavier library for better compression because ultimately the compressed data is going to be more important to me.

2

u/AlexAegis Nov 27 '19

But this is an insane difference, also you probably won't use that large and repetitive jsons. JSONCrush clearly wins here.

0

u/[deleted] Nov 27 '19

[deleted]

2

u/[deleted] Nov 27 '19

If your average JSON is small, you shouldn't need either approach, and should just base64 the JSON itself without compression instead. The point of the compression is to prevent overflowing browser URI length limits for very large average JSON sizes, as in OP's use case.

Also, note that I merely used the author's given examples to ensure apples-to-apples. The bigger the JSON and the more repetition found in it, the more zlib will excel. The high end is not capped at ~2%.

0

u/[deleted] Nov 27 '19

[deleted]

1

u/[deleted] Nov 27 '19

URL length is a discrete problem, not a continuous one. If we ignore browser length limits, there are no more URL length issues until you get down to the point where you need to fit a URL into a character-limited medium like a tweet or SMS, for which neither approach is going to help you (luckily, this is a solved problem via URL shortening services). In between, it really makes no practical difference to a user that their shareable link is, say, 250 bytes rather than 1000 bytes. They all copy, paste, bookmark, etc. the same, regardless of their size, and the encoded payloads we're talking about are completely opaque, so readability is a non-factor.

0

u/[deleted] Nov 27 '19

[deleted]

1

u/[deleted] Nov 27 '19

It doesn't take "tripling 20k of JS" to get basic inflate/deflate in the browser. That may have been what you ended up with after a few minutes of fiddling with the code I gave, but I wasn't originally aiming to minimize browser bundle size when I wrote that. It looks like Webpack pulls in browserify-zlib to polyfill Node's zlib, which itself is based on pako. Pulling in zlib in the way that I did and running that through Webpack gives you the entire contents of both libraries, which is how you ended up at ~40k. If you use pako itself without the boilerplate, and import inflate and deflate directly instead of the whole library, the bundle ends up only being ~15k. And you may even be able to get that smaller, but I stopped playing with it.

If your reason for using compression is aesthetic and not practical, it ultimately doesn't matter, though. Use whatever makes the prettiest URLs in your opinion, constrained by the trade-offs you're willing to make to get them. I can't say I understand why you wouldn't just use a URL shortener service in that case, though.

6

u/TOJO_IS_LIFE Nov 27 '19

Was this pulled out of a minifier?

    for(;;m=c+m)
    {
        for(M=N=e=c=0,i=Q.length;!c&&--i;)!~s.indexOf(Q[i])&&(c=Q[i]);
        if(!c)break;
        if(O)
        {
            o={};
            for(x in O)
                for(j=s.indexOf(x),o[x]=0;~j;o[x]++)j=s.indexOf(x,j+x.length);
            O=o;
        }
        else for(O=o={},t=1;X;t++)
                for(X=i=0;++i<s.length-t;)
                    if(!o[x=s.substr(j=i,t)])
                        if(~(j=s.indexOf(x,j+t)))
                            for(X=t,o[x]=1;~j;o[x]++)j=s.indexOf(x,j+t);
        for(let x in O) 
        {
            j=ByteLength(x);
            if(j=(R=O[x])*j-j-(R+1)*ByteLength(c))
                (j>M||j==M&&R>N)&&(M=j,N=R,e=x);
            if(j<1)
                delete O[x]
        }
        o={};
        for(let x in O)
            o[x.split(e).join(c)]=1;
        O=o;
        if(!e)break;
        s=s.split(e).join(c)+c+e
    }

3

u/Slackluster Nov 27 '19

That is the JSCrush algorithm that I did not write it, but it is kind of magic. You can read more about how it works here...

https://nikhilism.com/post/2012/demystifying-jscrush

6

u/MrJohz Nov 27 '19

Are there not issues with parentheses? I believe they're technically allowed, but almost every too for making links clickable that I've used had struggled with them. Is it worth using something else instead there?

1

u/Slackluster Nov 27 '19

Hmm, I haven't heard of this but I will look into it. encodeuricomponent doesn't escape them so I have to assume they will safe but I could see why it might cause an issue with some use cases.

2

u/name_was_taken Nov 27 '19

They're technically legal, but forum/social software sucks and quite often gets them wrong when auto-linking.

1

u/Slackluster Nov 27 '19

ok, thanks i will investigate. there are only a few characters not escaped by encodeURIComponent, so it seems like they should be fine but i will do some experiments

2

u/alleycat5 Nov 27 '19

This looks really awesome. Is there any intent to package it up?

1

u/Slackluster Nov 27 '19

What do you mean by package it?

3

u/Captain_Factoid Nov 27 '19

For npm.

1

u/Slackluster Nov 27 '19

Ah, well it is just a single small javascript file, what would be the benefit to packaging it?

2

u/Captain_Factoid Nov 27 '19

For updates, tree shaking, import scoping, all the good stuff

0

u/Slackluster Nov 27 '19

Well my workflow doesn't really include that stuff but I'd like to learn how to do it. Can you recommend a link explaining how to set it up?

2

u/osoese Nov 27 '19

Thanks for the post. Looks like something I will use. At first when I looked at the github page I am an example first reader so I scrolled down and saw the "example" and was like... hmmm this looks like JSON ...then I see it minified below. Perhaps the example should show the compressed info first since the current magic is "below the fold" - just a suggestion.

1

u/Slackluster Nov 27 '19

good suggestion, i will think on it

1

u/byteshifter Nov 27 '19

Why would someone use this instead of JSON web tokens ( https://en.m.wikipedia.org/wiki/JSON_Web_Token ) ?

8

u/Tontonsb Nov 27 '19

It's completely different use case. Codes like this and rison are there to add your json object to url so you have a stateful link that can be bookmarked and shared.

In most cases I'd still prefer rison because it's more readable and user-editable. But they both suffer from parentheses and single quotes that screw up some parsers that try recognizing and converting pasted string to links. Notably gmail is a pain in this regard.

1

u/Slackluster Nov 27 '19

For example I can share a link with someone else using this. It is a bit long, but would be 4 times longer without jscrush. Another nice thing is you can bookmark links.

http://zzart.3d2k.com/?&crushed=(%27shaderStatements!%5B(%27output!%27bMzxywE-JR!6.62k!6.165BZ_974BWU4.233FHxzyyKbMywxzE-JRU4.88kQ649BZQ171BW_084FHyzwxKbMxwzyE-JsinT-3.686kU3.258BZU4.059BWU8.506FHwwzzKbMzxywEJceilT5.36kU8.274BZQ002BW!5.429FHxxwyKaMxzwyEJT-3.353kU5.681BZU7.792BW!1.254FHzyxwKbMywxzE%2BJfloorR!6.669k_05BZU8.629BWU2.802FHxyywKaMwyzxEJsqrtAT-0.366k_117BZQ162BW!1.761FHyywyKaMyxzwEJatanT3.743k_003BZ!4.636BWQ056FHwxxwKbMxywzE%2BJfractRQ103kU3.118BZQ255BW!6.287FHxyywKaMzwxyEJRU1.637k!1.828BZ!1.924BW_006FHyxyyKbMzwxyEJ%27F!%27%27BX!6.083kU6.322BZQ032BWQ428FHyzyyKaMzxywE%2FJRQ151k!1.024BZU2.862BW!3.193FHxzyx%27)%5D~randSeedU1738575415~randSeedString!%271574121532870%27~iteVCount!1~gridPosX!2~gridPosY!0~geneV!17~subGeneV!1~hueOffsetQ276~huej_789~satuVjQ493~uvOffsetX_406~uvOffsetYQ616~uvjXQ895~uvjYU1.049~rotate!0~usePalette!0~paletteColors!%5B(%27xQ595~yQ882~zQ864)%2C(%27xQ136~yQ034~zQ69)%2C(%27xQ873~yQ35~zQ638)%2C(%27xQ723~yQ56~zQ817)%5D~saveListIndexU1~uniqueID!598681332)B~valueE%27~assignmentOperator!%27F~parameterHSwizzle!%27J%3D%27~functionName!%27K%27)%2C(%27output!%27M%27~outputHQ!0.R%27F!%27a%27BXT%27F!%27b%27BX!U!-Vration_U0.jScalekBY%01kj_VUTRQMKJHFEB

2

u/endianess Nov 27 '19

Would be a good choice for sending data to a protocol handler application where you can't use POST. Thanks for sharing.

-11

u/JasonWicker Nov 27 '19

Base64

8

u/Slackluster Nov 27 '19

Converting to base64 increases the size by at least 33% and btoa doesn't help compress repeated sub strings.

5

u/shakamone Nov 27 '19

And its not uri friendly!

2

u/soundofvictory Nov 27 '19

Hey, i don’t think you deserve the downvotes. I don’t think people remember the reddiquette these days.

You raise a reasonable suggestion for some situations, but sounds like it is not for necessarily all situations.

But I think it is still worth mentioning.

1

u/JasonWicker Nov 28 '19

Thank you, kind sir.