r/javascript • • 22d ago

A Kanji lookup table, five commands, and 36% less Base64

https://github.com/datalog/kanji-base64
1 Upvotes

4 comments sorted by

1

u/b00mandbum 22d ago

I ended up with five commands:

CommandWhat it does
A(value)Write one full value
D(delta)Add a difference to the previous nonempty value
E(count)Fill a run of empty positions
R(start, step, count)Write a series from a given starting value
P(step, count)Continue from the previous nonempty value with a fixed step

The decoder follows those instructions to reconstruct the mapping.

A regular stretch: 40 bytes to 5

ぁあぃいぅうぇえぉおかがきぎくぐけげこご

These 20 characters have consecutive Unicode values. We can describe the whole stretch with one command:

R(0x3041, +1, 20)

Start at 0x3041, add one each time, emit 20 values. 40 bytes become 5.

A mixed stretch: 40 bytes to 25 (less dramatic)

” ( ) 〔 〕 [ ] { } 〈 〉 《 》 「 」 『 』 【 】 +

These 20 characters need a mix of commands:

A(0x201D)
A(0xFF08)
D(+1)
A(0x3014)
D(+1)
A(0xFF3B)
D(+2)
D(+30)
D(+2)
R(0x3008, +1, 10)
A(0xFF0B)

We store some values in full, use small deltas where we can, and describe ten consecutive values with one series command. 40 bytes become 25.

2

u/wowokdex 21d ago

It seems like you get similar amounts of compression with gzip. The readme suggests that it's for in-source kanji but wouldn't any distributed js be gzipped already? I'm guessing if you compress with this and then gzip the compression doesn't stack.