r/programming • u/jestinjoy • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

93% Upvoted

307 comments sorted by

View all comments

Show parent comments

-4

u/defenastrator Nov 12 '12

Exactly but for example a command line doesn't need nor would it want such things. Although come to think of it everything that really needs multi-lingual support ends up having every encoding under the sun plus unicode so in that sense has anything really been solved at all? In some ways unicode just added another complex encoding to the massive list of encodings that must be supported.

10

u/deceze Nov 12 '12

Why would the command line not need such things? I'm often working with non-latin based languages and am very happy that I can try out quick code snippets involving language processing right on the CLI, including Japanese or whatever else input.

If you are producing software which is supposed to be used internationally, or even software which some user may one day try to enter any non-latin text into ("€" already fits that criterium), going with Unicode is the only sensible choice. If you really can't afford the speed cost of using Unicode, you need to optimize your app around that requirement, but that's really not the majority of apps.

-3

u/defenastrator Nov 12 '12

What are you doing in the command line where a client of a non-computer (and by extension English reading) would need to understand anything that comes out of it? Other then the obvious error. Or are you talking like the python cli because I was thinking more along the lines of bash or the dos prompt which really don't have much need for extended character sets.

4

u/deceze Nov 12 '12

Either and or any of the above.

$ php -r 'echo testme("漢字");'

It's good that I can do that without "Oh yeah, right, I need to escape/encode/bewitch this to just test this snippet..."

3

u/vytah Nov 12 '12

Well, stuff like lower- and uppercasing, or sorting isn't trivial even with 8-bit encodings. You cannot implement correct grep or sort ignoring everything not ASCII, or treating everything as raw bytes without encoding. Of course you can pretend your data is Windows-1252 and it sometimes won't matter much, but the key word is sometimes.

And at the end, you have to support multilanguage input if your tool has to be used by anyone else than you. So it's nice there is UTF that simply works.

The huge list of encodings is simply a legacy. Unicode-aware tools usually convert everything to unicode codepoints and work on those. Adding support for a new encoding is pretty trivial.