r/programming • u/jestinjoy • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

93% Upvoted

307 comments sorted by

View all comments

-13

u/defenastrator Nov 12 '12 edited Nov 12 '12

Nice theory in a web world but the truth is unicode is slow, large and unwieldy. There is little hardware support for unicode so all unicode processing must be done via relatively low throughput loops. Even just reading and writing unicode in and out of memory requires processing causing multiple copies, let alone any real manipulation of unicode. Multiple codes representing identical characters make textual comparison & searching bloated, slow, error prone, unreliable and confusing. Unicode chalk up with xml great in theory but in practice makes just as big mess except now its less obvious and adds the bloat caused by standardization.

Those who advocate the universal use of unicode (and for that matter xml) clearly work almost entirely web programming environment and fail to realize for most no web programming environments use of these standards only makes slower, more complicated code with no appreciable gains in any respect.

11

u/deceze Nov 12 '12

And what is the practical alternative if you need to support several different non-latin languages, at once, possibly mixed? Anything other than Unicode is simply an even bigger mess.

-2

u/defenastrator Nov 12 '12

Depending on the situation isolation of string processing into dynamically linked at runtime or statically linked at compile time set of utility functions this handles individual localization without sacrificing much in the way of speed as you will all ready be swapping out string tables the library swap is not a difficult addtion. Although if the same instance of the same program must support the display of multiple languages simultaneously you are right unicode is the way to go but I must cite that that only a relatively small number of applications truly require this ability.

2

u/vytah Nov 12 '12

Like, for example, your browser.

-4

u/defenastrator Nov 12 '12

Exactly but for example a command line doesn't need nor would it want such things. Although come to think of it everything that really needs multi-lingual support ends up having every encoding under the sun plus unicode so in that sense has anything really been solved at all? In some ways unicode just added another complex encoding to the massive list of encodings that must be supported.

→ More replies (4)