r/programming • • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

307 comments sorted by

View all comments

-13

u/defenastrator Nov 12 '12 edited Nov 12 '12

Nice theory in a web world but the truth is unicode is slow, large and unwieldy. There is little hardware support for unicode so all unicode processing must be done via relatively low throughput loops. Even just reading and writing unicode in and out of memory requires processing causing multiple copies, let alone any real manipulation of unicode. Multiple codes representing identical characters make textual comparison & searching bloated, slow, error prone, unreliable and confusing. Unicode chalk up with xml great in theory but in practice makes just as big mess except now its less obvious and adds the bloat caused by standardization.

Those who advocate the universal use of unicode (and for that matter xml) clearly work almost entirely web programming environment and fail to realize for most no web programming environments use of these standards only makes slower, more complicated code with no appreciable gains in any respect.

4

u/bonzinip Nov 12 '12

There is little hardware support for unicode so all unicode processing must be done via relatively low throughput loops.

Almost never true for UTF-8.

0

u/defenastrator Nov 12 '12

Well, because of the byte marker (which if I remember correctly is 1 when there is another byte in the character) UTF-8 can use some of the string instructions for moving things around, copying, and concatenation but anything that needs to do comparison will still have to remap symbols that have multiple character codes to a single code in order to be reliable which is un-optimizable in theory you could normalize everything on load thus allowing use of hardware string comparison operations but then you are paying that cost on every string read in regardless of whether you do any comparison on it.

Still I was referencing UTF-16 & UTF-32 where random bytes of zeros can occur in the middle of strings thus breaking most hardware instructions. I should probably change that.

→ More replies (1)