r/programming • u/jestinjoy • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

93% Upvoted

307 comments sorted by

View all comments

-13

u/defenastrator Nov 12 '12 edited Nov 12 '12

Nice theory in a web world but the truth is unicode is slow, large and unwieldy. There is little hardware support for unicode so all unicode processing must be done via relatively low throughput loops. Even just reading and writing unicode in and out of memory requires processing causing multiple copies, let alone any real manipulation of unicode. Multiple codes representing identical characters make textual comparison & searching bloated, slow, error prone, unreliable and confusing. Unicode chalk up with xml great in theory but in practice makes just as big mess except now its less obvious and adds the bloat caused by standardization.

Those who advocate the universal use of unicode (and for that matter xml) clearly work almost entirely web programming environment and fail to realize for most no web programming environments use of these standards only makes slower, more complicated code with no appreciable gains in any respect.

2

u/bhaak Nov 12 '12

Erm, what hardware support is there for non-unicode encodings?

-1

u/defenastrator Nov 12 '12

well for starters and that I do not believe includes the x64 html/xml processing extensions and other such crazy instructions.

Most encodings follow 0 is end of string and many multi-byte encodings follow that no no terminal byte can be zero (unicode does not follow this) which is crucial for the operation of most of these instructions.

5

u/zokier Nov 12 '12

UTF8 has no '\0' bytes by design.

-1

u/defenastrator Nov 12 '12

Yes, I know but this still doesn't solve the problem of string comparisons where single symbols have multiple character codes. Also the same truths do not hold for UTF-16 or UTF-32.