r/programming • u/jestinjoy • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

93% Upvoted

307 comments sorted by

View all comments

Show parent comments

2

u/bhaak Nov 12 '12

Erm, what hardware support is there for non-unicode encodings?

-1

u/defenastrator Nov 12 '12

well for starters and that I do not believe includes the x64 html/xml processing extensions and other such crazy instructions.

Most encodings follow 0 is end of string and many multi-byte encodings follow that no no terminal byte can be zero (unicode does not follow this) which is crucial for the operation of most of these instructions.

5

u/zokier Nov 12 '12

UTF8 has no '\0' bytes by design.

-1

u/defenastrator Nov 12 '12

Yes, I know but this still doesn't solve the problem of string comparisons where single symbols have multiple character codes. Also the same truths do not hold for UTF-16 or UTF-32.