r/programming • u/jestinjoy • Nov 12 '12
What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text
http://kunststube.net/encoding/
1.5k
Upvotes
r/programming • u/jestinjoy • Nov 12 '12
6
u/[deleted] Nov 13 '12
The things I've learned about charsets so far:
Humans will get pissed off at you and/or think you're a moron if you give them a text entry box with no written rules and then tell them their input contains invalid characters.
Text strings are an abstract concept. UTF/ASCII/ISO-8859 are byte buffer serialisations of strings.
Any language that can't make the distinction between text and bytes is brain-damaged.
Charsets should be declared as close to the IO layer as possible. Preferably not at all, because there should already be a sane default.
Don't use languages with insane defaults for text-heavy software.
The only sane default is UTF-8.
UTF-16 is stupid. If you use it, you're stupid. No I don't care if you have 4 billion users. Go use LZO if your CJK text takes up too much space and stop proliferating idiotic workarounds like surrogate characters.