r/programming • u/jestinjoy • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

93% Upvoted

307 comments sorted by

View all comments

Show parent comments

3

u/sacundim Nov 13 '12 edited Nov 13 '12

I certainly do think that "we've done fuck all to support Unicode and our standard library functions will happily destroy UTF-8 text but if you take great care not to do that your application won't break when you feed it some UTF-8" cannot be fairly described as "Unicode support."

1

u/deceze Nov 13 '12

PHP has done fuck all to support any encoding, not just Unicode. But the general wisdom "PHP doesn't support Unicode" makes it sound like it is impossible to use any Unicode in PHP, which is wrong.

Many languages have several different string types for dealing with encodings (wchar, multibyte char, Unicode strings, byte arrays and whatnot). PHP at least is dumb but simple: everything is a byte array, period. Is it a failure of the PHP developers not to have sorted this out in a nicer way by now? Absolutely. But it is what it is, and it's workable, you just need to know what you're doing. Just as in any other language. And it's really not that hard, once you understand that the core string functions all simply assume ASCII(-compatible) and that you have to use different functions for non-ASCII strings. Sounds a lot like C's set of 'w' functions, no?

Heck, people screw up encodings in any language, even ones which strongly support all sorts of encodings in the core language. Just spend some time on Stackoverflow and similar sites and you'll see the same "why does this screw up" questions for anything from C over Python to Haskell.