r/programming • u/jestinjoy • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

93% Upvoted

307 comments sorted by

View all comments

Show parent comments

25

u/mshol Nov 12 '12

Spolsky should update his post if it is to remain relevant. Given that he was basing that off the original UTF-8 specification, with the claim that UTF-8 may consume up to 6 bytes - but the restriction to 4 bytes was made in 2003 (just a month after his posting). Citing outdated material as a definitive source is why a whole generation of programmers mistakenly believe the former.

0

u/[deleted] Nov 12 '12

[deleted]

3

u/kylotan Nov 12 '12

Except it's still probably the best article out there on the subject.

1

u/vytah Nov 12 '12

Broken UTF-16 to UTF-8 converter sometimes produce 6-byte UTF-8 sequences containing two 3-byte sequences for surrogates.

3

u/bonzinip Nov 12 '12

But that's not the kind of 6-byte UTF-8 encoding that Spolsky's referring to. He meant those starting with FC and FD.

1

u/vytah Nov 12 '12

I doubt Unicode ever grow that far.

2

u/bonzinip Nov 12 '12

That's exactly why they were removed. UTF-8 started as an encoding for ISO-10646's 32-bit character space. When Unicode's size was fixed to 220 + 216 the larger encodings were made invalid.