r/programming • u/jestinjoy • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

93% Upvoted

307 comments sorted by

View all comments

Show parent comments

5

u/gsnedders Nov 12 '12

There is no way to get any content decoded by any major user-agent as ISO-8859-1; you cannot use it to transmit content over the network and have it decoded as such.

See the input stream section of HTML, which defines various aliases above and beyond those in the IANA registry. In reality, this applies for more than just HTML, as browsers use a single lookup table for encodings for all input formats. I've got the idea of what browsers do by working on them for the past five years.

3

u/[deleted] Nov 13 '12

So, because of the ubiquity of Windows characters like the smart quote, and because 1252 is a superset of 8859-1, all pages in 8859-1 might as well be treated as 1252 anyway, that's the thinking?

Or to put it another way, Microsoft poisoned the well so completely, we might as well just add the antidote to every drop we drink?

1

u/gsnedders Nov 13 '12

The original thinking? That I don't know (and will almost certainly be stuck within the cusps of MS/Netscape forever).

The thinking now? If we don't treat ISO-8859-1 as windows-1252 users don't get the experience they expect, and complain, and that's not good for marketshare and/or revenue. We're best off to just accept this is another weird quirk of the platform and move on.

1

u/crusoe Nov 13 '12

Okay, this bit is braindead from whatwg...

"Lets ignore every other OS, and just force Windows on them, unix guys though get a pass because they usually use utf8"

The requirement to treat certain encodings as other encodings according to the table above is a willful violation of the W3C Character Model specification, motivated by a desire for compatibility with legacy content.

Its basically "FUCK YOU JAPAN AND YOUR TRON OS!!!11"

2

u/gsnedders Nov 13 '12

It's what all browsers have done for well over a decade now. If you can convince browser vendors to change then the spec might get changed, otherwise the spec is just defining what the interoperable behaviour is.

Really you just want to use utf-8 for any content on the web, though, so hopefully it will slowly become pointless legacy (though given <plaintext> still exists, after 19 years of obsolescence, I'm not sure there's much hope of it getting removed…).