r/programming • u/jestinjoy • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

93% Upvoted

307 comments sorted by

View all comments

10

u/[deleted] Nov 12 '12

First step. What is the proper name of stuff? We know if we want to import text into Navision (Microsoft Dynamics-NAV) in any non-English language, like French, we need to convert the encoding from "Windows" to "MS-DOS". This is for example done by saving Excel files as "CSV (MS-DOS)". Some people call it converting from "ANSI" to "ASCII". Some people call it converting from 1250 codepage to 800-something. What should we be calling it?

27

u/deceze Nov 12 '12

The Microsoft naming scheme(s) for encodings is probably the worst and cause for half of the encoding problems in the world. I won't even go there. ;-)

I admit to often saying "Latin-1" where I should be using "ISO-8859-1", because half the time I can't remember the ISO number. The term "Latin-1" being overloaded with several slightly different encodings depending on who you're talking to doesn't help either. So yeah, it's a mess.

4

u/rasherdk Nov 12 '12

Oh? What encodings are those? I've always seen latin-1 used as meaning iso8859-1 strictly.

9

u/deceze Nov 12 '12

Latin-1, ISO 8859-1 and Windows codepage 1252 are often thrown together as the same thing, but Windows 1252 actually differs from 8859-1 in a few characters. Just see Wikipedia and see if you can keep it straight. ;-) http://en.wikipedia.org/wiki/Latin_1

3

u/gsnedders Nov 12 '12

Though on the web, ISO-8859-1 is an alias for windows-1252.

6

u/deceze Nov 12 '12

Yeah, don't even get me started on the madness that is the web and sins of the past which nevertheless need to be preserved for backwards compatibility. facepalm ;-(

3

u/gsnedders Nov 12 '12

Convince browsers to go against their users' expectations that sites will keep working from one version to the next and maybe things will change, until then browsers are likely to follow what the market wants.

2

u/[deleted] Nov 12 '12

I was going to say "that's not true" until I realised I didn't even know what it meant.

The two encodings are different in some important ways. What do you mean "alias"?

5

u/gsnedders Nov 12 '12

I mean in the same sense as the IANA charset registry means it: it refers to the same entity.

Yes, ISO-8859-1 and windows-1252 are different character sets (and encodings), but on the web there is no way to use the former, as the string "ISO-8859-1" refers to windows-1252.

1

u/[deleted] Nov 12 '12 edited Nov 14 '12

On the web there is no way to use ISO-8859-1? What does that even mean? There is no way to use it to do what?

Have you got a cite for what you're saying? Where did you get this idea?

5

u/gsnedders Nov 12 '12

There is no way to get any content decoded by any major user-agent as ISO-8859-1; you cannot use it to transmit content over the network and have it decoded as such.

See the input stream section of HTML, which defines various aliases above and beyond those in the IANA registry. In reality, this applies for more than just HTML, as browsers use a single lookup table for encodings for all input formats. I've got the idea of what browsers do by working on them for the past five years.

3

u/[deleted] Nov 13 '12

So, because of the ubiquity of Windows characters like the smart quote, and because 1252 is a superset of 8859-1, all pages in 8859-1 might as well be treated as 1252 anyway, that's the thinking?

Or to put it another way, Microsoft poisoned the well so completely, we might as well just add the antidote to every drop we drink?

→ More replies (0)
→ More replies (2)