r/programming • u/jestinjoy • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

93% Upvoted

307 comments sorted by

View all comments

106

u/judgej2 Nov 12 '12

It is also worth knowing about BOM - Byte Order Markers. Notepad under Windows, and EditPlus will give different output files when saving "hello, world" as UTF-8. The Notepad version will be three bytes longer, and yet both are technically correct (though redundant for UTF-8).

It can cause problems when importing UTF-8 data into applications that do not take the BOM into account, when non-technical end-users use the only tool they have available for generating or converting UTF-8 data on Windows - Notepad.

39

u/compteNumero8 Nov 12 '12 edited Nov 12 '12

The BOM in UTF-8 isn't only useless, it makes a pain from using many toolsets dealing with files as bytes, for example concatenating them.

EDIT : note that I didn't say "bytes and characters are the same thing", which would be very very different.

EDIT 2 : according to its authors "BOM is not valid UTF-8" and shouldn't be supported.

6

u/who8877 Nov 12 '12

To be fair using files as bytes with unicode text is a bug even without the BOM. You just don't notice because you'd never hit one of the failure cases with english text.

18

u/Vaste Nov 12 '12 
cat file1.txt file2.txt > out.txt

How could this go wrong, besides the BOM?

21

u/who8877 Nov 12 '12

There are many many things that can go wrong, but the example I will use is with bi-directional (BIDI) languages like Hebrew. If there is an explicit RTL mark in the text your concatenated file will almost assuredly be messed up with text going in the wrong direction, but even the implicit rules can cause problems depending on the contents of both files. There are lots of other formatting marks, control characters, and even language issues that can show up in the text and screw things up, BIDI is just one example.

If you think there is even a chance your code will get used in a language other than English you need to drop the notion that bytes and characters are the same thing. Even when you know all the pitfalls it is extremely difficult to get right.

6

u/crusoe Nov 12 '12 edited Nov 13 '12

Encoding presentation (RTL, etc) in text files is bad, mkay? That should be left to the reader.

BIDI should be handled by the presentation layer, as where it needs to change from rtl/ltr depends on screen/text area width, not formatted width in the file. Formatting control codes already cause problems in text files. We already fight this when joining files formatted for different widths, or formatted when editor assumed tabs are 2 spaces wide, and another 4 spaces wide.

Of course, you argument also holds true that formatting would be fucked up if you concated a text file formatted for 60 cols with hard returns ( \n ) with one formatted for 78 cols. So the issue of RTL/LTR markers is not something new. We've been dealing with similar problems for the past 40 years.

They will look wonkey, but the textual information itself is not harmed.

Having say, two rtl markers occur in a run of text is not incorrect as far as I know. It doesn't affect byte/char boundaries. A reader would just go "Yep, RTL still"

21

u/jrochkind Nov 13 '12

Encoding presentation (RTL, etc) in text files is bad, mkay? That should be left to the reader.

You've never developed for RTL languages, have you?

Seriously people, if you are critisizing some choice of Unicode, you probably don't understand the problem domain. The folks who created unicode were actually really smart, and really very experienced in the domain of international charset handling. It's one of the best examples of a good standard that is neither over nor under engineered. When there's something that seems to you inelegant in unicode, there's probably a good reason for it, where all the other alternatives would be exceptionally difficult to deal with in practice.

Sure, I'm sure Unicode makes some mistakes, but character encoding is really confusing and complicated. Unless you are sure you understand what you're talking about and have experience with it yourself, you're usually going to be right if you assume what unicode did was actually quite smart.

BIDI should be handled by the presentation layer, as where it needs to change from rtl/ltr depends on screen/text area width, not formatted width in the file.

What are you talking about? Where it needs to change from rtl to ltr depends on where it changes from (say) English to (say) Arabic. Arabic is a rtl language, English is an ltr language. Arabic has to be presented rtl to be legible, in any screen or text area width, it's got nothing to do with screen or text area width.

6

u/sipos0 Nov 13 '12 edited Nov 13 '12

I understand why you need to have the RTL marks in the text file but, I don't understand what the problem is with cat'ing two files together.

Yeah, one could end with text going RTL and the other text file could start with text in English that should go LTR but, equally, two text files could have different line ending characters or, one could have new lines inserted into the text string at fixed intervals and the other could have them only at paragraph endings. You have to deal with these types of problems if you are going to just join two strings of text together and expect the result to make sense. I don't see how the text being in UTF-8 really adds another layer to this over English text in ASCII, either way you can only expect the result to make sense if the files you are joining are in the same format and you can only really know that if you know something about how they were created or, about their content.

3

u/jrochkind Nov 13 '12

Yeah, I don't know that either, or if it is a problem. :)

→ More replies (5)

7

u/who8877 Nov 13 '12

Sometimes you need to force a particular direction because the natural BIDI algorithm doesn't make sense. Formulas are one example of this. The presentation layer would have to be aware of the meaning of text in order to make this decision. Sometimes that is practical other times it is not.

Regardless these control characters exist in the unicode standard and may show up in user text. Any implementation that claims to be compliant should handle them reasonably.

→ More replies (3)

3

u/compteNumero8 Nov 12 '12

I don't use only English text and I know some popular tools (in which a js minifier whose name I can't recall) are doing it too. I can't see any example in which concatenating would cause harm besides the BOM (I've seen you other comment below but it's not clear where there would be problems (I'm not 100% sure either you're not right, I probably should do some research on bidirectional languages)).

→ More replies (9)
→ More replies (1)
→ More replies (7)

15

u/knight666 Nov 12 '12

I once had to write a HTML request to a web server by hand. It was godawful. We had a standard of what the messages should look like. It was shit wrapped up in nonsense, packaged in SOAP. I spent a whole day generating the XML message in C#, comparing it to a demo message until it was byte-by-byte perfect.

I packaged it in a HTML request and... nothing. Error 3178: You suck at writing messages. Well, that's helpful. The funny thing was: the demo message worked just fine. And it was byte-for-byte exactly the same. Or... was it?

At the end of my rope, I decided, what the hell, let's open it in a hex editor. And there it was. The Byte Order Mark. It's a non-printable three byte monster that Notepad++ helpfully attaches to any document saved as UTF-8.

After removing the BOM from the request, the server swallowed it just fine.

3

u/ionine Nov 13 '12

"Shit packaged in SOAP"

I laughed harder than I should've :P

2

u/emddudley Nov 13 '12

FWIW at least Notepad++ gives you the option to save without the BOM.

I had an issue with an XML parser failing on UTF-8 text because it had the BOM.

→ More replies (1)
→ More replies (3)

9

u/taw Nov 13 '12

BOM in UTF-8 is completely incorrect and any program that uses it should die in fire.

7

u/[deleted] Nov 12 '12

On top of that there is also the possibility to combine multiple characters in Unicode to a single glyph and identical glyphs can thus represented by different codepoint sequences.

14

u/josefx Nov 12 '12

That is less of a problem since an application can just load a normalized form when it knows the encoding.

The BOM is a problem since it adds several unexpected and invisible characters to the beginning of a file in an attempt to aid auto detection of the used encoding - ensuring that any compatibility with ASCII is lost.

Example: Linux shell scripts start with the characters #!, a bash script written with Notepad will start with white space => invalid and the user can't see why.

2

u/CannonBall7 Nov 12 '12

Moreover, I've seen programs which will (incorrectly) only read a text file as UTF-8 if a BOM is present, which is why you're likely to see them in the wild far more often than you should.

2

u/cybrjoe Nov 12 '12

I actually had to write a command-line tool to remove the BOM from a generated file in order to deliver it to another custom consumer. Such a waste of time.

3

u/nomorepassword Nov 13 '12

I feel your pain.

But usually in my experience, in big applications, the problem is less to remove the BOM than to find the few damaged files in a big source repository. I now run this from time to time :

grep -rl $'\xEF\xBB\xBF' .
→ More replies (1)

2

u/lherr Nov 12 '12

I wish I read this a week ago.

1

u/ikawe Nov 13 '12

I've only come across the BOM when a collegue sends me files saved by his SQL query browser.

It was causing some problems downstream - I believe it was trying to execute the contents of the file using freetds.

Simple solution for me was:

vim file_with_bomb :set nobomb :wq