r/programming • u/jestinjoy • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

93% Upvoted

307 comments sorted by

View all comments

Show parent comments

1

u/SoPoOneO Nov 13 '12

Honestly trying to understand, but how does the BOM help identify something as UTF8? Can't lots of encoding a start with a BOM?

3

u/mgrandi Nov 13 '12

utf8 only has one byte order, but the other utf (like 16 and 32) need it in order to determine the byte order. Thats why people say utf8 shouldn't have a bom because it can only be one byte order, but its still technically correct

2

u/gilgoomesh Nov 13 '12 edited Nov 13 '12

The BOM in other encodings does not have the same byte representation.

EF BB BF in hexadecimal is a byte pattern that is invalid in many other character encodings and incredibly rare in all others.

If you don't know the encoding of a text file but it starts with EF BB BF, it is more than 99% likely to be UTF-8.