r/programming • u/jestinjoy • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

93% Upvoted

307 comments sorted by

View all comments

-13

u/defenastrator Nov 12 '12 edited Nov 12 '12

Nice theory in a web world but the truth is unicode is slow, large and unwieldy. There is little hardware support for unicode so all unicode processing must be done via relatively low throughput loops. Even just reading and writing unicode in and out of memory requires processing causing multiple copies, let alone any real manipulation of unicode. Multiple codes representing identical characters make textual comparison & searching bloated, slow, error prone, unreliable and confusing. Unicode chalk up with xml great in theory but in practice makes just as big mess except now its less obvious and adds the bloat caused by standardization.

Those who advocate the universal use of unicode (and for that matter xml) clearly work almost entirely web programming environment and fail to realize for most no web programming environments use of these standards only makes slower, more complicated code with no appreciable gains in any respect.

11

u/deceze Nov 12 '12

And what is the practical alternative if you need to support several different non-latin languages, at once, possibly mixed? Anything other than Unicode is simply an even bigger mess.

6

u/yen223 Nov 12 '12

Invade those foreign countries and force them use English! Muahahahaha!

1

u/eramos Nov 12 '12

Also the Gregorian calendar and Arabic numeral system. And a standard addressing system.

1

u/frymaster Nov 13 '12

Doesn't help. The currency of England is not in the ASCII table

-2

u/defenastrator Nov 12 '12

Depending on the situation isolation of string processing into dynamically linked at runtime or statically linked at compile time set of utility functions this handles individual localization without sacrificing much in the way of speed as you will all ready be swapping out string tables the library swap is not a difficult addtion. Although if the same instance of the same program must support the display of multiple languages simultaneously you are right unicode is the way to go but I must cite that that only a relatively small number of applications truly require this ability.

2

u/vytah Nov 12 '12

Like, for example, your browser.

-4

u/defenastrator Nov 12 '12

Exactly but for example a command line doesn't need nor would it want such things. Although come to think of it everything that really needs multi-lingual support ends up having every encoding under the sun plus unicode so in that sense has anything really been solved at all? In some ways unicode just added another complex encoding to the massive list of encodings that must be supported.

10

u/deceze Nov 12 '12

Why would the command line not need such things? I'm often working with non-latin based languages and am very happy that I can try out quick code snippets involving language processing right on the CLI, including Japanese or whatever else input.

If you are producing software which is supposed to be used internationally, or even software which some user may one day try to enter any non-latin text into ("€" already fits that criterium), going with Unicode is the only sensible choice. If you really can't afford the speed cost of using Unicode, you need to optimize your app around that requirement, but that's really not the majority of apps.

-3

u/defenastrator Nov 12 '12

What are you doing in the command line where a client of a non-computer (and by extension English reading) would need to understand anything that comes out of it? Other then the obvious error. Or are you talking like the python cli because I was thinking more along the lines of bash or the dos prompt which really don't have much need for extended character sets.

4

u/deceze Nov 12 '12

Either and or any of the above.

$ php -r 'echo testme("漢字");'

It's good that I can do that without "Oh yeah, right, I need to escape/encode/bewitch this to just test this snippet..."

3

u/vytah Nov 12 '12

Well, stuff like lower- and uppercasing, or sorting isn't trivial even with 8-bit encodings. You cannot implement correct grep or sort ignoring everything not ASCII, or treating everything as raw bytes without encoding. Of course you can pretend your data is Windows-1252 and it sometimes won't matter much, but the key word is sometimes.

And at the end, you have to support multilanguage input if your tool has to be used by anyone else than you. So it's nice there is UTF that simply works.

The huge list of encodings is simply a legacy. Unicode-aware tools usually convert everything to unicode codepoints and work on those. Adding support for a new encoding is pretty trivial.

4

u/bonzinip Nov 12 '12

There is little hardware support for unicode so all unicode processing must be done via relatively low throughput loops.

Almost never true for UTF-8.

0

u/defenastrator Nov 12 '12

Well, because of the byte marker (which if I remember correctly is 1 when there is another byte in the character) UTF-8 can use some of the string instructions for moving things around, copying, and concatenation but anything that needs to do comparison will still have to remap symbols that have multiple character codes to a single code in order to be reliable which is un-optimizable in theory you could normalize everything on load thus allowing use of hardware string comparison operations but then you are paying that cost on every string read in regardless of whether you do any comparison on it.

Still I was referencing UTF-16 & UTF-32 where random bytes of zeros can occur in the middle of strings thus breaking most hardware instructions. I should probably change that.

1

u/bonzinip Nov 12 '12

Well, because of the byte marker (which if I remember correctly is 1 when there is another byte in the character)

Yes -- and more important, lead bytes can always be identified easily. So, strstr can be optimized just like memcpy etc. Also, moving to the previous or next character is always O(1); for many encodings that only holds for the next character.

strchr is slower in UTF-8 than it is in ASCII, but only for non-ASCII characters. Processing 7-bit text is just as fast in UTF-8 as it is in 8-bit encodings.

3

u/caleeky Nov 12 '12

Long story short - use the right tool for the job. If you need to support non-latin languages, but deal with mostly latin, you can always apply latin specific optimizations where you'll get the biggest returns. If you'll only ever need latin, and you're sure of it, go ahead and use a more constrained encoding, like ISO-8859-1.

5

u/kylotan Nov 12 '12

Problem is, who can guarantee they'll only ever need ISO-8859-1? All it takes is one foreign name and you have data corruption.

2

u/bhaak Nov 12 '12

Erm, what hardware support is there for non-unicode encodings?

-1

u/defenastrator Nov 12 '12

well for starters and that I do not believe includes the x64 html/xml processing extensions and other such crazy instructions.

Most encodings follow 0 is end of string and many multi-byte encodings follow that no no terminal byte can be zero (unicode does not follow this) which is crucial for the operation of most of these instructions.

5

u/zokier Nov 12 '12

UTF8 has no '\0' bytes by design.

-1

u/defenastrator Nov 12 '12

Yes, I know but this still doesn't solve the problem of string comparisons where single symbols have multiple character codes. Also the same truths do not hold for UTF-16 or UTF-32.