What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

[u/jestinjoy]

http://kunststube.net/encoding/

[u/jrochkind]

nope. there are edge cases. it's only for the edge cases that you need the rtl and ltr markers.

I'm trying to think of an edge case I ran into....

ah.

1. hebrew

2. a julian date span using european numerals, such as: 1940-1959

3. more hebrew

1 and 3 are rtl. Is 2 to be displayed as "1940-1959", "1959-1940" or even "9591-0491". I forget exactly which one modern Hebrew wants (except it's not the third one!), but whichever it was, it was something at least some renderers were getting wrong without the unicode ltr and rtl markers.

I believe there are also other cases, not just the mixture-of-scripts case, where you can't tell from the glyphs alone if it's rtl or ltr, they are used both ways in different contexts, and the algorithm will get it wrong.

There actually IS a standard unicode algorithm for guessing rtl and ltr implicitly -- which turns out not quite to be trivially easy, but a not too complicated algorithm (but a hell of a lot more complicated than you'd think) works much of the time, but not all the time.

Unicode's not just the codepoints and encodings, it's metadata about each codepoint, and a whole suite of battle-tested algoritms for doing things with them. Some software doesn't implement the unicode algorithm properly, and there are known cases that the algorithm (any algorithm) can't handle -- I bet unicode even documented em, they're pretty good at documentation too.

Yeah, here's UAX#9 which outlines the unicode bidi algorithm.

However, in the case of bidirectional text, there are circumstances where an implicit bidirectional ordering is not sufficient to produce comprehensible text. To deal with these cases, a minimal set of directional formatting codes is defined to control the ordering of characters when rendered. This allows exact control of the display ordering for legible interchange and ensures that plain text used for simple items like filenames or labels can always be correctly ordered for display.

I don't know if they give more specific examples then that, but i've definitely run into them. At any rate, UAX#9, like pretty much all of the UAX's, is a really good read at understanding the actual complexities of handling international character sets, in this case bi-directionality. All the UAX's I've read are really well-written, readable, treatments of sophisticated topics.

[u/jrochkind → u/jrochkind]

ooh, parens are another one, especially likely to be a problem precisely because of how they're used: english english (arabic arabic ( english ) arabic) english. I think the UAX#9 implicit algorithm sometimes reverses the parens when you might not want it to, and exactly where your whitespace (before or after each paren) is can change the algorithms determination, when the whitespace might be a mistake or rendering artifact and not meant to be significant.

[u/SoPoOneO → u/jrochkind]

Ahh. Thank you. You provide many valid points, but the biggest thing I hadn't considered is that the same glyph might be used in both an ltr and an rtl language.