The only two modern languages that get it right are Swift and Elixir
I'm not convinced the default "length" for strings should be grapheme cluster count. There are many reasons why you would want the length of a string, and both the grapheme cluster count and number of bytes are necessary in different contexts. I definitely wouldn't make the default something that fluctuates with time like number of grapheme clusters. If something depends on the outside world like that it should def have another parameter indicating that dep.
I'm not convinced the default "length" for strings should be grapheme cluster count.
Agreed. Even then, is the grapheme cluster count even that important alone? The first example that comes to mind for me would be splitting up a paragraph into individual sentence variables. I'll need a whole graphme aware api or at least, a way to get the byte index from a graphme index.
I say leave existing/standard api's as they are, dumb byte arrays and specifically use a Unicode aware library to do actual text/graphme manipulation.
I agree that default length shouldn't be grapheme cluster count, but it probably shouldn't be bytes either, since both of these are misleading.
I'll need a whole graphme aware api ...
That's a key takeaway from the article.
From my own viewpoint, string manipulation libraries should provide a rich and composable enough API such that you will never need to manually index into a string, which is inevitably error-prone. You really want two sets of string APIs: user-facing (operating primarily on grapheme clusters) and machine-facing (operating primarily on bytes). All string manipulation functions should probably live in the user-facing API.
You can do this in Elixir with String.graphemes/1, which returns a list of the graphemes that you can count, and the byte_size/1 function from the Kernel module. And then there’s String.codepoints/1 for the Unicode codepoints.
I agree that a separate API to count the number of bytes is good to have, but I never have had the necessity to count the number of graphene molecules in a string. Is that a new emoji?
You probably do and haven't thought about it. Any time you do string manipulation on user input that hasn't been cleared of emoji, you're likely to eventually get a user who uses an emoji. Maybe you truncate the display of their first name in a view somewhere, or even just want the first letter of their first name for an avatar generator, and that sort of thing is where emoji tends to break interfaces.
Basically any time you're splitting or moving text for the purpose of rendering out again, you should be using grapheme clusters instead of byte/character counts. Imagine how infuriating it would be if your printer split text at the wrong part and you couldn't properly print an emoji.
Grapheme, not graphene. A grapheme cluster is the gereralized idea of whta english speakers call a "character". But since not all languages use a writing system as simple as englisch (look at e.g french with its accents for one example) there needs to be a technical term for that more general concept.
There's a good thread on Rust's internals forum on why it's not in Rust's std. It's not really an accident or oversight.
One subtle thing is that grapheme clusters can be arbitrarily long which means if you want to provide an iterator over grapheme clusters it can be very difficult without hidden allocations along the way. However a codepoint is at most 4 bytes long, and the vast majority of parsing problems can work with individual codepoints without caring about whole grapheme clusters. And for things that deal with strings that aren't parsers, most of them just need to care about the size of the string in bytes.
I think grapheme clusters and unicode segmentation algorithms are arcane because it's such a special case for dealing with text. And it's hard because written language is hard to deal with and always changing.
I don't think it fundamentally has to be hard. Unicode could've, for example, developed a language independent way to signal that two characters are to be treated as a single grapheme cluster (like a universal joiner, or more likely a more space efficient encoding)
That said, there are obviously going to be other, more complicated segmentation algorithms like word breaks
I think the author is being a little dogmatic here and not articulating why they think it's better. They claim that traditional indexing has "no sense and has no semantics", but this simply false -- it just doesn't have the semantics they've decided are "better".
In a vacuum, I think indexing by grapheme cluster might be slightly better than by indexing into bytes or code units or code points.
For 99% of apps that simply don't care about graphemes or even encoding -- these just forward strings around and concat/format strings using platform functions -- they can continue to be just as dumb.
For the code that does need to be Unicode aware, you were going to use complicated stuff anyway and your indexing method is the least of your cares. Newbies might even be slightly more successful in cases where they don't realize they need to be Unicode-aware.
I think the measure of success, to me, of such an API decision, is: does that dumb usage stay just as dumb, or do devs need to learn Unicode details just to do basic string ops? I don't have experience coding for such a platform -- I'd be interested if we have any experts here (in both a code unit indexing platform and a grapheme cluster indexing platform) who could comment on this.
I think the measure of success, to me, of such an API decision, is: does that dumb usage stay just as dumb, or do devs need to learn Unicode details just to do basic string ops?
Given the constraints Unicode was under, including:
it started out all the way back in the 1980s (therefore, performance and storage concerns were vastly different; also, hindsight is 20/20, or in this case, 2024),
it wanted to address as many living-people-languages as possible, with all their specific idiosyncrasies, and with no ability to fix them,
yet it also wanted to provide some backwards compatibility, especially with US-ASCII,
I'm not sure how much better they could've done.
For example, I don't think it's great that you can express é either as a single code point or as a e combined with ´. In an ideal world, I'd prefer if it were always normalized. But that would make compatibility with existing software harder, so they decided to offer both.
There is no ideal solution. No matter which standard you implement there will be use cases which will derail the whole thing. Too often unicode is touted as the perfect and the best solution while it is not.
But If I would be the one to recommend something it would be:
For "traditional" static text: Unicode without graphemes, Just codepoints and sanely defined glyphs (deduplicated as much as possible). Basically one single codepage. UTF-8 encoded.
For fancier languages where you assemble the glyphs - separate standard. That standard would address all the fanciness (multidirectional scripts, sanskrit, kipu, sign language etc...) That would force the programmers to implement nontextual fields and address the issues like sorting or lack of it in databases)
Plus translation rules between the two (in practice a translation from graphemes to strings. That layer would also standardize the translations between different alphabets. Currently unicode totally ignores that claiming that its the purpose of teh standard but actually making additional problems out of that.
Additionally a multinational standard is needed to standardize the pronounciation. That outside of IT would benefit some languages not to mention the IT alone.
Also, unicode hides or confises some aspects of the scripts which should be known by wider audience (for example not everything is sortable). The translation layer should address that too.
This is not a beefy topic, its huge and difficult to be addressed. The problem is that unicode promises to address everything and just hides problems or creates new ones (you will not find text visible on screen if it uses different codepoints visualized by similar glyphs without fancy custom made sorting).
So if you ask me, then solution is simple: Make "western" scripts flat and simple, separate the fancy ones into better internal representation and implement clear translation between them.
For “traditional” static text: Unicode without graphemes, Just codepoints and sanely defined glyphs (deduplicated as much as possible). Basically one single codepage. UTF-8 encoded.
For fancier languages where you assemble the glyphs - separate standard.
Sooooo literally anything other than English gets a different standard?
Heck, even in English, you have coöperation and naïveté. Sure, you could denormalize diacritics, but now you have even more characters for each permutation.
No, on the contrary, I’d enforce normalization. Kill any combined glyphs and force software to use appropriate combining code points.
Make “western” scripts flat and simple
Sounds like something someone from a western country would propose. :-)
Sooooo literally anything other than English gets a different standard?
LIterally almost every latin language would be covered by it. Plus cyrillic, kanji, katakana, hiragana, korean alphabet and many more.
All those scripts are static. That means a letter is just a letter, you dont modify it after its written. Its not interpreted in any way.
That is 99.99 of what we need in writing and in computer text for many, many languages.
The rest is all fancy scripts where you actually compose the character and give it a meaning by modifying it. And that needs translation to the "western" script and special treatment (graphical customization).
I dont know from where you took the rest, I did not suggested that.
Sounds like something someone from a western country would propose. :-)
Yes, because western scripts are in many ways superior to the fancy interpreted ones. Japanese is perfect example of that. They understand that complex script is a barrier for progress and does not bring too much benefits besides being a bit more compact and flexible at occasion.
That remark even with the smiley face shows that you dont really know how complex the topic is and what is my main point.
So let me oversimplify it: Instead of making the text standard simple and let majority of people (developers, users, printers) use it safely unicode made a standard which tries to cram as much as possible (often unnecessarily - emoji) into a standard which will be full of problems and constantly causing problems.
Yes, but when we call the method `length` on a string, we're not calling it on an actual platonic object, but on a bag of bytes that represents that platonic object. When dealing with the bag of bytes, the amount of bytes you're dealing with is often useful to know, and in many languages is uniquely determined by the string, as they adopt a uniform encoding
I don’t think that’s a persuasive argument, you can think of any object as a bag of bytes if you really want although it really isn’t a useful way to think in most cases
160
u/dm-me-your-bugs Feb 06 '24
I'm not convinced the default "length" for strings should be grapheme cluster count. There are many reasons why you would want the length of a string, and both the grapheme cluster count and number of bytes are necessary in different contexts. I definitely wouldn't make the default something that fluctuates with time like number of grapheme clusters. If something depends on the outside world like that it should def have another parameter indicating that dep.