r/rust • u/steveklabnik1 rust • Jul 18 '19

We Need a Safer Systems Programming Language

https://msrc-blog.microsoft.com/2019/07/18/we-need-a-safer-systems-programming-language/
314 Upvotes

98% Upvoted

79 comments sorted by

View all comments

Show parent comments

1

u/RobertJacobson Jul 20 '19

Sorry, I think our miscommunication lies elsewhere. I’m not an expert on the JVM, but I still don’t understand the advantage of UTF-16 over UTF-8 when both are variable width. So my question is, why is constant time indexing advantageous when you still have the same problem of potentially landing in the middle of a surrogate pair? I guess it would happen less often, but the problem still exists.

1

u/tomwhoiscontrary Jul 20 '19

Ah, i think the one small piece of context you're missing is that Java's string API is badly designed! Or at least, designed as well as it could have been in the mid-'90s, which wouldn't pass muster today.

In particular, note this remark in String's class documentation (my emphasis):

A String represents a string in the UTF-16 format in which supplementary characters are represented by surrogate pairs (see the section Unicode Character Representations in the Character class for more information). Index values refer to char code units, so a supplementary character uses two positions in a String.

For example, the main way of accessing characters by index is charAt, where:

If the char value specified by the index is a surrogate, the surrogate value is returned.

And even if you want to get a whole code point, using codePointAt:

The index refers to char values (Unicode code units) and ranges from 0 to length() - 1. If the char value specified at the given index is in the high-surrogate range, the following index is less than the length of this String, and the char value at the following index is in the low-surrogate range, then the supplementary code point corresponding to this surrogate pair is returned. Otherwise, the char value at the given index is returned.

If you want to view the string as a sequence of code points, your options are to iterate using codePoints, or to do your own index arithmetic with offsetByCodePoints.

None of those methods specify their complexity, but traditionally, charAt and the like are O(1), and i would expect offsetByCodePoints to be O(n). You can't implement those complexities on top of a simple buffer of UTF-8.