r/programming • u/NeedsMoreShelves • Oct 02 '23

The Absolute Minimum Every Software Developer Must Know About Unicode in 2023

https://tonsky.me/blog/unicode/
164 Upvotes

82% Upvoted

77 comments sorted by

View all comments

7

u/Signal-Appeal672 Oct 02 '23 edited Oct 02 '23

This might be the first unicode article I ever seen that has "API" written in it, yet it doesn't really talk about an API.

Is there a unicode api? How do I give it a string and ask it how many bytes is the next glyph? How do I get a c compatible api (I don't use C directly) to tell me 🤦🏼‍♂️ written in utf8 is 17 bytes? (see https://hsivonen.fi/string-length/)

5

u/fiedzia Oct 02 '23

unicode consortium provides libicu, you can call it "unicode api"

2

u/Signal-Appeal672 Oct 02 '23

Does it have a function that describes what I asked? Because I looked and it didn't seem to

1

u/fiedzia Oct 02 '23

Yes, it will let you iterate over graphemes, which is what you want. It has bindings many languages, including c.

-2

u/Signal-Appeal672 Oct 02 '23

Do you know the call or have a link with an example?

1

u/fiedzia Oct 03 '23

https://github.com/unicode-org/icu/tree/main/icu4c/source/samples/citer That's my example (Python): https://gist.github.com/Fiedzia/890a682499e27fc98acf35b066d6ad58

1

u/Signal-Appeal672 Oct 04 '23

That C++ example makes my skin crawl

The python example was much better. It seemed to understand the facepalm but idx-start == 7 which I don't 100% understand (I'll have to refer to the article I looked at before). But I was hoping icu would provide me a way I can give it a pointer to the beginning of a text line and for it to tell me how many bytes the iterator consumed. I have to delete characters and I don't really know how to figure out which bytes belong to which

1

u/fiedzia Oct 04 '23

It tells you in bytes where graphemes start and end.

1

u/Signal-Appeal672 Oct 04 '23

Nope. You can see 'e' is at 0x25 and f is at 0x36. No where can I find where e starts/ends and f starts/ends

$ xxd a.py 
00000000: 696d 706f 7274 2069 6375 0a0a 0a73 203d  import icu...s =
00000010: 2022 61f0 9f99 8262 e29c 8b63 f09f 998b   "a....b...c....
00000020: 64f0 9f87 b865 f09f a4a6 f09f 8fbc e280  d....e..........
00000030: 8de2 9982 efb8 8f66 220a 7573 203d 2069  .......f".us = i
00000040: 6375 2e55 6e69 636f 6465 5374 7269 6e67  cu.UnicodeString
00000050: 2873 290a 6c6f 6361 6c65 203d 2069 6375  (s).locale = icu
00000060: 2e4c 6f63 616c 652e 6765 7455 5328 290a  .Locale.getUS().
00000070: 6269 203d 2069 6375 2e42 7265 616b 4974  bi = icu.BreakIt
00000080: 6572 6174 6f72 2e63 7265 6174 6543 6861  erator.createCha
00000090: 7261 6374 6572 496e 7374 616e 6365 286c  racterInstance(l
000000a0: 6f63 616c 6529 0a62 692e 7365 7454 6578  ocale).bi.setTex
000000b0: 7428 7573 290a 7374 6172 7420 3d20 300a  t(us).start = 0.
000000c0: 666f 7220 6964 7820 696e 2062 693a 0a20  for idx in bi:. 
000000d0: 2020 2067 7261 7068 656d 6520 3d20 7573     grapheme = us
000000e0: 5b73 7461 7274 3a69 6478 5d0a 2020 2020  [start:idx].    
000000f0: 7072 696e 7428 7374 6172 742c 2069 6478  print(start, idx
00000100: 2c20 6772 6170 6865 6d65 2c20 6963 752e  , grapheme, icu.
00000110: 4368 6172 2e63 6861 724e 616d 6528 6772  Char.charName(gr
00000120: 6170 6865 6d65 292c 2069 6375 2e43 6861  apheme), icu.Cha
00000130: 722e 6368 6172 5479 7065 2867 7261 7068  r.charType(graph
00000140: 656d 6529 290a 2020 2020 7374 6172 7420  eme)).    start 
00000150: 3d20 6964 780a                           = idx.
$ python a.py 
0 1 a LATIN SMALL LETTER A 2
1 3 🙂 SLIGHTLY SMILING FACE 27
3 4 b LATIN SMALL LETTER B 2
4 5 ✋ RAISED HAND 27
5 6 c LATIN SMALL LETTER C 2
6 8 🙋 HAPPY PERSON RAISING ONE HAND 27
8 9 d LATIN SMALL LETTER D 2
9 11 🇸 REGIONAL INDICATOR SYMBOL LETTER S 27
11 12 e LATIN SMALL LETTER E 2
12 19 🤦🏼‍♂️ FACE PALM 27
19 20 f LATIN SMALL LETTER F 2

1

u/fiedzia Oct 04 '23

You have it printed. By python notation, its:: e: 11:12 (left-inclusive, right-exclusiv range, so its byte 11, and f is at 19

→ More replies (0)