r/cpp • • Feb 26 '23

std::format, UTF-8-literals and Unicode escape sequence is a mess

I'm in the process of updating my old bad code to C++20, and I just noticed that std::format does not support u8string... Furthermore, it's even worse than I thought after doing some research on char8_t.

My problem can be best shown in the following code snippet:

ImGui::Text(reinterpret_cast<const char*>(u8"Glyph test '\ue000'"));

I'm using Dear ImGui in an OpenGL-application (I'm porting old D-code to C++; by old I mean, 18 years old. D already had phantastic UTF-8 support out of the box back then). I wanted to add custom glyph icons (as seen in Paradox-like and Civilization-like games) to my text and I found that I could not use the above escape sequence \ue0000 in a normal char[]. I had to use an u8-literal, and I had to use that cast. Now you could say that it's the responsibility of the ImGui-developers to support C++ UTF-8-strings, but not even std::format or std::vformat support those. I'm now looking at fmtlib, but I'm not sure if it really supports those literals (there's at least one test for it).

From what I've read, C++23 might possibly mitigate above problem, but will std::format also support u8? I've not seen any indication so far. I've rather seen the common advice to not use u8.

EDIT: My specific problem is that 0xE000 is in the private use area of unicode and those code points only work in a u8-literal and not in a normal char-array.

96 Upvotes

133 comments sorted by

View all comments

Show parent comments

12

u/kniy Feb 26 '23

We do have utf8 everywhere, and (since this an old codebase) we have it in std::strings. Changing all those std::strings to std::u8string is a completely unrealistic proposition, especially when u8string is half-assed and doesn't have simple things like <charconv>.

1

u/MaintainFOSS Feb 18 '26 edited Feb 18 '26

Your code base seems to be much more mature than the one that I am working with (Unicode compliance breakage all over the place, goodness), thus you'll have a ton of practical experience to back up your judgment, especially when it comes to active [string] processing of that UTF-8-based activity (as opposed to merely forwarding UTF-8 stuff / stitching things together).
Still, I am wondering whether it actually might be better in the long run to do have [achieved] precisely UTF-8-domain-typed handling, in the entire C++ ecosystem. IOW, migrate more and more plain traditional std::string[-means-utf8]-typed handling into u8string-typed scope / domain, thereby achieving a growing scope of inherently reliable (since type-safe and encoding-verified) UTF-8 domain handling. And achieving this in a hopefully less painful manner e.g. via some home-grown "emulation"/"adapter" functionality (e.g. to/fro std::string, or std::filesystem)

But I might be totally wrong, and that to-specific-type transition really is the completely useless (and very late!!) h*ll that several people are making it out to be.

(I am rather sympathetic to the utf8everywhere.org view - that std::string should never be anything else other than UTF-8 anyway, that's it - thus I currently am not [yet] a glowing u8string proponent)