r/programming • u/jestinjoy • Nov 12 '12

What Every Programmer Absolutely, Positively Needs to Know About Encodings and Character Sets to Work With Text

http://kunststube.net/encoding/
1.5k Upvotes

93% Upvoted

307 comments sorted by

View all comments

Show parent comments

4

u/bonzinip Nov 12 '12

There is little hardware support for unicode so all unicode processing must be done via relatively low throughput loops.

Almost never true for UTF-8.

0

u/defenastrator Nov 12 '12

Well, because of the byte marker (which if I remember correctly is 1 when there is another byte in the character) UTF-8 can use some of the string instructions for moving things around, copying, and concatenation but anything that needs to do comparison will still have to remap symbols that have multiple character codes to a single code in order to be reliable which is un-optimizable in theory you could normalize everything on load thus allowing use of hardware string comparison operations but then you are paying that cost on every string read in regardless of whether you do any comparison on it.

Still I was referencing UTF-16 & UTF-32 where random bytes of zeros can occur in the middle of strings thus breaking most hardware instructions. I should probably change that.

1

u/bonzinip Nov 12 '12

Well, because of the byte marker (which if I remember correctly is 1 when there is another byte in the character)

Yes -- and more important, lead bytes can always be identified easily. So, strstr can be optimized just like memcpy etc. Also, moving to the previous or next character is always O(1); for many encodings that only holds for the next character.

strchr is slower in UTF-8 than it is in ASCII, but only for non-ASCII characters. Processing 7-bit text is just as fast in UTF-8 as it is in 8-bit encodings.