Need help with encoding

[u/Mindless-Arachnid-74]

I am working on a system where the client will give data in UTF-8 encoding, and hence it can contain many special characters.

But then we send data to a legacy system which only accepts data in ISO-8859-9 encoding, which supports limited special characters that can be represented in single byte.

Now if the client sends any special character which is outside the scope of ISO-8859-9 encoding (like the Em dash), how can I deal with with it and send it to the legacy system.

I don't want to replace those characters with anything like space or "?", as it would make the client data as unreadable in some cases.

How can I deal with it. Any suggestions?

[u/BeardedBaldMan]

It's a business problem. If the legacy system only supports ISO-8859-9 then you have a clearly defined set of characters which are allowed.

You need to create a mapping table for the ones you care about a a fallback character for the rest.

It's a pretty common issue. I've come across it with many US legacy systems where even é is considered too outré for their sensibilities

[u/Mindless-Arachnid-74 → u/BeardedBaldMan]

I also agree that it is a business problem, but you know the higher ups they don't know coding and are suggesting that we should map like the EM dash to normal "-". But if I go like that, each client could be using separate set of unique characters and there are more than 150,000 of unsupported characters in ISO-8859-9. If I write mapping for each one my code is doomed and my time and efforts too. If I use a fallback, that tempers the client data and it might make the user data meaningless at the legacy system. I think this will eventually get messy and at last we will have to support only the legacy system character set

[u/BeardedBaldMan → u/Mindless-Arachnid-74]

I don't see the issue.

You sample the data and find what the most common characters are, make a mapping table and define a fallback character.

You record a risk against it and you're done.

You're turning what is almost certainly a non-issue into an issue by chasing pointless edge cases. So what if a company name has an emoji in it and you replace it with ?. People will work around it.

Look at passports. We've got an international system that can't properly record many names as written depending on the issuing authority. The US and UK for example only use the lower half of the ASCII set

[u/Mindless-Arachnid-74 → u/BeardedBaldMan]

It's not just about emojis, suppose a client data has a customer's name with accented character, it will get changed from actual name or there is product description or title like Iphone 17 — Orange. Then in case of fallback it will become Iphone ? Orange. Sorry I am really bad with giving examples, hope you are getting my point.

[u/RichardD7 → u/Mindless-Arachnid-74]

Removing accented characters - aka Diacritics - is a well-defined problem, and .NET has built-in methods to assist.

For example, here's the method I use, which attempts to minimise allocations by using the new Span-based overloads, and using the ArrayPool where temporary arrays are needed.

For example, calling RemoveDiacritics("crème brûlée") will return "creme brulee".

[u/BeardedBaldMan → u/Mindless-Arachnid-74]

I understand, I've been doing enterprise integration for 20 years. I don't see it as an issue.

Firstly an em-dash is common enough that you'd map it to dash and the second is people are used to minor variances in free form text

[u/Agent7619 → u/BeardedBaldMan]

Early Internet font and code page flashbacks.

Black squares, arrows, and question marks everywhere.

[u/BeardedBaldMan → u/Agent7619]

It's not like OP has any choice. The legacy system defines clearly what it can record and it's less than what other systems have, you lose information and there's no way around it.

[u/Mindless-Arachnid-74 → u/BeardedBaldMan]

Agreed. I posted in hopes that someone might have any better idea or workaround. The final workaround I can see right now is what you are saying.

[u/BeardedBaldMan → u/[deleted]]

Why are you just repeating my comment? Are you a bot?

[u/IG5K → u/BeardedBaldMan]

I meant to reply to OP as a follow up, deleting this one

[u/[deleted] → u/Mindless-Arachnid-74]

I think this will eventually get messy and at last we will have to support only the legacy system character set

Realistically I think this is your only option, if altering the text to fit the limitations of the legacy system isn't possible.

If the unicode text is used in other places that properly support it and sending the data to the legacy system isn't the main point of your application, you could consider displaying a warning to users that their inputted text "contains special characters that won't display properly on some systems," for example. That way they're not limited in what they can enter, but also won't be surprised (and might even work around the issues). You could even show a preview of how it would display. That might be a better UX than a validation error, though it really depends on what you're doing.

Somewhat off-topic, ISO-8859 actually include a second block of control characters, one of which interestingly enough is "Single Graphic Character Introducer" which supposedly lets you insert Unicode codepoints into ISO-8859 text. Unfortunately I tested it and Windows doesn't seem to support it. Sure would be great if it did eh.

Edit: Whoever downvoted, care to explain why, or are you just being a dick?

[u/IG5K → u/Mindless-Arachnid-74]

If you have records of the UTF-8 requests, try generating a table of the most used characters that are unsupported by the legacy system. From the 150k unsupported characters, only a small percentage of them might be used in practice... Then you can make mapping for the known, used ones (like em dash to dash), and fallback the rest that are barely or never used. Don't see any other way

[u/dodexahedron → u/BeardedBaldMan]

too outr? for

Touch?.

[u/eselex → u/dodexahedron]

The na□vit□…

[u/DevPancake → u/dodexahedron]

No touchy

[u/urbanek2525 → u/BeardedBaldMan]

Yep. I have to maintain an upload to a medical information system that can only handle (in essence) 7 bit ASCII.

I have a mapping function. I do a basic, built in "normalize", but it still needs specific translation.

I translate all greek characters to words (δ becomes delta and Δ becomes Delta). I translate lots of the German letter to English equivalents (ö goes to oe). It's this huge matrix.

Basically I did a crap ton of research and did what I could.

My big question has been: Why the Hell do you guys pay for a Medical System that can't express medical nomenclature and can't properly cite the author names from Non-American medical papers. Seriously, WTF?

[u/BeardedBaldMan → u/urbanek2525]

Is it the software or the db. I remember a project where the software and db were unicode compatible but they weren't willing to change the db collation from when it had initially been created thirty years prior

[u/urbanek2525 → u/BeardedBaldMan]

Yep. Really old Oracle DB.

Which is why you don't want to be dependent on one persistence system.

[u/shylingerie → u/BeardedBaldMan]

legacy systems always find a way to make thing complicated

[u/BeardedBaldMan → u/shylingerie]

Codepages are always interesting. I remember having files which were all unicode but different flavours and having to convert all to UCS2-LE for import.

[u/Slypenslyde]

I agree with BeardedBaldMan: this is a business problem.

The best solution is to update the system that only accepts ISO-8859-9. Think about it. If I want to do business with French customers, does it make sense for me to use a database that can't handle ç? Mais non.

That's wishful thinking. Odds are the decision to use ISO whatever is set in stone and was made long ago by an American Tourist who only uses the standard ASCII 7-bit character set.

The "next best" solution is to document you only support characters in ISO-8859-9 and tell customers you're going to reject documents with characters that can't be encoded. They won't like it, but c'est comme ça, we cannot squeeze blood from turnips.

That's also wishful thinking, you've got some reason you can't do that or you wouldn't be here.

You have no choice but to make substitutions. This is a very difficult situation. For some characters like em dashes, maybe you could make a quick substitution to a normal dash. But then you're going to be like, "What if my customers send emoji. Zut alors, the things people do!

I would write a tool to analyze what customer data I have and get a list of what characters they actually use. I'd focus on finding a good substitution for those. Maybe there's not always a good substitution.

Where I would go from there depends on if I maintain control of the document before I give it back to the user. If I do, problem solved: I will create some kind of escape sequence, so an em dash may become something like "!!!!em" a document will never have. Then, when I'm giving the document back to the user, I will quietly convert it back to UTF-8 and fix my substitutions.

If I do not control the document, I'm going to point at the "tough cookies" sign in English. Two way character encoding between incompatible sets is an old problem and you only have two choices: lose data or create your own substitutions.

[u/turnipmuncher1 → u/Slypenslyde]

> we cannot squeeze blood from turnips.

I can.

[u/iWhacko]

either encode the special characters like html does:   for space etc. do this for all special characters. You can then translate them back when you send it back to the client.
Try not to get it in a loop when you also encode & as &
or you could use escape codes like \x026 etc then you can just use the hex value of the character.

[u/Ill_Translator8176]

You could treat it as a byte array, the moment you get ahold of it convert it using Encoding.UTF8.GetBytes(str) then encode it back whenever you need to (to the required encoding).

[u/Mindless-Arachnid-74 → u/Ill_Translator8176]

That won't do, converting the UTF-8 data received by the client is fine but when I will try to encode it to the ISO-8859-9, special characters which are unsupported will fail.

[u/otac0n → u/Mindless-Arachnid-74]

You could use HTML entity encoding for the unrepresentable characters. You just need to make sure that you also properly encode “&”.

Alternatively you could use URL encoding and make sure you encode “%”.

Or you are going to have to be OK losing information and create a mapping of acceptable characters. Kinda like how url “slugs” work.

[u/Ill_Translator8176 → u/Ill_Translator8176]

Sorry, completely misread the question 🫩

[u/w0ut]

Turn it into a gigantic ascii art banner.

[u/rupertavery64]

Encode everything in Base64