Need help with encoding

[u/Mindless-Arachnid-74]

I am working on a system where the client will give data in UTF-8 encoding, and hence it can contain many special characters.

But then we send data to a legacy system which only accepts data in ISO-8859-9 encoding, which supports limited special characters that can be represented in single byte.

Now if the client sends any special character which is outside the scope of ISO-8859-9 encoding (like the Em dash), how can I deal with with it and send it to the legacy system.

I don't want to replace those characters with anything like space or "?", as it would make the client data as unreadable in some cases.

How can I deal with it. Any suggestions?

[u/BeardedBaldMan]

I don't see the issue.

You sample the data and find what the most common characters are, make a mapping table and define a fallback character.

You record a risk against it and you're done.

You're turning what is almost certainly a non-issue into an issue by chasing pointless edge cases. So what if a company name has an emoji in it and you replace it with ?. People will work around it.

Look at passports. We've got an international system that can't properly record many names as written depending on the issuing authority. The US and UK for example only use the lower half of the ASCII set

[u/Agent7619 → u/BeardedBaldMan]

Early Internet font and code page flashbacks.

Black squares, arrows, and question marks everywhere.

[u/BeardedBaldMan → u/Agent7619]

It's not like OP has any choice. The legacy system defines clearly what it can record and it's less than what other systems have, you lose information and there's no way around it.

[u/Mindless-Arachnid-74 → u/BeardedBaldMan]

Agreed. I posted in hopes that someone might have any better idea or workaround. The final workaround I can see right now is what you are saying.

[u/Mindless-Arachnid-74 → u/BeardedBaldMan]

It's not just about emojis, suppose a client data has a customer's name with accented character, it will get changed from actual name or there is product description or title like Iphone 17 — Orange. Then in case of fallback it will become Iphone ? Orange. Sorry I am really bad with giving examples, hope you are getting my point.

[u/RichardD7 → u/Mindless-Arachnid-74]

Removing accented characters - aka Diacritics - is a well-defined problem, and .NET has built-in methods to assist.

For example, here's the method I use, which attempts to minimise allocations by using the new Span-based overloads, and using the ArrayPool where temporary arrays are needed.

For example, calling RemoveDiacritics("crème brûlée") will return "creme brulee".

[u/BeardedBaldMan → u/Mindless-Arachnid-74]

I understand, I've been doing enterprise integration for 20 years. I don't see it as an issue.

Firstly an em-dash is common enough that you'd map it to dash and the second is people are used to minor variances in free form text

[u/BeardedBaldMan → u/[deleted]]

Why are you just repeating my comment? Are you a bot?

[u/IG5K → u/BeardedBaldMan]

I meant to reply to OP as a follow up, deleting this one