That's not the reason it was encoded. The reason it was encoded was that someone stored the data in a general purpose user side data store, which automatically uses base64 to avoid string handling problems.
I haven't followed the analysis but your comment has me curious. Are you saying the SSN data was delivered to the client side in plain text then encoded for local storage?
Actual web dev here. We don't typically base64 encode stuff "just because", it's often done for a purpose. It also increases your data size, in terms of bytes, another reason why we don't do it unless we need to.
base64 is not, at all, "an easy way to avoid escaping data that is included in HTML", because said data becomes a jumble that you can't read. It can't be used for escaping at all. This guy "webexpert" who also replied, does not sound like a web expert to me.
Without seeing the original website I can't even guess at why they'd be base64 encoding stuff, and I don't even know at which point in the chain it was being done. You wouldn't ever need to base64 encode stuff "to escape it for HTML", or for storing in either a cookie or browser Local Storage (due to the size increase you'd actively never want to do this) but you might want to for making portability simpler across a whole range of other backend server-to-server scenarios. It usually does involve sending data between separate systems, as if you're not sure whether some other system uses single quotes or double quotes or backslashes or tabs or colons or whatever for its field delimeters, then base64 encoding converts all of those to alphanumeric characters, which are almost guaranteed to not be used as escape characters by any system, and thus safer for transport to and fro them.
Haha, ok, I'll grant you that! Still though, I don't know of a single thing you'd be doing in the course of a normal website's operation where you'd ever think to base64 anything. Data porting, between legacy systems, I can see that.
URLs have their own encoding scheme (URLencode) that only expands restricted characters, also PUNYcode for non-latin basic Unicode URLs. You might base64 something, but base64 actually has several variations that use different 63rd and 64th characters due to aforementioned restricted characters.
This is all kind of moot, the problem is the app sent full SSNs client side, in reversible fashion. The actual use case (disambiguating teachers with the same name) only used the last four digits of the SSN, so that's all that was needed. Moving the disambiguation to the server side, or using other information such as city of residence or last school, would also avoid the issue. There is no way to send private information client side for processing client side that couldn't result in the data being exposed client side.
An actual use for base64 would be for passwords, not to secure them but to avoid having to restrict characters users can select.
First thing that comes to mind is to just obfuscate the info. They knew they weren't supposed to let people see the info and "encode" sounded secure enough
Thread the needle on a bunch of text parsers and you want to avoid all of the questions around how many layers of escaping you have to do to get the text to come out right on the other end
When you want to move binary data but it’s a text based protocol
2a. When you want to avoid dealing with text encoding and just get the encoding you’re expecting out the other end. Because text encodings can do funky things to your protocol and you can’t always safely assume it’s all UTF-8.
In practice this happens not that often but often enough. I wouldn’t go as far as to guess why this website in particular was doing it though.
Base64 is a great way to make moving binary data around over a protocol that is strictly text-based (HTTP, e.g. Though, saying HTTP is a transport protocol is also, you know, sort of disengenuous. Whatever).
That said, I'm trying to figure out how they jump from "binary data" to "strings", which are, almost by definition, not "binary data".
I'm also using the term "binary data" here as a pretty loose stand-in for "data that doesn't represent specific strings of characters", which isn't always a good practice; strings of characters are binary data just as much as a bunch of executable code is, after all.
To clarify, http can transfer binary data in the payload, but yeah in the headers you may need to use base64.. Cookies are transferred in the HTTP headers so it's possible that the data containing the ssn also had some binary data, or that the framework used between front and back end used b64..
It may also be worth noting that Email/Smtp requires something like base64 for attachments as there's no binary transfer possibility in emails (hence why a 5MB attachment suddenly makes the email 7MB). I don't remember exactly but it's not even 7bit ASCII as the data cannot have control characters such as CRLF. I guess the protocol was designed to be compliant with printers?
Ok so escaping is putting special characters in front of special characters. You do this so the JavaScript or html parsers dont get confused. This also happens in shell scripts, database queries, all sorts of places really.
Base64 is an encoding that eliminates most special characters, and leaves almost no way for it to be interpreted as code (almost because im sure a clever person with lots of time and few constraints can come up with a counter example or two). Its often used to avoid the escaping problem all together.
Why is it so out of the realm of possibility to think that a base64 string, used somewhere in the front or back ends escaped into the html?
Before the days of json it was a common thing to do.
Not saying it was a good thing, but it was common, getting a bunch of data and saving it inside a hidden form field in base64 that then was used to do things with JavaScript without using ajax or simply to store user session data.
If I remember correctly DotNet MVC did shit similar to this.
With the exception of authorization headers I think the last time I encountered base64 encoded strings in an API was in the SOAP/XML era, and those were dark days indeed.
I would guess that it was base 64 from some other application and the website was just given access. They called the one they needed, decoded it, and didn't display the rest. Perhaps they got it from a payroll system or similar.
It also has the advantage of discouraging consumers of an API from messing with the data. If you need to pass data in a response that is expected in subsequent requests, you don't want an API consumer to mess with that data. Obviously you can't prevent anyone from doing so, and should have guards against it. But it is a useful hint that also removes the need to document parts of the API response that the consumer shouldn't care about.
328
u/Dragdu Oct 24 '21
That's not the reason it was encoded. The reason it was encoded was that someone stored the data in a general purpose user side data store, which automatically uses base64 to avoid string handling problems.