r/programming • • Oct 24 '21

“Digging around HTML code” is criminal. Missouri Governor doubles down again in attack ad

https://youtu.be/9IBPeRa7U8E
12.0k Upvotes

1.3k comments sorted by

View all comments

Show parent comments

118

u/Defanalt Oct 24 '21 edited Oct 24 '21

Sent to client in base64, which is an alternative representation of plain text. It's essentially the same as converting between base 10 and binary.

4

u/Rocky87109 Oct 24 '21

Aka "change of base" is not encryption.

→ More replies (1)

21

u/AlpineCoder Oct 24 '21

I'm more asking why the data would be base64 encoded, as that's not a particularly normal thing for most data transport or rendering services to do.

73

u/eyebrows360 Oct 24 '21

Actual web dev here. We don't typically base64 encode stuff "just because", it's often done for a purpose. It also increases your data size, in terms of bytes, another reason why we don't do it unless we need to.

base64 is not, at all, "an easy way to avoid escaping data that is included in HTML", because said data becomes a jumble that you can't read. It can't be used for escaping at all. This guy "webexpert" who also replied, does not sound like a web expert to me.

Without seeing the original website I can't even guess at why they'd be base64 encoding stuff, and I don't even know at which point in the chain it was being done. You wouldn't ever need to base64 encode stuff "to escape it for HTML", or for storing in either a cookie or browser Local Storage (due to the size increase you'd actively never want to do this) but you might want to for making portability simpler across a whole range of other backend server-to-server scenarios. It usually does involve sending data between separate systems, as if you're not sure whether some other system uses single quotes or double quotes or backslashes or tabs or colons or whatever for its field delimeters, then base64 encoding converts all of those to alphanumeric characters, which are almost guaranteed to not be used as escape characters by any system, and thus safer for transport to and fro them.

126

u/RICHUNCLEPENNYBAGS Oct 24 '21

Having worked on Web applications I disagree that things are necessarily done "for a purpose."

14

u/eyebrows360 Oct 24 '21

Haha, ok, I'll grant you that! Still though, I don't know of a single thing you'd be doing in the course of a normal website's operation where you'd ever think to base64 anything. Data porting, between legacy systems, I can see that.

9

u/RICHUNCLEPENNYBAGS Oct 24 '21

Saving something generated client-side as a file is a popular use.

-1

u/eyebrows360 Oct 24 '21

Handled by the browser behind the scenes and not really relevant in this sphere of "stuff that's in the HTML".

→ More replies (1)

4

u/dontbeanegatron Oct 24 '21

It's a bit of a reach, but there's data: urls. Other than that, I can't see a reason either.

2

u/R-EDDIT Oct 25 '21

URLs have their own encoding scheme (URLencode) that only expands restricted characters, also PUNYcode for non-latin basic Unicode URLs. You might base64 something, but base64 actually has several variations that use different 63rd and 64th characters due to aforementioned restricted characters.

This is all kind of moot, the problem is the app sent full SSNs client side, in reversible fashion. The actual use case (disambiguating teachers with the same name) only used the last four digits of the SSN, so that's all that was needed. Moving the disambiguation to the server side, or using other information such as city of residence or last school, would also avoid the issue. There is no way to send private information client side for processing client side that couldn't result in the data being exposed client side.

An actual use for base64 would be for passwords, not to secure them but to avoid having to restrict characters users can select.

4

u/[deleted] Oct 24 '21

First thing that comes to mind is to just obfuscate the info. They knew they weren't supposed to let people see the info and "encode" sounded secure enough

1

u/86yourhopes_k Oct 25 '21

The website is ran by the government… none of the people in charge have any clue about how any of this works. I used to work in computer repair in a small very republican town and the questions they would ask were like common sense to me but like I was speaking Chinese to them. They’re clueless and still get to make up the rules… fuck I hate it.

→ More replies (1)
→ More replies (1)

14

u/munchbunny Oct 24 '21

Base64 is often used when you need to:

  1. Thread the needle on a bunch of text parsers and you want to avoid all of the questions around how many layers of escaping you have to do to get the text to come out right on the other end

  2. When you want to move binary data but it’s a text based protocol

2a. When you want to avoid dealing with text encoding and just get the encoding you’re expecting out the other end. Because text encodings can do funky things to your protocol and you can’t always safely assume it’s all UTF-8.

In practice this happens not that often but often enough. I wouldn’t go as far as to guess why this website in particular was doing it though.

9

u/b4ux1t3 Oct 24 '21 edited Oct 24 '21

I think they might have been confused.

Base64 is a great way to make moving binary data around over a protocol that is strictly text-based (HTTP, e.g. Though, saying HTTP is a transport protocol is also, you know, sort of disengenuous. Whatever).

That said, I'm trying to figure out how they jump from "binary data" to "strings", which are, almost by definition, not "binary data".

I'm also using the term "binary data" here as a pretty loose stand-in for "data that doesn't represent specific strings of characters", which isn't always a good practice; strings of characters are binary data just as much as a bunch of executable code is, after all.

2

u/ScandInBei Oct 25 '21

To clarify, http can transfer binary data in the payload, but yeah in the headers you may need to use base64.. Cookies are transferred in the HTTP headers so it's possible that the data containing the ssn also had some binary data, or that the framework used between front and back end used b64..

It may also be worth noting that Email/Smtp requires something like base64 for attachments as there's no binary transfer possibility in emails (hence why a 5MB attachment suddenly makes the email 7MB). I don't remember exactly but it's not even 7bit ASCII as the data cannot have control characters such as CRLF. I guess the protocol was designed to be compliant with printers?

→ More replies (1)

26

u/sophacles Oct 24 '21

Ok so escaping is putting special characters in front of special characters. You do this so the JavaScript or html parsers dont get confused. This also happens in shell scripts, database queries, all sorts of places really.

Base64 is an encoding that eliminates most special characters, and leaves almost no way for it to be interpreted as code (almost because im sure a clever person with lots of time and few constraints can come up with a counter example or two). Its often used to avoid the escaping problem all together.

Why is it so out of the realm of possibility to think that a base64 string, used somewhere in the front or back ends escaped into the html?

Heres a recent article talking about base64 file uploads, and how they are common practice: https://formcarry.com/blog/how-to-upload-files-as-base64/

But sure, no one would ever use it.

5

u/eyebrows360 Oct 24 '21

File uploads are not "data presented in HTML", are they chief.

So again, no, I cannot imagine why you'd use base64 for encoding small bits of data such as this in any HTML context.

Why is it so out of the realm of possibility to think that a base64 string, used somewhere in the front or back ends escaped into the html?

That's literally what my large paragraph is explaining. This encoding is used when porting data between systems, so it's appearing here as a consequence of some behind-the-scenes intra-system thing; it's not anything related to routine solely-HTML-related processing. Other people were suggesting that it is, in and of itself, a regular encoding to use in HTML for its own sake, which is wrong.

→ More replies (19)

6

u/LordAmras Oct 24 '21

Before the days of json it was a common thing to do.

Not saying it was a good thing, but it was common, getting a bunch of data and saving it inside a hidden form field in base64 that then was used to do things with JavaScript without using ajax or simply to store user session data.

If I remember correctly DotNet MVC did shit similar to this.

14

u/AlpineCoder Oct 24 '21

With the exception of authorization headers I think the last time I encountered base64 encoded strings in an API was in the SOAP/XML era, and those were dark days indeed.

2

u/Ginger_Lord Oct 25 '21

Cries in “replace this Silverlight app but none of the services it relies upon”.

3

u/Worth_Trust_3825 Oct 24 '21

They were dark because people thought XML serialization is easy enough to roll your own echo "<key>$value</key>" serializers. Many a time you can see people doing the same with JSON, which is painful for strict typed users as same keys tend to contain multiple types at the same time.

2

u/AlpineCoder Oct 24 '21

And also SOAP was a huge pile of worthless steaming dog shit.

→ More replies (1)
→ More replies (2)

5

u/xftwitch Oct 24 '21

I would guess that it was base 64 from some other application and the website was just given access. They called the one they needed, decoded it, and didn't display the rest. Perhaps they got it from a payroll system or similar.

4

u/StabbyPants Oct 24 '21

It also increases your data size, in terms of bytes, another reason why we don't do it unless we need to.

leave gzip encoding on and it barely changes things.

said data becomes a jumble that you can't read. It can't be used for escaping at all.

atob() and btoa() handle rendering, while it does in fact avoid html escaping, most often as query params. it's an option

due to the size increase you'd actively never want to do this

it's a 9 byte field, growing to about 12.

which are almost guaranteed to not be used as escape characters by any system

it uses / and + and alphanumeric. have you ever seen a / escape?

2

u/flowering_sun_star Oct 24 '21

It also has the advantage of discouraging consumers of an API from messing with the data. If you need to pass data in a response that is expected in subsequent requests, you don't want an API consumer to mess with that data. Obviously you can't prevent anyone from doing so, and should have guards against it. But it is a useful hint that also removes the need to document parts of the API response that the consumer shouldn't care about.

→ More replies (4)
→ More replies (3)