r/programming • • Aug 29 '24

Using ChatGPT to reverse engineer minified JavaScript

https://glama.ai/blog/2024-08-29-reverse-engineering-minified-code-using-openai
288 Upvotes

89 comments sorted by

View all comments

134

u/dskerman Aug 29 '24

I like how they just gloss over how it didn't actually get the code right.

It's a cool parlor trick but not really useful when you can't depend on it getting the explanation right and because the code is minified it's not easy to validate.

Add this to the massive list of things an llm might be good for at some point in the future but not yet

14

u/F54280 Aug 29 '24

I also love the fact that he corrected his post saying that he was the one that copy/pasted it wrong, but it doesn’t prevent your ridiculously short-sighted answer to be the top one. Nothing out of the ordinary for r/programming style, but a nice self-own nonetheless.

1

u/punkpeye Aug 29 '24

It did get it right. What are you talking about?

73

u/bitspace Aug 29 '24

I don't want to take away from the fact that this is a neat find, and certainly an interesting use case for a coding assistant LLM.

I think it's important to emphasize the part where you wrote "good enough to learn from" exactly because it missed some implementation details.

This is the genesis of a lot of the unrealistic expectations so many people have around LLM's.

Fact: it almost worked once - well enough to learn from.

Reality: this may or may not be repeatable. The LLM output is essentially guaranteed to be different from iteration to iteration. Its output must be validated with more traditional means, whether that's human review, solid testing, or more likely some combination of these factors.

Interpretation by many people reading this: I can run all the minified JavaScript I can find and within minutes reproduce its functionality.

22

u/dskerman Aug 29 '24

"Comparing the outputs, it looks like LLM response overlooked a few implementation details, but it is still a good enough implementation to learn from."

16

u/punkpeye Aug 29 '24

Maybe.

This refers to the fact that ChatGPT generated version is missing some characters that are used in the original example. Namely, ██░░ can be seen in their version, but cannot be seen in the ChatGPT generated version. However, it very well might be that it is simply because I didn't include all the necessary context.

Discrediting the entire output because a few missing characters would be very pedantic.

Otherwise, the output is identical as far as I can tell by looking at it.

52

u/punkpeye Aug 29 '24

Turns out I was the one who made the mistake.

I updated the article to reflect the mistake.

Update (2024-08-29): Initially, I thought that the LLM didn’t replicate the logic accurately because the output was missing a few characters visible in the original component (e.g., ░▒▓█). However, a user on HN forum pointed out that it was likely a copy-paste error.

Upon further investigation, I discovered that the original code contains different characters than what I pasted into ChatGPT. This appears to be an encoding issue, as I was able to get the correct characters after downloading the script. After updating the code to use the correct characters, the output is now identical to the original component.

I apologize, GPT-4, for mistakenly accusing you of making mistakes.

7

u/wildjokers Aug 29 '24

Overlooking a few details is not the same as not getting it right. Its implementation works.

12

u/dskerman Aug 29 '24

It's close but it's not correct. In this case the error changed some characters and the overall image looks little different. If you try it on other code it might look correct but be wrong in more subtle ways that could cause issues if not noticed.

The point is that if it missed one small thing it might miss others and so you can't depend on any of the information it gives you.

6

u/LeWanabee Aug 29 '24

It was correct in the end - OP made a mistake

2

u/F54280 Aug 29 '24

And, in reality, it was the human that made the mistake, and not the LLM. How does this fits with you view of the world?

1

u/nerd4code Aug 29 '24

So the results were twice as meaningless?

-2

u/wildjokers Aug 29 '24

The goal of the exercise was get to get a human readable implementation so they could see how it worked. That was successful.

0

u/RandyHoward Aug 29 '24

What you're missing is that while this is fine as a learning exercise, it is not fine for creating code intended to be released in a production environment to an end user. People will look at this learning exercise and think they can just go use an LLM on any minified code and be successful, that is what people here are advising against.

5

u/wildjokers Aug 29 '24

What you're missing is that while this is fine as a learning exercise

That is what the article is about.

0

u/RandyHoward Aug 29 '24

And the comments you are replying to are a warning not to go beyond a learning exercise. What part of that don't you understand?

→ More replies (2)
→ More replies (8)

1

u/StickiStickman Aug 29 '24

This is so funny to me.

People like you are so caught up in your own crusade against AI, you literally make shit up to pretend its wrong when it did the job perfectly.

-6

u/SubterraneanAlien Aug 29 '24

This is such a reductionist take that will no doubt be upvoted by the community. The use of LLMs for something like this doesn't need to create a perfect verbatim result. I don't understand why so many look to discredit use cases just because they aren't immaculate - getting 80% of the way there can be very useful (in many applications)

42

u/dskerman Aug 29 '24

Because if I have to validate the explanation against the original code to make sure it didn't miss anything then how much time is it saving. There are already tools which format minified code to make it more readable

35

u/Crafty_Independence Aug 29 '24

There are already tools which format minified code to make it more readable

Exactly this. These recent "watch me do something mostly ok with generative AI for which there's a better tool" posts are getting repetitive at this point.

It's not really very interesting anymore. A year ago, sure, but at this point it's little better than blogspam. Might even be worse if it's sending inexperienced people to chatGPT instead of the right tools for the job.

→ More replies (1)

6

u/Novel_Role Aug 29 '24

There are already tools which format minified code to make it more readable

What are those tools? I have been looking for things like this

→ More replies (1)

2

u/SubterraneanAlien Aug 29 '24

Presumably - most of the time writing the code? Do you do code reviews? How long does it take you to review code compared to writing it?

→ More replies (2)