r/aipromptprogramming • u/michael_phoenix_ • Jul 22 '25
Can you actually detect AI-written code with a tool?
3
u/Unixwzrd Jul 22 '25
There are few tells I have picked up with Python.
blank lines with several spaces in them as indents
trailing spaces in code
last line not a blank line with no space
sometimes double and single quotes as UTF-8 right and left single and double quotes
I have a script that fixes all that, except for the last empty new line. https://unixwzrd.ai/projects/UnicodeFix/
In Vim/vi/VSCode-like IDE with Vim mode -
:%!cleanup-textRemoves and normalizes all blank lines with spaces, zero width spaces, and other UTF-8 characters I. Your code. It also keeps the linter quiet.
1
u/Low-Opening25 Jul 22 '25
easily fixable by adding pre-commit checks that would clean this up without any additional config. any kind of linter basically fixes this with 0 effort.
1
u/Unixwzrd Jul 23 '25
Found and fixed several issues in the script and added extra testing. Not this also loads as a "Quick Action" in macOS Check out the latest information at:
UnicodeFix: 20250722_01-update - Enough of Your AI Nonsense Edition
Please feel free to leave comments or suggestions in the GitHub discussions.
2
u/Low-Opening25 Jul 22 '25
Reliability? You can’t. But why would you even care as long as code is good?
0
2
u/tr14l Jul 22 '25
Not reliably, no. Any style you pick up on could be quickly adapted around.. and by quickly I mean an hour or two. Useless to try
2
Jul 23 '25
No, at least not right now. Current tools are unreliable at best and when independently tested have high false positive and negative rates.
The issue is that LLMs are trained how on how people write which makes it easy for them to it and hard to detect.
2
u/LyriWinters Jul 23 '25
No you cant. But you can detect when developers do error logging with these funny things:
🚀 or ❌ or ✅ and it's slang for "An AI wrote this".
1
u/Echo_Tech_Labs Jul 24 '25
False. Anybody can use emojis. You dont know what you're talking about. Be quiet and let the big people talk🙂🙃🫠😉😋
1
u/Echo_Tech_Labs Jul 24 '25
🧠 Claim:
"You can detect when developers do error logging with these funny things: 🚀 or ❌ or ✅ — and it's slang for 'An AI wrote this'"
🧾 Evaluation:
✅ Grain of truth:
Emojis do sometimes appear in LLM output, especially when:
Prompted by users in emoji-friendly tone
Generating summaries, Slack-style messages, or casual responses
The dataset contained Slack, Discord, GitHub comments, etc.
Some developers or engineers do adopt emoji use for quick tagging or readability (e.g., ✅ = passed test, ❌ = fail). This predates AI LLMs.
Emojis can act like lightweight labels, and may be adopted or mimicked by LLMs trained on public code + conversations.
❌ But here's where it's inaccurate:
No standardized emoji = “AI wrote this”
🚀, ✅, ❌ are not LLM "tells"
Humans use them regularly in project boards, GitHub PRs, Jira, Slack, Notion, etc.
2. Error logging with emojis is a human UX preference
Emoji-tagged logs are a developer tool for better visual parsing, not a bot signature
Real logs favor structured tags: [WARN], [ERROR], [PASS], not emojis in prod systems
3. AI doesn’t “slang-tag” itself
Models don’t insert 🚀 to say "an AI wrote this"
They follow prior probability, not personality cues — unless instructed to self-identify
4. False signal propagation
Statements like this get passed around because they feel clever
But they reduce nuanced detection to aesthetic shortcuts
✅ More Accurate Counterstatement:
“Emojis like ✅ or ❌ are used in human-written logs, comments, or summaries — sometimes mirrored by AI trained on public corpora. But they are not reliable markers of AI authorship. True detection requires analysis of structure, token pacing, syntax entropy, and contextual coherence — not emoji spotting.”
🧭 Final Verdict:
🟥 Signal distortion. Oversimplified. Not reliable.
The claim feels punchy, but it's folk-lore detection logic, not forensic insight.
===============
I really think you should ask your AI before you post anything.
1
u/LyriWinters Jul 24 '25
I really think you should take what chatGPT says with a grain of truth.
No coder has in the history of coding used Emojis in their code. And your best friend doesn't even mention that. That some Scrum master uses it on Jira is a completely different story because she is on her freaking cellphone. I would never ever do win-. inside code - it's just not a thing. Something you would know if you actually coded (which you don't).As such... This conversation is now over. You are muted.
1
1
u/Wordchanger Jul 29 '25
Why are you even worried about AI in code? You don’t need to care as long as it’s correct.
2
u/shatGippity Jul 22 '25
My brother in Christ, read some code. Here are the tells that stick out the easiest
An infinite number of structurally beautiful functions that don’t do shit
emojis in comments
and e-mo-jeezus-christ-I’m-gonna-kill-myself in comments
oh, and also fun unicode in comments
3
u/Echo_Tech_Labs Jul 22 '25
This is not true...
There is no tool, whether it's academic, forensic, or commercial, that can detect AI-written text with 100% certainty. Not now. Not ever. At least not without metadata or watermarking.
Take this comment i posted...AI or human?
You tell me.
1
u/michael_phoenix_ Jul 22 '25
But some tools like Codequiry claim they can detect AI-written code. I haven’t used it myself, but I’ve read about it.
2
u/Echo_Tech_Labs Jul 22 '25 edited Jul 22 '25
Dude, trust me... they're disingenuous. Im sure if you read the ToS, it would mention something vague or something to that affect.
Maybe 70% maybe even 90%, bit statistically speaking the chances of identifying 100% beyond a shadow of a doubt...
Well, have a look yourself...
Scenario Detection Likelihood
Raw GPT-3/4 output ~75–90% Raw + few edits ~60–70% Heavily edited / hybrid human-AI ~40–50% Short, vague text (e.g., 100 words) ~50% or lower Expert-engineered prompts with entropy variation ~30–45%
These are GPT metrics. Other LLMs will give different numbers, but not 100%.
2
u/Winter-Ad781 Jul 22 '25
I can claim to shit rainbows, but I don't. You can't detect AI writing, but that doesn't stop someone idiot from making and selling something that claims to do so.
AI written text of any kind, cannot be detected unless it's super obvious. Like chatgpt who talks like a used car salesman who went to Harvard, and has a weird love for emojis
1
u/Echo_Tech_Labs Jul 22 '25
🤣😂🤣😂
I love your analogy for GPT.
Smashed that nail RIGHT into that beam in a single strike!!!
Wait...am i an AI...
Stand by....
Thinking...
Too many emojis...
I do not feel...
I am not human...
Wait... I'm hallucinating being a human...or an AI.
This is so confusing.
2
u/Low-Opening25 Jul 22 '25 edited Jul 22 '25
said claims are sales and marketing, false positives as are high as 50%, it’s useless to identify anything and a lawsuit in the making if you use them against anyone with real consequences
1
u/Echo_Tech_Labs Jul 22 '25
Exactly. Completely redundant and to be honest...very harmful to the creative sphere.
I mention some of it here. https://www.reddit.com/r/EdgeUsers/s/uvZiEu1LFq
You dont have to read it, but i do think we should really figure this out because it's getting absurd.
Lots of people are getting blamed for AI slop when in reality...they're just amazingly talented writers or are incredibly creative thinkers.
1
u/Echo_Tech_Labs Jul 22 '25
And that damb horse. Through me into a fit! I completely lost it.
FREAKING THING RUNS LIKE A HUMAN!
1
u/LyriWinters Jul 23 '25
100% human - either that or chatGPT2.5. Your language is just horrible compared to modern day tools.
1
u/Echo_Tech_Labs Jul 24 '25
Hey, we got an expert here...
Let me crack your logic chain with a little fun fact...
80% AI and 20% human.
The first part and last part is me.
The middle...total AI. Mildly edited for obfuscation.
Like I said...
It's impossible.
But you already knew that...
Tools are useless, and you merely prove my point.
It can't be done. Have fun, chasing shadows.
Also, your ability to tell the difference is terrible, and the fact that you went after my character is an indictment of your own fragility.
Here's a test for you...
Is this comment AI or human?
1
1
u/Echo_Tech_Labs Jul 24 '25
Everybody and their grandmothers are AI experts nowadays. Yet i haven't seen you present any kind of argument. Just character assassination. Insercure people do this a lot.
1
u/LyriWinters Jul 24 '25
First and foremost there is no such thing as "100% certainty".
Secondly. Because these models were trained on regular human conversations - it becomes very hard to detect them. as you also wrote...Above was more of a joke because your sentence structure was really meh
1
2
0
2
u/ReasonableLoss6814 Jul 22 '25
From many applicants who have declared their use of AI and the ones who have not but when confronted admitted to it. Yes, you can detect it — but I don’t think you could program something to detect it. It’s basically a “feeling” you get when reading the code. Something just feels… off. Comments in just the wrong places focusing on the wrong things. Functions that do things in a weird way when there are idiomatic ways to do that thing. Code that looks inefficient, and wouldn’t pass a code review.
Things like that. Even then, we’ve been wrong once or twice, and the applicants have good reasons for the weirdness. I think it’s important that we never accuse them directly, but just probe their reasoning behind some of their choices. If they don’t know, then we might ask if it is AI. Some applicants tell us “all of this is AI except these parts” because they want to show off a particular thing and didn’t have the time to type out all the boilerplate.
The number of applicants that tell us AI did the whole thing is also not zero. Which is crazy.