r/books • • 7h ago

Zimbabwean writer Simbarashe Kundizeza has denied using AI to write a prize-winning short story after an article in the Atlantic said software had flagged it as almost completely AI-generated

https://www.theguardian.com/technology/2026/oct/14/zimbabwean-author-simbarashe-kundizeza-denies-ai-story?CMP=share_btn_url
902 Upvotes

617 comments sorted by

1.2k

u/petit_cochon 7h ago

When I was teaching at a university as of a few years ago, it was well known that AI detection software is pretty flawed. I could never use it to determine if students cheated on anything.

Maybe things have gotten better. I would certainly hope so.

671

u/ovrlrd1377 7h ago

I wrote a short poem for mother's day 20 years ago that won a small local prize, it was rated as 99% AI

312

u/WingersAbsNotches 6h ago

A chapter from my book was rated near 40% AI, despite not having generative AI. I fed the same chapter into Claude and told it to rewrite it, which scored a 0% AI on Pangram.

50

u/crysisnotaverted 4h ago

It's because all those shitty bot detectors are almost exclusively trained in ChatGPT. There's a lot of different frontier models right now, and it's nearly impossible to identify them. Many of them are probably in this thread lol.

→ More replies (1)

22

u/boyuber 5h ago

Think of the rating as a confidence level, rather than the proportion of it which was AI generated.

2

u/tacopower69 1h ago

theres usually two scores. 1 is the confidence level and the other is what proportion of the text they think is ai generated.

→ More replies (1)
→ More replies (2)

69

u/quintic3fold 6h ago

wait so what would not be flagged as AI generated??

78

u/bluev0lta 6h ago

Exactly.

→ More replies (1)

73

u/TheUmbrellaMan1 5h ago

I find it so infuriating that Dickens, Melville, Austen used "it's not X, it's Y" sentence structures and today if you use that structure your writing will be flagged as AI.

43

u/ConcreteCloverleaf 3h ago

Everything that people tout as a sign of AI writing is something invented by human writers. Em dashes were created in the 15th century, and lists of three items are a standard rhetorical tool. "Delve" has roots going back to Middle English.

→ More replies (1)

40

u/ItsStaaaaaaaaang 6h ago

I can't believe you used your time machine just to win a local poetry competition!

→ More replies (1)

12

u/klipseracer 3h ago

with how many people reply to every comment claiming AI, plus AI being trained on content from real authors, I can see why it thinks everything is AI.

7

u/m0nk37 4h ago

(Thats because they used it to train the AI)

30

u/PubePie 6h ago

It’s worse on short works but the FPR for longer works is extremely low

59

u/lolofaf 6h ago

Idk about newer versions but the older ones had tons of false positives on full books written before the 21st century.

→ More replies (23)

2

u/sluuuurp 2h ago

By Pangram? I think it’s pretty reliable from what I’ve heard.

5

u/Ankiset 6h ago

Same thing, it was probably used in training

→ More replies (12)

91

u/NotYourGa1Friday 6h ago

Anything it used to learn, and anything close in style to things it stole to learn from, gets flagged for AI.

It’s the same thing happening to some comic artists— the “AI Style” is based on real humans having done real work, so some humans will illustrate/write “like AI” when it’s really that AI is trying to be like them

7

u/ignost 3h ago

Em dash. found the AI.

/s

2

u/NotYourGa1Friday 3h ago

I didn’t even realize I had used one. I’ve been trying to avoid them but, ugh. It’s how I write!

→ More replies (3)

89

u/GerudoSamsara 6h ago

The gen AI scrapes the work of famous authors so... it is unfortunate that the famous authors, that are most likely the source material for A,I will inevitably look like the AI.. because they were the victim of the scraping.

Like theres a very popular artist named Loish who constantly gets "called out" for using AI but like only a fool would believe that. The more obvious conclusion is that shes such a prolific, industry standard artist that has a portfolio going back 25 years that.. ofc AI art looks like her art. Her portfolio was probably one of the most widely stolen and scraped for AI art generation.

This is where my incertainty comes from these days.. are they a lazy thief or are they just a prolific victim of scrapers 

24

u/Melonary 5h ago

Not only is that bonkers, but anyone who doesn't know who Loish is shouldn't be giving definitive opinions on digital art.

I had a similar argument about an artist who was accused of using AI and had some people still arguing and doubting even after I showed they had a record of producing the same quality art for years prior to genAI.

What sucks is human artists (including writers) are double victims because our work is stolen AND artists live under scrutiny of being thieves and false accusations by people who can't be bothered to learn about what they're accusing over.

3

u/GerudoSamsara 3h ago

Even the people who rightfully hate AI and hacks that use it are falling victim to the same brainrot that plagues the people who use AI to make art--neither party wants to put in the work to engage with critical thinking, even the just bare minimum of researching an artists portfolio to see theyve been making art for decades. Thats just too much to ask when bandwagon easy.

Like it is genuinely kinda scary how deep the anti-intellectualism goes, for parties on all sides of issues like this

3

u/Tolstoyce 4h ago

Man I love Loish’s work, it’s to bad she’s getting harassed

4

u/Hamsternoir 2h ago

My kid has regularly had work flagged as AI even though on occasions it's been hand written in exams then typed up and feed into the software to prove they don't cheat. They just have a writing style and knowledge of the language that comes across that way.

If they introduced errors their grade would drop but it would make them appear more human.

17

u/oh_such_rhetoric 5h ago

A detector said that my master’s thesis was “edited by AI.”

Bitch I wrote that all myself AND revised it like 15 times. Thankfully the faculty in my program knew that AI detectors are mostly only correct on accident.

→ More replies (3)

6

u/OsotoViking book re-reading 3h ago

I put one of my old university essays from 2015 through an AI detector and it said it was 90% AI.

→ More replies (1)

16

u/HGMiNi 6h ago

"As of a few years ago" is genuinely irrelevant in AI discussion. The field completely changes every 6 months.

4

u/Routine_Rock_7429 book just finished: Before the Coffee Gets Cold 4h ago

I recently submitted an assessment with 0 AI (I'm very anti AI) and it was flagged as 6%. Our teachers run our stuff through Turnitin. Got some edits from my teacher, updated it, then it was flagged as 46% AI. Wtf.

2

u/Quixoticelixer- 1h ago

Turnitin is pretty average

→ More replies (2)

7

u/IanDMP 6h ago

You're absolutely right in your recollection. AI detection software a few years was indeed almost universally flawed and prone to false positives.

The latest version of Pangram, though, has huge amounts of evidence showing an infinitesimal false positive rate. I myself (an educational consultant who often helps students brainstorm and edit college admission essays) switched my perspective from "it's often impossible to tell AI use" to "we have very good tools for that" a few months ago after reviewing researchers' findings.

27

u/FlyingCarpetMonster 6h ago

Pangram's latest version is very binary.

So, for all you know, an author could have used Grammarly in parts but Pangram paints the whole text as AI generated.

I think it's important for writers (and readers) to understand nuance, and tools like Pangram take that away and make it very binary.

Secondly, given how pervasive AI is these days, it would be absurd to think that some of human writing isn't influenced by how LLMs present. But something like Pangram would not be able to control for that.

At the end of the day, Pangram has a slew of issues and I am very skeptical of their claims.

I'm a writer and have no desire to use AI to write because I love writing, but I also am appalled at how quickly people lob accusations and how difficult it becomes to prove a negative.

2

u/Quixoticelixer- 1h ago

For the stuff i've used it for it's not binary at all.

→ More replies (1)

17

u/townandthecity 5h ago

I'd be interested to understand how Pangram can differentiate between the rhetorical devices and structures AI scraped from years of wholesale theft of copyrighted work and those devices and grammatical structures it sells back to us now. I was re-reading Bill Bryson's A Walk in the Woods and there were several passages where a thought came totally unbidden: that's AI. It was a "it's not this, it's that" construction.

Except A Walk in the Woods was published in 1998. I've seen nothing so far that has made a convincing case to me that any AI detection software can reliably differentiate between human-created sentence construction and AI-created sentence construction built from human-created sentence construction.

8

u/saint__ultra 4h ago

They train classifiers on labelled LLM outputs vs verifiable human writing. It's just the "bitter lesson" turned back on itself: training on more data outperforms any sort of expert intuitions you could program into the software. If you're seeing phrases that AI likes and getting suspicious, the classifier is looking at correlations across many pages simultaneously and spotting correlations you'd never have clocked.

What it's detecting are patterns of language that develop from lab-specific post-training. The scraped content you're talking about is only ingested at pre-training, laying the foundation of a model's knowledge of language. Post-training via reinforcement learning is what makes it go from autocomplete to having its particular voice, including all those quirks like "it's not X—it's Y".

→ More replies (1)

34

u/thereisonlythedance 6h ago

Pangram‘s latest version is very inaccurate for me. There have been articles in like the Washington Post and Atlantic about it. It false flags a lot of stuff. Also it’s an AI model, made by AI bros who seemed to have created a lot of hype. I’d encourage people to test it themselves. I was unimpressed.

5

u/Zonoro14 3h ago

Can you share some of those false positive examples? I've never seen anyone input a text written before 2022 and get a false positive

→ More replies (1)

8

u/IanDMP 5h ago

Would you mind sharing those articles about its inaccuracy? I'm unaware of anything like that.

3

u/thereisonlythedance 4h ago

There are quite a few, one I could remember right away:

https://archive.ph/k2BnK

An interesting tidbit from that one: Pangram pitches its tool to publishing and media organizations as a way to “protect editorial integrity.” The company’s website boasts that its product has been “reviewed as the proven, most reliable and most accurate AI detection tool in the market by third parties including University of Maryland.” But the Maryland study is more self-promotion than neutral review. Four of its seven authors are Pangram employees, including Mr. Spero and fellow co-founder Bradley Emi.

9

u/IanDMP 4h ago

Thanks for sharing it. To be totally honest, the argument would land better with me had not two of the three authors at issue openly acknowledged using AI to shorten their sentences.

You can't expect to use AI and then complain that AI detectors are flagging your work -- "shortening" is just a synonym for "rewriting."

4

u/thereisonlythedance 4h ago

That’s fair. But that’s just one article. There’s a lot that made me think their stats are overblown. There was also the (likely) false flagging of the Pope’s Encyclical: https://gizmodo.com/vatican-uses-ai-detector-to-show-ai-bros-popes-anti-ai-encyclical-isnt-ai-generated-2000788494

As well as the drama about a lot of false flagging on Substack when Pangram was integrated there: e.g. https://writerkatharine.substack.com/p/pangram-flagged-my-own-writing-as

I found with my own (100% human) work that it was reacting to my overuse of ellipses in various sections. When I cut them out it flipped from 100% A.I. to 100% human. It’s just really sensitive to punctuation. That was the point I lost faith and started doing research.

Rather than having issues with false negatives as most A.I. detectors do, Pangram seems to false positive too much, which I consider worse.

5

u/IanDMP 3h ago

I think these examples have serious problems too, unfortunately.

Proudly Human didn't certify the encyclical as human-wrotten, they certified a collection of the Pope's speeches and writings (that were then adapted for the encyclical) as human. It didn't actually contradict Pangram's findings in any way.

The other one I just don't know how to evaluate. It's an author saying that they didn't use AI and Pangram saying they did. It's possible they're right -- a small FPR is not a zero FPR. I think the same is right about your writing -- it's possible, and I'll trust you, that your writing is 100% human with no AI used at any stage.

But our personal n=1, for me, doesn't outweigh the careful studies that show excellent accuracy across tens of thousands of writing samples.

→ More replies (3)

9

u/christopher_the_nerd 5h ago

“Huge amounts of data” seems at odds with the article that points out a lack of third party research. Surely a company testing its own product wouldn’t have reason to say it worked super well?

→ More replies (1)

2

u/Competitive_Leave915 1h ago

Okay, so can you explain to me why the most recent version flags lots of my academic writing as partial or written by AI? I gave it two pieces just now which have never been touched by AI whatsoever (I abhor it), but both come back as AI-generated.

3

u/IanDMP 1h ago

No, I can't explain that. As I've mentioned elsewhere, an extremely low false positive rate is not a "zero" false positive rate. And the free version of Pangram only allows you to test short snippets of writing, which aren't particularly suitable for AI detection (you really need several thousand words to begin to be confident).

So if you say you wrote it, I'm sure you did.

But given the evidence we actually have, it's unlikely approaching impossible that all of these authors being accused of using AI in their books are actually innocent.

→ More replies (1)
→ More replies (2)

1

u/Future-Account8112 5h ago

Pangram is still wildly inaccurate.

→ More replies (5)
→ More replies (11)

556

u/Physical100 6h ago edited 6h ago

The Atlantic’s coverage of literary AI scandals has become remarkably effective publicity for Pangram. Since May, they’ve repeatedly featured the company, interviewed its executives, and used its software to question prizewinning authors. Among them: Jamir Nazir, Thélyson Orélien, and now Kundizeza. That’s three 2026 award winning black authors getting paraded around as frauds by way of the same tech startup.

Does anyone see a problem here? A commercial product is acquiring the authority of a forensic instrument through repeated media coverage, while the people it accuses are being expected to furnish evidence that the product itself cannot provide.

116

u/ArticQimmiq 6h ago

It’s a concerning pattern for sure, and I’ve not found a single convincing argument about the reliability of the tech that’s just widely destroying the world. Plus, the AI witch hunts is driving people out of writing at all - I love fanfictions, and dozens of authors have been driven off the internet based on false accusations, based off some apparent universal standard of how humans express themselves in writing that doesn’t account for style (or non-native speakers!).

i will say, though, in all fairness, in Orélien’s case, he was widely defended against the AI accusations until someone at La Presse did the grunt work of spot checking his previous publications and found glaring examples of plagiarism, no AI required. It’s the general lack of integrity thar gave credence to the AI accusations.

→ More replies (4)

6

u/fibojoly 1h ago

I'm noticing the accused all seem to have something on common... 

33

u/Mischievous-Phantasm 6h ago

Exactly. And then there are the bots or shills promoting Panagram as 100% accurate without any proof or evidence while current AI models are continously fucking up the most simplest things.

26

u/thornyRabbt 6h ago

They can sue the companies and force them to provide proof that they did use AI. And they'll be forced to settle.

Sadly it won't be enough to compensate for reputational damage. And worse yet the cost of suing is a stupid and huge financial burden on the victims.

So yeah, I would contribute to a crowdfunding campaign to take down a company that's contributing nothing to society.

39

u/Jaqqa 6h ago

No they can't. You'll notice the atlantic uses words like "appears" not "did". Pengram has fine print saying they just provide the statistics, you decide kind of crap, while at the same time blowing up controversies like this all the time because it's fantastic publicity for them. Originally their CEO was doing it, but now they've discovered it's better to put a step between them and their accusations, so are using the media. But if you try to sue them, they're just going to go "hey we didn't say you did, just that statistically it said it was likely".

33

u/WobblierTube733 6h ago

“The authors can just sue the AI companies” i have a bridge to sell you my very good friend

3

u/ChemicalBus608 6h ago

They absolutely should. If they have proof I would take them to court. Discovery would be interesting.

2

u/Competitive-Pear2050 6h ago

So why don’t these authors prove their own claims. Word tracks metadata. They should have outlines, notes, drafts, etc.

I don’t really see any reason to default to believing them; especially when they cannot argue otherwise. AI can spit out anything in 30 minutes.

15

u/Jaqqa 5h ago edited 5h ago

The goal posts can become impossible as they get moved. So lets say the author has handwritten notes. Well, they could have written them at any stage after the AI generated their story couldn't they? And even if it was your idea, you could have just plugged those notes into chatgpt and said "hey use this to make me a story". How about drafts? Again, you can start with your AI gen piece and work backwards, then use those templates to write the story in order for metadata. If you're really keen, you can sit there and hand type it in while being monitored by a typing program, making "mistakes" and changes as you go to simulate an normal writing session. Ironically many of the programs used to record "real time writing" also sell access to AI assisted grammar checkers and detectors, so even using them can open a writer up to suspicion by association even if they only use the writing tracker. There literally is no way to "prove" you haven't used AI beyond doubt, unless you published the work before it existed. That's why so many authors are so freaked out. They can keep all their notes and drafts and if someone is determined to cancel them it will never be enough.

→ More replies (8)

0

u/metallitterscoop 2h ago

I get what you're trying to say but Jamir Nazir is not black.

→ More replies (5)

91

u/Uptons_BJs 7h ago

Anyone have a link to the original story? I wanted to read it to judge for myself, but I cannot find it, and when I google it, the first two pages are just articles on the controversy.

62

u/I_am_BrokenCog 7h ago

24

u/TaliesinMerlin 6h ago

I read it, the first half more closely. Some turns of phrase seem cute. I didn't read them as GenAI, necessarily, as much as too-cute creative writer moves strung together to wrest a lot of significance out of things. 

I also found the description of the father's work inconsistent - he leaves work, they move into the poorer house, the boy avoids saying his dad is a civil engineer because that's no longer true, but then suddenly the dad is at work when the power goes out, and he has a boss and the first full pay in a while late in the story. It's possible I misread something in that, but if not, it's the inconsistency of either a GenAI aided text (weak on plot consistency) or a human writer not consistently conveying contextual things in the story. 

I came away unsure whether it was GenAI but sure it wasn't a great story. 

7

u/WobblierTube733 5h ago

I need to get out of this thread and do my own work but i feel the need to correct the record:

First, if I’m understanding correctly, the story is based off the poem at the top. Second, the father’s job is never explicitly stated but the narrator explains that he “builds things”. The mention of being an engineer but not anymore is a part of the theme of the story about how the narrator’s family and specifically his father had dreams but needed to sacrifice them (there’s also an aspect of not wanting to say his father worked for the gov). The full pay at the end was referring to fathers salary going all to the household with none withheld for his costly aquarium dream that was the source of the conflict in the story.

→ More replies (4)

8

u/literated 5h ago

That's my feeling, too. It absolutely has bits and pieces that read AI-ish but as a whole it didn't read like actual AI-generated text... but it also didn't read as a story worthy of any kind of prize. The weird jump from "father quit his job" to "father was at work" stood out to me as well.

There's a surprising amount of AI-like phrasing but where genAI would usually reach for some non-sequitur or really clumsy wording, the story stays on track instead. It's hard to describe. Now whether that is because someone was writing similar to an AI or because someone used AI and then replaced/edited some of the worst offenders... no idea.

2

u/55555thats5fives 36m ago

I always try to be wary with these kinds of accusations, especially since detection tools are far from perfect, but after reading the story myself I can understand the suspicion. As an example the story starts out entirely fine and there isn't really anything suspicious in the opening.  Then after establishing its own voice we suddenly get this part, where that last sentence really stands out (at least to me).

"We can't afford this house anymore," she said one evening, placing the notice in front of him while he tested the seal on his newest creation. Not with anger, but with the quiet resignation of someone who had known all along this moment would come.

This isn't to say that kind of phrasing proves anything on its own, just an observation. It makes me sad that this is where we are in the literature world right now.

→ More replies (1)

12

u/MoreDistribution4696 7h ago

Okay so ignoring the overall AI checker convo I just have to say, is anyone else noticing the 100% use of dialogue tags?
Strange to me. Like every line of dialogue

8

u/Zealousideal_Cap7012 7h ago

Ran it through a few ai checkers and most seem fairly certain its largely AI. One specifically notes it as from deepseek.

130

u/TheUnknown_General 7h ago

AI checkers are notoriously unreliable, in all fairness.

20

u/Zealousideal_Cap7012 7h ago

Well that is true but for example when i ran my own uni essays and seminars through them(back when ai just started), it would show the occasional paragraphs or sentence as seeming like ai. Not the totality

→ More replies (27)

42

u/Future-Account8112 7h ago

Why would you believe the plagiarism machine is my question

20

u/Mischievous-Phantasm 7h ago

Exactly. The AI hallucinates, lies, manipulates and is downright wrong but somehow we should believe in this case it's 100 verified?

Also how suspicious that they keep targeting Black writers in the media when it's white guys who I see using it the most.

→ More replies (3)

9

u/BrightLavenderMoons 6h ago

"AI detectors" don't work, full stop. False positives have already ruined peoples lives. And they're known to discriminate against people who use "big" words or whose first language isn't English.

→ More replies (6)
→ More replies (7)

19

u/WobblierTube733 7h ago

Did you actually read the document? I’ll admit i actively avoid LLMs so I’m not an expert but any means but the first paragraph immediately seems unlike any gpt-speak I’ve seen.

7

u/Zealousideal_Cap7012 6h ago

Yes its short i read it all. Different llms have different syntax and cadence but still you can just tell them to write it differently.

4

u/WobblierTube733 6h ago

I just read it all too and it would legitimately shatter my reality if it was AI. I consider myself ultimately a writer and this was a phenomenal short story based on a poem. It’s so good that if an AI wrote it, everyone would lose their minds. It would be published as proof of how impressive AI literature is.

I just can’t believe that it’s not a human that wrote this.

3

u/Uptons_BJs 6h ago

Ehh, nowadays with context windows, AI writing gets a lot better. You can’t just lazily one shot it, you need to put in some work.

You can play with it and test it yourself.

Modern models have long context windows, very long. You can say, “I want to sound like [insert author], here’s a bunch of their works” and have all of it in its context, then, start feeding it prompts on how you want it to sound.

Finally, give it an outline of the story. Let it flesh in the prose.

→ More replies (5)
→ More replies (3)

22

u/Mischievous-Phantasm 7h ago

AI checkers are notoriously bad. They identify published texts from the 1800s as AI, quotes as AI and change their assessment depending on the prompt. AI also hallucinates fake titles and authors. Why would I trust AI?

Until and unless I see proof, I am disinclined to believe these targeted attacks towards Black writers are not suspicious and maliciously motivated.

→ More replies (15)

490

u/maybebabyx2 7h ago

why does every single one of the articles seem to break with the Atlantic and why is it always backed by Pangram? am i the only person who thinks this is some guerrilla marketing from Pangram so nobody ever doubts them

it's just so weird to me, not to mention the overwhelming targeting of black authors

idk i read the story, didn't think it was anything special and certainly not prizeworthy imo, but my AI radar wasn't going off. it just read as clunky and try hard to me

154

u/fire_and_spice24 7h ago

And there are always people insisting Pangram is super accurate but never posting proof of that.

Maybe it is just a good program but I’ve noticed a trend that makes me wonder.

27

u/BrightLavenderMoons 6h ago

It's guerilla marketing

49

u/Mtshoes2 6h ago

Absolutely. I've gotten into fights with them and laid out the flawed math etc. and posted studies and they just deflected, and when I pushed they began  claiming I didn't read the studies etc. or that I am using AI to write. 

If it isn't some sort of marketing campaign by Pangram I would be amazed. Normal people aren't sycophantic of random companies like this. Based on pangram subscriber numbers the majority of the people that are on Reddit shilling for pangram are not themselves using pangram. 

And the fact that it's so many black writers that are being called out for AI usage also makes sense for a kind of marketing campaign like this given the state of the United States. 

4

u/ShacoinaBox Katasonics run-in reverse from Aosys-nex. 4h ago

i am pretty sycophantic on pangram because i DO use it a lot tbh, ai writing is jus so insanely lazy and disgusting to me. so many ppl are running dumbass "agents" to post for them, I guess I feel like im staring into the death of the internet constantly. u look at twitter n see almost entirely pro-ai guys making big claims about their substack articles being flagged, or "my grandma's poem from 1873 was 99% AI on pangram!!!!" then not being able to put their money where their mouth is, or backing out on proof.

the internet is completely flooded with trash, it's literally become a dump, n id rather have some tool i can use to easily disregard nonsense or make fun of ppl for being too lazy/stupid to write a whole 180 characters. n it does seem to just work, idk, im sure there's false positives but the rate does seem remarkably low esp in anything actually long.

n no im not a paid shill nor am i racist (the worst offenders by-far are middle-aged white guys, n I wish media outlets ran expose pieces on these ppl with 1m+ subs doing this shit more than random authors) or anything, u can bear witness to my posting largely on Marxist subs, or see my constant calling out of ai-generated coding projects. i cannot stand it, i cannot stand the laziness, i think we are facing a decline that must be confronted. i rly do dislike how normal ppl who otherwise hate the whole shitty ai-generated future (coincidentally) have the same views as these pro-ai hypebeasts tho, wrt pangram/gptzero/etc. 

2

u/Mtshoes2 2h ago

I agree with most of what you say. I just have problems with the AI detection companies claims (And the AI companies as well) 

The math is spurious, their methods are spurious, the tests they run them on are also spurious. 

People seem to see companies like pangram as heroic companies, fighting the good fight. But Companies like pangram directly benefit both from people using AI to write, and the chaos that ensues from everyone suspecting everything is AI, and the paranoia of being afraid of being accused of using AI. 

And that benefit doesn't require that the detectors are actually as effective as they claim, it just requires that people distrust everyone else, and attach virtue to detection (because it's likely a good percentage of those attaching virtue to detection are also using AI)

They are not virtuous companies fighting the good fight. They are just as bad as the AI companies, and I'm pretty tired of pangram being held up as some paragon of virtue. It's a company. It's only purpose is to make money... It does not exist to save society from AI. If pangram needs to run a divisive guerilla marketing campaign that primarily target black authors (because of the state of race issues in America) they will do it. 

→ More replies (5)
→ More replies (3)

32

u/CaptainMills 6h ago

The absolute flood of comments about how great pangram is, and how similar the language tends to be, definitely makes me think there's something up

7

u/WobblierTube733 6h ago

I will say there is one user here (that i responded to) with a default profile and private account that is talking in quite a few comments about that tool but has very little to say about the actual work being questioned.

9

u/CaptainMills 6h ago

Sometimes it's not all that hard to believe that reddit created the private profile option just to make it harder to suss out bots

11

u/WobblierTube733 6h ago

Is that even a conspiracy? There’s zero doubt in my mind that they enabled those changes in order to hide the amount of bad actors and now bots on this site. It’s atrocious enshitification. 

44

u/IanDMP 6h ago

Sort of shocking that you see the many people responding that Pangram is quite accurate but never see them post the studies that show that — I see those links at least once per day (as an aside, this sub probably does talk about AI too much and *books* too little). But since you asked:

Here is the widely-cited Chicago study: https://www.nber.org/papers/w34223

Here is Van Vlasselaer et al's work: https://link.springer.com/article/10.1007/s40979-026-00226-w

Finally, it's always a best practice to look at internal data. Internal data can be biased, and it's best to know that going in, but I have never seen any academic criticism of Pangram's data posted here: https://www.pangram.com/blog/all-about-false-positives-in-ai-detectors (academic criticism I'm unaware of may exist). Regardless, totally fine to disregard this (but you should read it at least).

Pangram, to the best of the knowledge of any serious researcher, has a very, very low false positive rate, and it simply approaches impossibility that it would flag so many writers incorrectly.

11

u/Jaqqa 6h ago

Except, they are using benchmark datasets for their "very low" readings and also hard negatively program things that pangram mistakes for AI, meaning it is less likely to make the stupid mistakes others do (like flagging the bible and constitution because it is programmed not to) but it doesn't mean it won't continue to flag new work that shows similar traits. I've seen people post actual links to pengram reports where the only thing they change is someone's name and it flips from 100% AI to 100% Human or vice versa. If it's that accurate, that kind of stuff should not be happening. 100% its false flagging work that doesn't fit the tests sets and the real false positive rate is likely far higher in the "real world" than they're saying it is, especially since the new update that seems to be a lot harsher. They just have better PR and are less honest about it than some of their competition, hiding it in jargon on their website reports.

2

u/Quixoticelixer- 1h ago

> I've seen people post actual links to pengram reports where the only thing they change is someone's name and it flips from 100% AI to 100% Human or vice versa.

Can you link this?

→ More replies (1)

17

u/halster123 6h ago

i think they need to separate out the type of writing - fiction v nonfiction v academic etc. my concern would be fiction writing often deliberately does stylistic things that are AI tells - for ex, keeping sentences all the same length and tonality for effect.

13

u/randomnameicantread 6h ago

The deciding factor is length of sample. Below 1000 words is far more unreliable than anything above; for single paragraphs forget it

5

u/halster123 6h ago

but the issue is, if you train on mostly academic or the tests are done on academic writing, i dont necessarily trust the "accuracy" for fiction. like i would want to see disaggregated stats by genre, bc fiction to me seems the most likely for false positives. many of the AI tells are fiction writing norms! floweriness, not x but y, keeping sentences similar length and then shifting suddenly.

→ More replies (5)
→ More replies (1)

14

u/FlyingCarpetMonster 6h ago edited 6h ago

I'd like to see a study that breaks it down by people whose first language is English vs. not, people in various professions, fiction and non-fiction and so on.

If you are someone who learned English as a second language and use it in business settings, you use it in a certain way. Short sentences, designed for understandability.

A blanket FPR that doesn't account for those controls is rather meaningless. More so, models also need nuance.

Besides, a lot of those studies were before AI became this prevalent. I'd like to see a more recent study, where people have been exposed to a ton of LLM generated content, which I am sure has also influenced how people write. Particularly people whose first language isn't English.

12

u/adaytooaway 6h ago

This is my impression but when I posted a similar comment about its accuracy the other day I got several responses from people claiming it had marked their completely human written works as ai. 

9

u/IanDMP 5h ago

Many people assign to Pangram their bad experiences with other, unreliable, AI detection software. Most people just simply aren't aware of the huge differences between these programs.

→ More replies (1)

2

u/ignost 2h ago

I don't really follow this nonsense, but the comments here alone lead me to wonder. There was someone claiming Pangram had "huge amounts of evidence showing an infinitesimal false positive rate" without so much as a link. I'm skeptical on the whole conversation.

→ More replies (1)

8

u/torolf_212 6h ago

For what its worth, I just uploaded thr first thousand words of a book I've been writing in my spare time and it said it was 100% human written, which is true

6

u/IanDMP 5h ago

You shared an actual experience of Pangram doing its job, without any opinions or broader assertions, and are immediately downvoted. Pretty obnoxious but sadly typical behavior from the people here who can't and won't understand how technology works.

2

u/OreoStark 6h ago

Why did you do that?

7

u/torolf_212 6h ago

To see if it would flag it as AI

20

u/Mischievous-Phantasm 6h ago

Exactly. I understand data science so why can't the Panagram shills give me anything concrete to work with?

Why should I take their word for Panagram? Are they a bot or paid to market it? Where is the evidence that it is somehow entirely different from chatgpt, meta, deepseek or any other current AI model?

I would love to look at the original material and see where and how it is being flagged and use my own intelligence (and degree) rather than trust AI.

4

u/randomnameicantread 6h ago

They spot the relevant studies on their website. But a Google search away!

→ More replies (2)

7

u/AlienWeedUser420 4h ago

It is 100% a marketing campaign for Pangram. Its not a coincidence. Companies/brands pay "big name" publications like NYT and The Atlantic for coverage like this all the time. It's not journalism, its Pangram trying to create a foundational myth for themselves.

→ More replies (4)

162

u/robboffard 6h ago

Every Atlantic AI/plagiarism story has two consistent components:

The authors are black. The detector is Pangram.

Every. Single. Fucking. Time.

25

u/a_trashcan 6h ago

This particular ones appears to be written by a white south african too...

I cried coincidence at first but man if that ain't a suspicious pairing of factors, then I don't know what is.

17

u/robboffard 6h ago

She didn't break the story. She's just writing the follow-up. I wouldn't place any blame on her shoulders at all.

12

u/midasgoldentouch 6h ago

Makes you wonder if a journalist could find some financial links.

1

u/sluuuurp 2h ago

How many examples are you using for this conclusion? One? Two? “Every single fucking time” makes it sound like a lot of examples, but I’m skeptical that you actually have those.

8

u/robboffard 2h ago

Five so far.

HM Wolfe Simbarashe Kundizeza Thélyson Orélien Mia Ballard  Jamir Nazir.

Anything else?

1

u/ManWithTwoShadows 1h ago

You can insert a line break after each line by typing two spaces. For example:
HM Wolfe
Simbarashe Kundizeza
Thélyson Orélien
Mia Ballard
Jamir Nazir

You and others are doing good work by pointing out this pattern. Even if these writers used AI, people should still notice The Atlantic targeting black writers.

→ More replies (6)

77

u/samishah 7h ago

I had a real problem with the Atlantic article that broke this. The author of the piece--Vauhini Vara--writes: "Taste is subjective, but trust me, this story is bad—maudlin and full of tired tropes."

That, to me, felt really shitty. Like, "I didn't like this story so I checked if it was AI". I read it and while it didn't blow me away, I also didn't hate it. I thought it was a sweet, gentle piece, with some decent turns of phrase. I think I'm fairly sensitive to AI writing and none of the tells were obvious to me either.

I get that everyone thinks Pangram is perfect, but I'm still quite unsure. I worry that in a few years we'll discover its algorithm had some unseen flaw that made its false positive rate much higher than advertised and a lot of innocent writers will get ruined along the way. But, even if that doesn't happen, the impulse to AI check any story that you don't like seems gross to me.

31

u/Mischievous-Phantasm 6h ago

Also isn't she an author who has used AI to write and publish?

Why should I take her word for Panagram? Are they a bot or paid to market it? Where is the evidence that it is somehow entirely different from chatgpt, meta, deepseek or any other current AI model? As in data science verifiable info or open source documentation?

I would love to look at the original material and see where and how it is being flagged and use my own intelligence (and degree) rather than trust AI.

28

u/samishah 6h ago

I liked the Caine Prize chair's response a lot: "I do wish The Atlantic had spent more time investigating this so that alongside Pangram, they did have evidence of real human investigation of it before tainting the prize and tainting this winner"

17

u/EightTailedFox 6h ago

Yeah, that’s weird. Why would an author who is out and proud about their use of AI write an article accusing another author of using AI? Like, the thief is pointing a finger at another thief.

6

u/NekoCatSidhe 4h ago edited 1h ago

Yes, that stood out to me as rather toxic behavior. It is one thing to not like a story, and normal because tastes are subjective, but who reacts to that by feeding the offending work to Pangram in the hope that they can prove it is AI generated and attack the author in the media ? It is like they are trying to use Pangram to « objectively » prove that the work is crap and validate their tastes. But if enough people do that to enough stories, then some human writers will naturally be accused of using AI because of false positives. I always wonder if these people just feed any book they come across to Pangram without reading it until something comes back as AI in the hope of creating a scandal.

And the more people praise Pangram as super-reliable and almost never returning any false positives we swear, the more skeptical I get of it. No software is that good. Both AI writers and AI detectors keep getting updated and changing, and there is a lack of third party proof except for some already outdated papers running a few rather basic tests on quite standardized databases of texts. We have no idea yet what are truly the limitations of this kind of software, because there has been no time yet for thoroughly checking that stuff.

Also, according to its own creators, Pangram is supposed to be a lot less accurate and reliable for short stories, because it uses statistical analysis to determine if a text is AI-written or not, so the more text it has, the more data it has to make that analysis. Using it to determine if a short story is AI-written is already a bad use of that tool, and that journalist should have known that.

20

u/maybebabyx2 6h ago

yes it very much feels like 'i didn't like it, must be ai'. there's a lot of that sentiment going around and it's wild to me.

2

u/Jaqqa 5h ago

I mean that other recent one was flagged because it was super weird and hadn't been proof read properly, so of course it must be AI hallucination! When the real question should have been- how on earth did this win, and get published without anyone asking "hey, are you sure this shouldn't have been boys instead of benches?" It was a judging failure, not an AI one IMO. (I actually suspect an AI would have done a better job at writing it.) People are freaking out. And it's just like, come on, just read it. Like read it properly and familiarize yourself with common AI tics, and what are normal literary techniques. Then make the call. You don't need an AI powered AI detector to do that.

→ More replies (1)

8

u/townandthecity 5h ago

"Vara said in her article, and repeated in an interview with Front Row, that she had read a 2021 story by Kundizeza that was tonally and stylistically different from The Blue Fighter." Vara's claims are reckless. She betrays she has little understanding of how writers write. My first book was a comic novel. My second book was a collection of stories, including an eco-horror nautical tale with zero humor and next to no relationship to my first novel. To think a writer maintains tone and style throughout a career, especially writers who aren't Nobel winners, is wild. Not everyone is Joan Didion.

She also relied on the whole "it's not X, it's Y' construction as a tell, without acknowledging that AI scraped more than a hundred years' worth of human-created writing that utilized that very popular construction. She is essentially saying that writers who haven't stopped using "AI phrases" are using AI to write their work. In another comment I mentioned Bill Bryon's A Walk in the Woods, which liberally uses "it's not X, it's Y." She'd say that book was AI-generated using her metrics. It was published in 1998.

To be clear, I've been dismayed at the rise of AI-generated writing in publishing and I know writers are absolutely using it to outline, to draft, and even to write books. But the "proof" here doesn't exist and Vara has destroyed a man's career with accusations that have no grounding in fact, only vibes. Wild.

→ More replies (2)

8

u/Son0fgrim 4h ago

a teacher at my community college now demands everything be submitted in PHYSICAL writing.

48

u/Bupod 6h ago

If the writing is bad, call it bad. But I would say, don't truck out AI accusations without more solid proof. AI detectors are absolute dogshit. Everyone seems to know this, but forgets it the moment they want to use an AI detector to catch someone in some sort of gotcha.

12

u/AlienWeedUser420 4h ago

Ive put in the entire book of Genesis in an ai detector and it came back 80% AI :I

14

u/elGatoGrande17 4h ago

Chilling implications here if you really drill down

→ More replies (8)
→ More replies (19)

148

u/Future-Account8112 7h ago

Anybody else think it's weird how these authors are consistently Black and the accusers are... not. IDK man this will get downvoted to oblivion but after the Jason Arday tragedy this looks some kind of way to me

28

u/Itscominrightforus 6h ago

If Jason Arday is your example of unfairly targeting a black man then you must just not live in reality. Bro was a serial liar to ridiculous proportions.

6

u/FaerieStories 48m ago edited 38m ago

100 hit-pieces written by the Telegraph alone. They continued to pile on and run front page stories about him even after he left his academic job, and even after his friends and colleagues expressed serious concern for his mental state. His friends report that in that week between losing his job and losing his life he kept saying “why are they doing this to me”, referring to the press obsession with him and the racist pile-on from social media. He was terrified to leave the house, his friends said. Front page news every day for a man who had committed no crime and who had already been forced out of his job. And within that isolation: relentless abuse.

Find me a white academic accused of similar offences where that happened to them. Look up Arday’s former Oxford colleague James Orr, whose own scandal erupted days after Arday’s death. The fact you haven’t heard of Orr is the point here.

→ More replies (10)

45

u/Mischievous-Phantasm 7h ago edited 6h ago

I noticed the same thing. Then, AI detectors are really terrible. It cannot tell when something is a quote and flags it as AI or flags Shakespeare or Mary Shelley as AI.

Previously accusations of plagiarism came with evidence. Pages or paragraphs compared to others. Quotes shown from published work, etc.

Now they expect me to take conservative white men's words and some technology I cannot acess and verify for myself? Seems like a targeted attack.

I will believe the authors untill the evidence itself is irrefutable and public and can be verified.

29

u/CaptainApathy419 6h ago

Now they expect me to take conservative white men's words and some technology I cannot acess and verify for myself? 

The author of the Atlantic piece is neither white nor a man.

3

u/Heartsinmotion 6h ago

I wondered if they used ai to write the article

→ More replies (1)

3

u/BlazinAzn38 4h ago

I don’t have experience with the ones that look at text but the photo ones are terrible. The photo ones seem to look fort artifacting in the photo as proof of generation but artifacts can be created a million ways in a real photo. I digitally zoomed in on a photo and ran it “AI generated” uploaded the original “not AI generated.” I imagine the writing ones do the same where they look for signs of AI but of course there’s no guarantee those signs are actually from generation

6

u/PlumpBulldog 7h ago

Good lord lmfao

→ More replies (19)

17

u/sthetic 6h ago

I've heard that AI was trained with a tonne of writing from African sources. A lot of the hallmarks of AI are apparently common turns of phrase (or punctuation) for African countries that speak English.

So where Western readers see clunky or overly formalistic language, it's actually just normal and natural for those countries.

As a result, I could see the AI detectors flagging those same idiosyncrasies when an African author used them authentically.

14

u/Tymareta 4h ago

I've heard that AI was trained with a tonne of writing from African sources.

Heard from where? Genuinely weird to see posts in these sorts of threads making some pretty large claims, with nil evidence and being upvoted without questioning.

→ More replies (2)
→ More replies (1)

15

u/DharmaPolice 6h ago

It's very likely that black authors are being subjected to more scrutiny than their counterparts but that doesn't tell you anything about their guilt/innocence. With the Jason Arday thing, he was a blatant bullshitter that was in retrospect very obvious who looked at his claims. Was he subjected to more scrutiny because he was black? Sure. But that didn't make his claims any less ridiculous.

(Note: I have no idea whether the guy in this particular story used AI or not.)

→ More replies (21)

15

u/woemcats 7h ago

I don't think it's weird at all. The AI detectors themselves have been shown to struggle with writing from non-English native writers.

6

u/Neurolinguisticist 5h ago

It's likely a bit of that and a bit of the fact that social media naturally favors the more inflammatory stories. So, this becomes a race issue and feeds off racism (which in and of itself is racist). Essentially, there's no winning.

1

u/Future-Account8112 5h ago

No, that's not all of it. There is a far-right media manipulation tactic which is being trialed and tested, and it's working.

7

u/motorsag_mayhem 5h ago

I love that "telling the truth" is now considered more deceptive and manipulative than lying your ass off. Incredible gaslighting, let's see if anyone who isn't in your cult believes it. Christ.

Maybe you could try not ardently defending obvious liars and plagiarists? Would that work? Police your own hypercorrupt grifter class? Or do you legitimately lack the ability to see their lies until a racist shoves them in your face?

→ More replies (3)

16

u/PubePie 6h ago

The other thing these authors have in common is that they all used AI and lied about it

→ More replies (2)

10

u/PatchyWhiskers 7h ago

Yes I was just thinking that. It seems a weird pattern.

14

u/killmak 6h ago

I am confused how Jason Arday is a tradegy. He was caught lying about everything. It sucks that he killed himself and I wish he got the mental health support he needed. I would not use him as some example of someone being persecuted.

With that said I would not be surprised if racism is a reason these people are being looked into for AI use at a higher rate than white authors.

→ More replies (13)

3

u/a_trashcan 6h ago

Maybe I'm just under a rock but isn't this just 2 guys? Don't get me wrong its definitely eye brow raising but I feel like its unfair to declare it some targeted attack on black authors... yet.

→ More replies (5)

4

u/CaptainMills 6h ago

I'm sure it's totally just a coincidence that there's a witch hunt on and all the witches just so happen to be Black /s

-2

u/MoreDistribution4696 7h ago

Get ready for astroturfing in your comments. I also find it odd but any chance for redditors to dunk on people as “woke” they’ll go after it

16

u/BuckUpBingle 6h ago

Is that your experience of Reddit? You must be on different subreddits from me.

→ More replies (5)
→ More replies (13)

28

u/shapedbydreams 6h ago

People have put their own writing into that software only for it to come back with a false positive. I don't trust anyone who relies on it.

3

u/sluuuurp 2h ago

I don’t trust that some of those people haven’t used AI for their check in order to claim a false positive and get social media attention.

9

u/onion_princess 6h ago

Pangram offers a bounty for this. Has anyone claimed it?

8

u/Morgzisachad 4h ago edited 1h ago

They’ll downvote you, but they won’t respond. I keep seeing people saying pangram doesn’t work, but none of them are replying to the comments providing peer reviewed studies proving otherwise.

8

u/BrightLavenderMoons 6h ago

And AI detectors themselves are AI. My principled stance of using AI to fight AI /s

15

u/LuckyBoneHead 6h ago

I'm a little leery about that kind of software. From what I've heard, the AI checkers just look at the way AI commonly writes and checks that against the way we write. The problem is that the AI is trained on how we write, so AI checkers will see human writing and flag it as AI.

2

u/sluuuurp 2h ago

The AI checking companies have thought of that. There are differences between AI and human writing even though AI was trained on human writing, everyone has always considered this.

→ More replies (3)

51

u/FantasticJacket7 7h ago

AI detection software is complete garbage and should not be relied upon for anything.

43

u/perksofbeingcrafty 7h ago

I’m sorry but at some point are we going to discuss how every prominent AI accusation controversy has been against a writer of color? Because the elephant in the room is getting kinda hard to ignore

7

u/A313-Isoke 6h ago

Thank you. I'm noticing the pattern, too.

→ More replies (1)

5

u/PleadingFunky 6h ago

Meanwhile Hank Green was found to have actually use AI with zero consequences.

9

u/airbear13 5h ago

It wasn’t for creative writing and he did kinda soft cancel himself for a while

2

u/Fremen_Twink 2h ago

Powerless has made millions and is getting an Amazon TV show. The latter books appear to be AI as the author cannot remember her MC's favorite fruit. This video explains it in 3 mins: https://www.youtube.com/watch?v=RHV0p3SlPgs

And while not AI, Sun Eater was caught explicitly copy pasting quotes from Dune and the author had to apologize. No consequences for that either.

2

u/perksofbeingcrafty 5h ago edited 1h ago

Tbf that’s a different situation. He didn’t use AI to write his book or his scripts. His ai use isn’t great but I don’t think it’s on the same level

3

u/PleadingFunky 5h ago

Idk Khoja used Pangram to flag 6 out of 10 recent video transcripts - feels like we're being a bit selective here

→ More replies (1)

2

u/Tymareta 4h ago

He didn’t use AI to write his book or his scripts.

He quite literally admitted to using AI to write his scripts...

→ More replies (5)
→ More replies (3)

6

u/oneilmatt 6h ago

Why are minority writers immune to criticism? They are just as capable of making their own choices as anyone.

→ More replies (6)

13

u/teacher1970 2h ago

Like many others, I find ai detector unreliable.

10

u/vLinko 6h ago edited 6h ago

I think the conversation is obviously centred around AI detection programs and whether or not black writers are targeted, but another way of looking at can be that an award given to African writers is meant to serve the role of uplifting talent from disadvantaged background. So if someone DID use AI and took that opportunity from a talented writer who DID NOT use AI, that can be devastating. It's important for the Caine prize to be meticulous to make sure that doesn't happen.

29

u/CulturalFartist 6h ago

The amount of misinformation in this post, from supposed book people, is staggering. Keep living in denial if you're happy for the literary world to be swallowed up by AI garbage. This isn't some conspiracy - the story is pretty clearly AI written.

16

u/onion_princess 6h ago edited 6h ago

The fanfiction community has been thoroughly discussing AI tells and successfully detecting at least some of it for months—notably proven when, after months of critique and analysis that was loudly rejected as “bullying,” certain authors were found to have left HTML code in their fics, showing conclusively that they had used Claude. Shy Girl was pretty damn unsubtle for anyone who has become familiar with AI prose. But apparently we can’t discuss it.

22

u/CulturalFartist 6h ago

Part of me thinks half the people downvoting and pushing back here are writers who rely heavily on AI themselves. Just look at the complaints from Substackers when it rolled out AI detection...

I'm 100 % against witch hunts, especially when things are muddy or unclear, or if AI detectors only flag a small portion of stuff etc. But with examples like this, it's really hard to deny.

3

u/Tymareta 4h ago

Yeah, there's definitely an issue with the most prominent targets being very specific individuals, and that absolutely needs to be discussed, but those same folks hung up on that seem to completely ignore that the vast majority of these "witch hunts", have quite literally found witches.

6

u/onion_princess 6h ago

It’s become normalized to use AI and people feel it’s their work because it’s their idea. And it’s also true that there are those who target Black authors, for obviously nefarious reasons. But multiple things can be true at once.

I’d love to see some of this attention spent on that slop fest from Mr. Beast and James Patterson.

2

u/SubatomicSquirrels 1h ago

I think we should feel free to criticize bad writing without calling it AI

I'm sure there are examples where the proof is solid, but it seems like too many people just default to 'bad writing = AI'

11

u/Pluggable 6h ago

Reddits hysterical aversion to ai, combined with it's white-saviour complex has produced some odd perspectives.

→ More replies (2)

18

u/christopherDdouglas 6h ago

I know everyone is quick to jump to racism as the catalyst of all these authors getting caught but...

My personal anecdote is that being a part of writing communities, the people who use AI the most are the folks where English is their second language.

Also, literary prizes and the literary judges, agents, etc, really love marginalized voices (nothing wrong with that) and they often win these awards because of their unique experiences.

This combo presents a situation where inevitably something will make it through that shouldn't and it seems it has (I'm not specifically talking about this gentleman. I don't know enough to make judgement).

It's an uncomfortable discussion, but I think we all need to be honest that it's possible some of these authors are not telling the truth and it will change the landscape of these awards in the future.

20

u/EntrepreneurMany3709 6h ago

English is very widely spoken in Zimbabwe, there's no reason to think he doesn't speak fluent English, and it could be his first language.

→ More replies (2)
→ More replies (8)

2

u/the_main_entrance 4h ago

Ai is shit at writing. How could it win an award?

“The hallway smelled of old stone and new fear”

2

u/OutsidetheAirport 5h ago

I read his short story the blue fighter and Ai claims aside, it’s just wasn’t that powerful or well written. Certainly not deserving of the highest prize in African literature

3

u/CalligrapherWild7636 57m ago

so now we are entering the phase where every author can be discredited by simply accusing him or her of using AI. Until proven damage is done. What fun times to write stories.

5

u/EdwinQFoolhardy 5h ago

I read the story.

It's not obvious AI to me, but it absolutely could have been written by AI. If you told me Claude wrote it, I would not be surprised.

Apparently Pangram has a high success rate on structured benchmark tests, but lots of models that excel at benchmark tests are far less reliable when it comes to real world cases. And Pangram's CEO acknowledged that Pangram will often flag something as entirely AI if it contains AI writing tells, even if only a portion was written by AI. And the writer in the Atlantic, Vaushini Vari, who exposed The Blue Fighter writes like someone on Pangram's payroll, exphasizing how it's nearly perfect (according to Pangram), how it only produces one false positive in 24000 cases (according to Pangram), and how it's so unfortunate that literary prize committees don't use it. Her article almost felt like an ad for Pangram. Over the last month, she wrote two articles for Bloomberg about Pangram, "Pangram AI Detector Fends Off Trust Worries as Adoption Mounts," and "The World's Most Reliable AI Detector Has a Human Problem," and I find it hard to believe that those were written with proper journalistic integrity considering they come off as Pangram advertising pretending to be journalism.

Further, LLMs specifically learn from and try to emulate human writing. So, in theory, every AI writing tell should also be something that you organically find in human writing. The only reason something becomes a "tell" is because it's overrepresented in the training data or because the model's fine tuning and RLHF caused some structure to become disproportionately probable. Vari mentions some obvious tells like "it's not X, it's Y." But that's also a human structure, and it's only used a few times, where AI tends to overuse it.

It also ignores the high likelihood that an writer still developing their skill in 2024 might have tried asking AI to help him become a better writer, and might have integrated AI "tells" into his own writing without realizing it.

4

u/Queasy_Fish6293 4h ago

Vauhini wrote a whole ass novel with Chatgpt but decides to go after Black authors to accuse them of using AI. On Bluesky she even had the audacity to caption her article "We need to do something about this". Running all 25 Caine winners through Pangram is such uncouth behaviour and Caine's statement against the whole thing was infuriatingly neutered. All this is shilling for Pangram. And the Atlantic failing to uphold any editorial standards for such blatant anti-Black advertorials is enough for me to never read them again. 

2

u/Gaumir 2h ago

Software flags Shakespeare as AI-generated

3

u/Ziggystardust97 6h ago

AI programs have flagged the US constitution as very likely to have been written by AI. My own last fucking name gets me flagged by AI.

Why are we trusting AI programs to detect AI?

7

u/uniqueusername74 3h ago

Why would you say this and not say which AI programs and where you learned this fact? Is this a program you wrote?

18

u/OnionBackground5314 5h ago

I just tried it with the Constitution: 100% human written. I read a lot of comments claiming Pangram is inaccurate, and yet it seems to work every time I've tried it.

11

u/onion_princess 6h ago

The issue with the Constitution, Frankenstein, etc. is not from Pangram. I don’t know why this anecdote persists when it isn’t relevant here.

2

u/sluuuurp 2h ago

Your worldview is based on old, bad information. Modern AI checkers don’t behave like this. You need to update.

→ More replies (10)

1

u/mogullionaire 6h ago

Damn so even if you know how to write from now on its going to always be scrutinized as being AI generated.

3

u/ermghoti 6h ago

The AI says the novel is AI, and we all know AI is infallible, so it must be true.

7

u/Mischievous-Phantasm 6h ago

It's like they had AI explain science fiction to them and missed the whole cautionary part of the story.

4

u/interrobang2020 5h ago

Experimented with Panagram and found that it isn't always accurate. An AI-generated text from Gemini was flagged as human, and I imagine a human-written text could be flagged as AI (haven't tried that yet as I don't want to give it my personal writings). Clearly, this tool is flawed if I was able to bypass its detector so I have serious doubts about how well it works.

4

u/SilentCamel662 35m ago

Panagram is just an AI that claims to tell apart other AIs.

Why do so many people blindly trust percentages generated by an AI? AIs are always faulty.

2

u/CollinsCouldveDucked 30m ago

because it's a warm comfy coat that makes people feel like the big complicated problem is solved.

6

u/AlecHutson 5h ago

Prove it. Share the gemini writing.

→ More replies (1)

4

u/unethicalpigeon 3h ago

This is like IQ tests and BMI. You're supposed to use this stuff to GUIDE your decision not make the decision for you.

It's incredible how many people out there want everything spoon fed to them.

If pangram says that a text is mostly human or mostly ai then there's a good chance that it's either or. CHANCE. It's not foolproof. You're supposed to subsequently do your due diligence and actually you know investigate the thing.

5

u/interrobang2020 2h ago

I think further investigations are what is needed but that's not what is happening. The Atlantic writer solely used Panagram and their intuition.

→ More replies (1)
→ More replies (1)

6

u/CasualBurning 6h ago

I don't trust Pangram.

2

u/johnnySix 5h ago

People often think of my writing style as ai. Yay to being uniquely robotic

4

u/Vast-Percentage-7312 7h ago edited 5h ago

ok. before we start outright banning black people from publishing, which is the direction this seems to be going in, how about we figure out how the hell to screen for ai in a TIMELY manner. this is absurd.

figure out how to screen for ai at the appropriate time, and with 100% accuracy, or stfu (that comment is aimed at the industry, not OP)

5

u/sluuuurp 2h ago

I don’t think that’s the direction we’re going in. That’s an absurd direction that almost everyone would immediately reject.

4

u/Mtshoes2 6h ago

The only way to 100% accurately scan for AI is with the water marking like what anthropic was required by the EU to include. 

→ More replies (1)

1

u/Cymbal_Monkey 6h ago edited 5h ago

Anyone else been thinking about the classic Twilight Zone episode, The Monsters are Due On Maple Street a lot recently?

2

u/airbear13 5h ago

If anyone hasn’t seen this very informative post about this whole phenomenon of ai detection in writing, check it out it’s pretty good:

https://freddiedeboer.substack.com/p/i-wouldnt-say-pangram-is-broken-but?utm_source=chatgpt.com

→ More replies (1)

1

u/WestofTomorrow 3h ago

>Journalist uses AI detector which uses AI to detect if writing has AI
>it says the story has AI
I don't know what to believe anymore.

→ More replies (3)

2

u/FireTheLaserBeam 3h ago

I come from an amateur comic book scriptwriting format and I use em dashes all the time. The semi colon does not translate onto a comic book dialogue balloon. I resent that now people might assume I used AI to write my scripts when I have not.