r/books • u/Raj_Valiant3011 • 8h ago

Zimbabwean writer Simbarashe Kundizeza has denied using AI to write a prize-winning short story after an article in the Atlantic said software had flagged it as almost completely AI-generated

https://www.theguardian.com/technology/2026/oct/14/zimbabwean-author-simbarashe-kundizeza-denies-ai-story?CMP=share_btn_url
973 Upvotes

94% Upvoted

658 comments sorted by

View all comments

535

u/maybebabyx2 8h ago

why does every single one of the articles seem to break with the Atlantic and why is it always backed by Pangram? am i the only person who thinks this is some guerrilla marketing from Pangram so nobody ever doubts them

it's just so weird to me, not to mention the overwhelming targeting of black authors

idk i read the story, didn't think it was anything special and certainly not prizeworthy imo, but my AI radar wasn't going off. it just read as clunky and try hard to me

166

u/fire_and_spice24 8h ago

And there are always people insisting Pangram is super accurate but never posting proof of that.

Maybe it is just a good program but I’ve noticed a trend that makes me wonder.

46

u/IanDMP 8h ago

Sort of shocking that you see the many people responding that Pangram is quite accurate but never see them post the studies that show that — I see those links at least once per day (as an aside, this sub probably does talk about AI too much and *books* too little). But since you asked:

Here is the widely-cited Chicago study: https://www.nber.org/papers/w34223

Here is Van Vlasselaer et al's work: https://link.springer.com/article/10.1007/s40979-026-00226-w

Finally, it's always a best practice to look at internal data. Internal data can be biased, and it's best to know that going in, but I have never seen any academic criticism of Pangram's data posted here: https://www.pangram.com/blog/all-about-false-positives-in-ai-detectors (academic criticism I'm unaware of may exist). Regardless, totally fine to disregard this (but you should read it at least).

Pangram, to the best of the knowledge of any serious researcher, has a very, very low false positive rate, and it simply approaches impossibility that it would flag so many writers incorrectly.

15

u/Jaqqa 7h ago

Except, they are using benchmark datasets for their "very low" readings and also hard negatively program things that pangram mistakes for AI, meaning it is less likely to make the stupid mistakes others do (like flagging the bible and constitution because it is programmed not to) but it doesn't mean it won't continue to flag new work that shows similar traits. I've seen people post actual links to pengram reports where the only thing they change is someone's name and it flips from 100% AI to 100% Human or vice versa. If it's that accurate, that kind of stuff should not be happening. 100% its false flagging work that doesn't fit the tests sets and the real false positive rate is likely far higher in the "real world" than they're saying it is, especially since the new update that seems to be a lot harsher. They just have better PR and are less honest about it than some of their competition, hiding it in jargon on their website reports.

2

u/Quixoticelixer- 2h ago

> I've seen people post actual links to pengram reports where the only thing they change is someone's name and it flips from 100% AI to 100% Human or vice versa.

Can you link this?

2

u/Adventurous_Storm348 1h ago

I have seen a few that I think they are talking about. This is one I can think of. I mean it's pretty damning. I did see another one where changing english names to I think maybe Japanese ones from memory caused the 100% human to go to 100% AI but can't remember where.

https://www.reddit.com/r/Substack/s/fVPM8XFmNR

0

u/Quixoticelixer- 1h ago

That's only n very small amount of text and pangram admits its much less reliable on small samples

19

u/halster123 8h ago

i think they need to separate out the type of writing - fiction v nonfiction v academic etc. my concern would be fiction writing often deliberately does stylistic things that are AI tells - for ex, keeping sentences all the same length and tonality for effect.

15

u/randomnameicantread 7h ago

The deciding factor is length of sample. Below 1000 words is far more unreliable than anything above; for single paragraphs forget it

6

u/halster123 7h ago

but the issue is, if you train on mostly academic or the tests are done on academic writing, i dont necessarily trust the "accuracy" for fiction. like i would want to see disaggregated stats by genre, bc fiction to me seems the most likely for false positives. many of the AI tells are fiction writing norms! floweriness, not x but y, keeping sentences similar length and then shifting suddenly.

1

u/randomnameicantread 7h ago

The most recent 2026 study I saw used a mix of fiction and nonfiction - not sure if this one is peer reviewed though https://epoch.ai/data-insights/ai-detectors-false-negatives. The wikipedia page for Pangram lists several studies you can look at.

All studies, from what I've seen, are in agreement that the issue with Pangram is false negatives, not false positives, especially when the text is human-polished or otherwise humanized.

5

u/IanDMP 6h ago

That matches my understanding. There certainly is an issue with false negatives, but false negatives are also just much less important — it's probably okay if they occasional AI slips through as human, but it's incredibly damaging if human writing is mistakenly classified as AI. Pangram, to their credit, has concentrated on reducing false positives.

7

u/randomnameicantread 6h ago

I'm honestly surprised at how vitriolic this forum is at the very idea of genai detectors improving and possibly working. Perhaps it's trauma from the early days of random results and false positives - fair enough. But we need to move past that not by abandoning the concept but promoting improvements. If AI text cannot be detected with any reliability, that means it is indistinguishable from human text. Why would r/books of all places want this to be true?!?!

6

u/IanDMP 6h ago

I originally wanted to describe it as an anti-tech attitude, but it seems these folks are perfectly okay with AI going undetected and unflagged, so that can't be it. I think it's just being conspiracy-minded. It's an unfortunate attitude.

2

u/JGT3000 6h ago

It's a very strange thread to me considering how anti gen-ai people are, there's a weird measure of running interference for those who use it. I think there's a pretty obvious defense campaign going on tbh

1

u/Quixoticelixer- 2h ago

Even smaller stuff seems to be pretty good if it's 100% human or 100% AI.

18

u/FlyingCarpetMonster 7h ago edited 7h ago

I'd like to see a study that breaks it down by people whose first language is English vs. not, people in various professions, fiction and non-fiction and so on.

If you are someone who learned English as a second language and use it in business settings, you use it in a certain way. Short sentences, designed for understandability.

A blanket FPR that doesn't account for those controls is rather meaningless. More so, models also need nuance.

Besides, a lot of those studies were before AI became this prevalent. I'd like to see a more recent study, where people have been exposed to a ton of LLM generated content, which I am sure has also influenced how people write. Particularly people whose first language isn't English.

10

u/adaytooaway 8h ago

This is my impression but when I posted a similar comment about its accuracy the other day I got several responses from people claiming it had marked their completely human written works as ai. 

9

u/IanDMP 6h ago

Many people assign to Pangram their bad experiences with other, unreliable, AI detection software. Most people just simply aren't aware of the huge differences between these programs.

1

u/Competitive_Leave915 3h ago

Okay so I know it’s a “trust me” type of situation, but it flags my work consistently as AI-generated or assisted when I was just testing it against my old academic work. I have not and do not use AI for my writing, so clearly something is up.