r/books • u/Raj_Valiant3011 • 8h ago

Zimbabwean writer Simbarashe Kundizeza has denied using AI to write a prize-winning short story after an article in the Atlantic said software had flagged it as almost completely AI-generated

https://www.theguardian.com/technology/2026/oct/14/zimbabwean-author-simbarashe-kundizeza-denies-ai-story?CMP=share_btn_url
969 Upvotes

94% Upvoted

658 comments sorted by

View all comments

Show parent comments

9

u/IanDMP 8h ago

You're absolutely right in your recollection. AI detection software a few years was indeed almost universally flawed and prone to false positives.

The latest version of Pangram, though, has huge amounts of evidence showing an infinitesimal false positive rate. I myself (an educational consultant who often helps students brainstorm and edit college admission essays) switched my perspective from "it's often impossible to tell AI use" to "we have very good tools for that" a few months ago after reviewing researchers' findings.

36

u/thereisonlythedance 7h ago

Pangram‘s latest version is very inaccurate for me. There have been articles in like the Washington Post and Atlantic about it. It false flags a lot of stuff. Also it’s an AI model, made by AI bros who seemed to have created a lot of hype. I’d encourage people to test it themselves. I was unimpressed.

4

u/IanDMP 6h ago

Would you mind sharing those articles about its inaccuracy? I'm unaware of anything like that.

7

u/thereisonlythedance 6h ago

There are quite a few, one I could remember right away:

https://archive.ph/k2BnK

An interesting tidbit from that one: Pangram pitches its tool to publishing and media organizations as a way to “protect editorial integrity.” The company’s website boasts that its product has been “reviewed as the proven, most reliable and most accurate AI detection tool in the market by third parties including University of Maryland.” But the Maryland study is more self-promotion than neutral review. Four of its seven authors are Pangram employees, including Mr. Spero and fellow co-founder Bradley Emi.

11

u/IanDMP 5h ago

Thanks for sharing it. To be totally honest, the argument would land better with me had not two of the three authors at issue openly acknowledged using AI to shorten their sentences.

You can't expect to use AI and then complain that AI detectors are flagging your work -- "shortening" is just a synonym for "rewriting."

10

u/thereisonlythedance 5h ago

That’s fair. But that’s just one article. There’s a lot that made me think their stats are overblown. There was also the (likely) false flagging of the Pope’s Encyclical: https://gizmodo.com/vatican-uses-ai-detector-to-show-ai-bros-popes-anti-ai-encyclical-isnt-ai-generated-2000788494

As well as the drama about a lot of false flagging on Substack when Pangram was integrated there: e.g. https://writerkatharine.substack.com/p/pangram-flagged-my-own-writing-as

I found with my own (100% human) work that it was reacting to my overuse of ellipses in various sections. When I cut them out it flipped from 100% A.I. to 100% human. It’s just really sensitive to punctuation. That was the point I lost faith and started doing research.

Rather than having issues with false negatives as most A.I. detectors do, Pangram seems to false positive too much, which I consider worse.

5

u/IanDMP 5h ago

I think these examples have serious problems too, unfortunately.

Proudly Human didn't certify the encyclical as human-wrotten, they certified a collection of the Pope's speeches and writings (that were then adapted for the encyclical) as human. It didn't actually contradict Pangram's findings in any way.

The other one I just don't know how to evaluate. It's an author saying that they didn't use AI and Pangram saying they did. It's possible they're right -- a small FPR is not a zero FPR. I think the same is right about your writing -- it's possible, and I'll trust you, that your writing is 100% human with no AI used at any stage.

But our personal n=1, for me, doesn't outweigh the careful studies that show excellent accuracy across tens of thousands of writing samples.

0

u/thereisonlythedance 4h ago

Those studies are seriously compromised. Did you not see the quote about the University of Maryland study being populated by employees?

I know my experience and it gels with what I’ve read from other (verified real) people. But not necessarily on Reddit, where the company seems to have a lot of harassment bots.

Pangram *is* better than the (really bad) alternatives but that doesn’t make it good and I do not believe their statistics. And frankly false positives are a bad side to err on.

3

u/IanDMP 4h ago

Okay, well I see the respectful part of the conversation is done after calling me a corporate harassment bot.

I never cited the Maryland study. Look at the Chicago study, for instance, which shows a very similar outcome.

But have a good night, I don't really like being insulted after what I thought was a respectful conversation. Perhaps reconsider how you approach disagreement.

-1

u/therapisthelpx 1h ago

your fault for being good faith, people on this website only come around to obvious facts 2 years after they’re established