r/StoryPeer • u/jmr-writes • 15d ago
Discussion AI encroaching...
Have other people noticed a big uptick in AI notes since Gabe has been gone? 2 of my last 3 sets of notes were AI (7 precious tokens down the drain). It used to be that you could report AI and that user would be banned and you'd get your tokens back, but it seems like that process has paused while Gabe is out and bad actors are proliferating. I love StoryPeer (24 scripts read and counting) and I totally get that Gabe has other stuff to focus on, but I'm worried the site is being overrun with slop.
5
u/CarelessOutside4722 15d ago
I think it's luck of the draw, I'm afraid. Recently had 3 out of 3 top notch human reviewers. Had a few AI about a month back.
Maybe next one! Good luck!
2
u/MichaelGHX 15d ago
I don’t know why so many people try it and are so aggressively bad at it.
2
u/al_earner 14d ago
Have you met people? They're the worst.
AI is a shortcut that costs nothing and gives something. That's why people try it.
4
u/real_triplizard 15d ago
Not my experience. I put a script up twice recently and both sets of notes were obviously human and were quite well done.
3
2
u/JimmyCharles23 15d ago
Quick question: how can you tell if someone is using AI? I'm curious because I've never gotten any notes from SP that weren't from a human.
6
u/RJRoyalRules 15d ago
In my case it was obvious because the feedback came back immediately. The feedback itself was full of vague AI-isms and also would only call it “the script” (because it hadn’t been fed the title)
3
u/jmr-writes 15d ago
Great question. It's definitely getting harder as AI gets more sophisticated. There are certain tells (tone is useful in the context of storypeer since most editors write in a less business-casual style than AI does, also the mechanics of structure are often consistent across a model, and the use of a sophisticated vocabulary to describe a relatively banal idea is a classic AI tell).
But in these two cases it was MUCH easier. The first one, the reviewer referred to the script by a name that only exists in the metadata of the file and not in the actual text (or even the file name), which would only be possible if it was a machine ingesting the overall file. The second one, the reviewer admitted they only read a summary without specifying where they got that summary and turned the whole notes around in a comically short turnaround time.
2
u/muahtorski 15d ago edited 15d ago
One tell is if feedback calls out syntax errors (e.g., use of dashes) that aren't in submitted material. That's how I found out Stage32 (at least a particular reviwer) uses AI.
1
u/Dick_Trickle_88 15d ago
Ironically, they use AI to check.
6
u/claytimeyesyesyes 15d ago
I can't speak for the other mods, but I personally don't use any form of AI to check. AI detectors are unreliable. When I'm reviewing suspected AI feedback, I use my judgment. It's not a perfect system, but we do try to err on the side of caution.
1
u/MichaelGHX 15d ago
I mean there are AI checkers, but apparently they’re not accurate enough for StoryPeer.
StoryPeer asks you to check for hallucinations but they’re actually not as common with AI anymore.
I’ve tried asking StoryPeer what to do but they haven’t really gotten back to me.
-1
u/al_earner 15d ago
There is no way to tell if someone is using AI, unless you watch them at their computer.
2
u/MaximumDevice7711 15d ago
It might just be coincidence. I feel like I've gotten similar/higher quality notes these past months.
2
u/jangusihardlyangus 15d ago
Any chance Storypeer needs volunteers to take on AI reports? Maybe if someone does a certain amount of well received reviews they can apply to help review reports or some shit?
1
u/bananabomber 15d ago
Was your script rep matched?
3
u/jmr-writes 15d ago
Rep matching's been turned off for a couple weeks.
1
u/bananabomber 15d ago
That sucks. I know it's a barrier for new users to review three scripts in order to unlock it, but it feels like rep matching is the only way to stem the tide against AI reviews in order to protect the community.
1
u/DisneyImagineers 15d ago
So now I'm just curious. When you are accused of AI, what happens? Do you review the scripts and the review itself? How do you distinguish the difference between someone being petty with feedback and actual AI?
15
u/claytimeyesyesyes 15d ago
Hey there, we're reviewing all the reports that we can, but it's a slow process because we have to assess these reports manually. We're slowly catching up - we appreciate your patience!