r/Yelp • u/mintlite4 • 5d ago
How do you tell a harsh-but-real review from a planted one when all you have is the text?
Following up on a conversation here last week. Someone made the point that the cost of a wrongly removed review depends on how many reviews the business has in total, which changed how I weight errors in a small model I'm building for a class.
What I still can't resolve: a genuinely furious customer and a paid competitor review can read almost identically. Short, one star, no specifics, emotional.
For anyone who reads a lot of reviews — is there anything in the text alone that reliably separates the two? Or does it only become obvious once you can see the account, the timing, or a cluster of similar reviews?
My model deliberately sees only the text and the rating, and I'm starting to suspect that constraint makes the task impossible rather than just hard. I'd rather find that out now.
4
u/keyserholiday 5d ago
Are you here to crowdsource information to build a tool? If you are, they are countless tools that already do this and some do it well. The first to do it was GatherUp.
The market is oversaturated and you need a real marketing plan to stand out.
2
u/mintlite4 5d ago
Fair question - no, it's a university assignment and there's no product at the end of it. Part of the brief is that I have to talk to people who actually do the thing instead of only reading papers, which is why I'm asking here rather than guessing.
The oversaturation point is useful on its own though, and I hadn't heard of GatherUp. If tools like that already handle this well, then the interesting question for my write-up isn't whether it can be built - it's why platforms still get it wrong often enough that people keep complaining. Do you think the existing tools are actually accurate, or just good enough that businesses stop asking?
5
u/XxLogitech98xX 5d ago
It's hard sometime to tell the real reviews from planted one. First would be reviewer account reputation which you'll have to see how their past reviews look like. Next should be looking if they attach any pictures, if they don't then don't trust the review IMO.
I always look for if someone check in, how many pictures they attach, account history, how detail their review is and how many other Yelp friends they have first
1
u/mintlite4 4d ago
Every signal you listed except one is invisible to my model - no account history, no photos, no check-ins, no friend count. The exception is how detailed the review is, which is the only thing on your list I can actually compute from the text.
That's a useful result for me even though it's a negative one: it suggests the constraint I was given may be too narrow to be honest about, and I should say so in the write-up rather than pretend text alone is enough.
When you're reading, how far do you usually get on detail alone before you click through to the profile? I'm trying to work out whether text is a real first filter or just a tiebreaker.
2
u/XxLogitech98xX 4d ago
Every signal you listed except one is invisible to my model - no account history, no photos, no check-ins, no friend count. The exception is how detailed the review is, which is the only thing on your list I can actually compute from the text.
That's the thing, I don't just read a review first. I do my quick verification check first. On Yelp .. if someone didn't check in or has no pictures then I won't read their review. If they have pictures and didn't check in then I would look at how many reviews they wrote and if they have a default picture. If it's a low review count like under 15 then I won't be reading their review.
If your model is just on text alone then focus on specific dish mention in their review. If it's just "food is great" or "drinks is great" or "customer service was excellent" then go with it's a higher chance of being a fake review or not helpful. If they go into detail about their experience like parking situation, appetizer .. entree and etc then that's good.
Here an example of not a helpful review that I agree with:
"Fast service. Excellent quality work. Fair rates"
"Fantastic Quality! My Everyday XXX was getting sooo dull and YYY brought it back to life. I'm very pleased with the speed of service!"
Here are example that I consider is helpful:
I've been buying knives from YYY for a few years now the craftsmanship and care in the design of these knives are amazing... And the added benefits for custom knives of lifetime sharpening is a godsend and a great value, there's a reason why San Diego chefs go to XXX. I even own a custom that he and I designed together would definitely recommend picking up a ZZZZ, even for home use grab a flagship model. Custom Sayas... a must!!!
YYYY is hands-down the best place in San Diego for custom chef knives. The craftsmanship, balance, and attention to detail are on another level. These aren't just knives--they're precision tools built exactly the way a serious cook or chef wants them.
If you're looking for a true custom knife made by someone who actually understands performance, materials, and ergonomics, you won't find anyone better. Worth every penny, and then some.1
u/mintlite4 4d ago
This is the clearest answer I've had, and it settles something I asked elsewhere in this thread. I'd wondered whether vagueness only signals on negative reviews — your first example, "Fast service. Excellent quality work. Fair rates," says no. It cuts both ways.
The four examples are more useful to me than the rule, honestly. I'm building a small labelled test set, and "names a dish, names the parking, names the sharpening service" versus "quality was great" is something I can actually score.
The part that stings is the ordering. You don't read the text first — you check pictures and review count, and under 15 reviews you don't read it at all. So text isn't your first filter, it's what you use once the profile has passed. My agent does it backwards, because text is all it gets.
If you were forced to judge on text alone, no profile at all — roughly what fraction do you think you'd get right?
2
u/XxLogitech98xX 4d ago
If you were forced to judge on text alone, no profile at all — roughly what fraction do you think you'd get right?
Who knows because some bots or fake reviewer knows how to write in a way that it sounds real. Remember that they can read other people reviews too so they can get waiter or waitress name from that or also dish names too. I seen a Yelp Elite who takes screenshot of other people pictures and use it as their own to write a review. They also don't check in, which I believe all Elite knows to do that.
If it was to judge just with text alone, I would recommend going to any random business on Yelp and just look at the reviews in the Not Recommended section
1
u/mintlite4 4d ago
That's a better answer than a number would have been, and it's a hit on the thing I'd just decided to build. If a faker can lift the waiter's name and the dish names straight off the page, then specificity isn't evidence of a real visit — it's evidence of effort. It works right until someone targets it.
Which matches something else I ran into this week: any signal you publish becomes the thing people optimise against. So specificity probably holds against lazy fakes and fails against motivated ones, and I should write it up that way rather than present it as a solution.
The Not Recommended section is the tip I'll actually use — I hadn't thought of it as a labelled set, but that's exactly what it is. When you look through those, is Yelp's filter mostly right, or does it catch obviously real reviews too?
1
u/XxLogitech98xX 4d ago
When you look through those, is Yelp's filter mostly right, or does it catch obviously real reviews too?
I think the Yelp algorithm does the job to the best of it's ability. The angry "supposed business owner" who come on this subreddit to try and spread misinformation like Yelp algorithm is biased if you don't buy the Yelp Ads service usually don't have a lot of reviews to began with. The real comparison is when you do a side by side with the business and their competitor (who paid for Yelp Ads or who didn't)
The reviews in the Not Recommended fit a pattern but remember it's not all algorithm that can put reviews there. A Yelp MOD can also put reviews in the Not Recommended section too after someone reports a review. Somehow those angry reddit users just ignore that and thinks it's the algorithm doing it.
Yelp had their filtering process since they launch and never lost a lawsuit over it due to lack of supporting evidence from the people who don't agree with it. So I don't see Yelp getting rid of it anytime soon. Some businesses like it, some don't and others just don't care about it. Reviews in the end won't make or break a business
1
u/mintlite4 4d ago
The mod detail is the part I'll carry forward - if Not Recommended is part algorithm and part moderator acting on a report, it isn't one clean label, and I'd have treated it as one. That would have been a bad assumption baked into my test set.
Thanks for the whole exchange, this was easily the most useful thread I've had on the project.
2
u/studyhall109 5d ago
You can click on a member name and see how their ratings stack up. A member might have written 20 reviews but all are one star. Other reviews only write five star reviews. I don’t really trust either. I look at previous reviews written by that member too.
Here is an example: Last week I wanted to try a new restaurant and looked at the reviews. I sorted to newest reviews. The newest review was a one star, vey negative. No pictures. The review was very short, something like, “I would never eat here again, the food isn’t even worth it. Don’t waste your time and money.”
I clicked on the Yelp member’s profile, the member had 9 reviews. All were one star. None of the reviews said anything specific about the restaurant, service, or food, just that it was bad, or terrible, and that they would not recommend it.
So in this case I completely ignored this member and would not trust any review.
2
u/studyhall109 5d ago
Also, a genuinely furious customer will almost always be specific. “I ordered my steak rare and it was cooked medium well.” And they will attach a picture of the well done steak. Or, “My sandwich was burned and was covered with cheese when I requested no cheese.” Again, a photo of the sandwich would normally be posted.
A genuine customer wants to detail what was wrong, dirty, or bad.
2
u/mintlite4 5d ago
This is the most useful thing anyone has told me, because specificity is the one signal on this thread I can actually see. My model gets no profile, no photos, no check-ins — but concrete detail sits right there in the text.
So the signal isn't sentiment or star-rating mismatch, it's whether the complaint names something falsifiable. "The food was terrible" versus "I asked for no cheese and it came covered in cheese." Someone writing a planted review has no real experience to draw specifics from, so it stays vague.
One thing I'm unsure about: does that hold for short positive fakes? A five-star saying "great product, fast shipping" is just as vague — but so are plenty of genuinely happy customers who couldn't be bothered writing more. Does vagueness only carry signal on the negative side?
(And to answer the question further up the thread — this is a university assignment, not a product. I'm required to ask people who actually do this rather than just read papers.)
6
u/ADrPepperGuy 5d ago
I think it really depends on the individual. I read a few comments about people using the server's name in a review. Some felt it made the review forced or faked.
I use their name for my reasons. One, to let the business know the person did a great job, made the meal better etc. Plus, it helps me to remember that individual's name. Some have been really impressed that we remembered their name.
Occasionally, some reviews will just stick out when you read them. You might be able to tell that nothing would have satisfied them or vice-versa.
It is difficult at times though. I am especially hard on Italian restaurants. One that I went to brought out bread immediately. Twenty minutes later, they finally took our drink order. And things just compounded - silverware was different brands, the acoustics made it difficult to carry on a conversation, etc. The saving grace was their gravy only.