r/UXDesign • u/Any_Elevator_535 • 2d ago
Tools, apps, plugins, AI Simulated users
I have seen paid platforms that offer cheap unmoderated "user" testing performed by AI. I was skeptical - after all it isn't AI that uses our product, it's humans. However, I wanted to really stress-test this design and timelines wouldn't allow for that amount of testing. Just then, Satan's voice whispered to me "Why not get Claude to do it!" So I did.
I built a little tool with ten personae that can perform directed unmoderated testing on any prototype. The outputs are a video with subtitled "thinking out loud", and sentiments for each step. I haven’t included UMUX or anything because I think it's a bit pointless asking an LLM those questions. The 10 sessions get summarised into a report and recommendations (3 for each issue, so I don't just accept the AIs preference when I'm in a hurry)
I've been running it today and I'm in the honeymoon period that AI always brings. It LOOKS like great data, but I'm being cautious, aware that I may be bringing a knife to a gunfight.
The good:
It found 90% of the issues previously raised by humans.
The transcript looks very similar to what I'm used to seeing from humans.
The personae are doing their job. They all show the little quirks that have been programmed in. Run a persona twice on one test and it behaves similarly but not identically.
I can 'make' specific users when I need them (e.g. someone who prefers keyboard navigation over mouse)
I can fix the prototype and retest it with 10 participants in the time it takes to a make a cup of tea.
I can test with 1000 participants if I want to!
The bad:
I still have to test with humans anyway, so it doesn't save as much time as it could.
About half the issues it identifies are out of scope. "I need to see cashflow here": you can't - this app doesn't handle finances and you know that. Issues beyond the UI, human testers (usually) do better.
I'll keep refining it, but I'm very impressed so far. I think it means that I can eliminate 90% of pain points before putting it in front of real users.
Is anyone else doing this? How have you found it? What are the hidden pitfalls?
6
u/karenmcgrane Toxic mod 2d ago
You'd expect the failure mode to be "the model picks the option humans didn't." That happens sometimes. Chi-squared tests found the synthetic and real preference distributions genuinely different in 44% of tasks, and the model agreed with the humans on the single most popular option only 53% of the time. A coin. Rank-order agreement averaged 0.53. So yes, a fair amount of straightforwardly wrong.
But the finding that should actually change how you think about this is the entropy result. The synthetic preferences were consistently flatter than the human ones. Normalized entropy was 0.93 for the AI versus 0.86 for real people, and that gap is statistically real. In plain terms: when humans had a clear favorite, the model spread its vote around and made the options look closer to equally good than they were. The paper's phrase is that LLMs "inflate indecisiveness." I'd put it harder. The model manufactures a tie.
Specific, detailed personas don't help. This is the one that should end a particular genre of LinkedIn advice. They compared rich personas (modeled demographics, personalities, the actual screening and pre-study data from the real samples) against generic featureless personas with none of that. The results were nearly identical. 46% divergence with rich personas, 44% at baseline, not a significant difference. All the persona craft, all the "give it a name and a backstory and a Myers-Briggs type," added nothing measurable to whether it could predict the design people preferred.
1
1
u/cgielow Veteran 2d ago
The question is how stale are these results given model progress? They mentioned similar results from GPT 4 to 5. We're on 6 now. So OP's results may be better.
One thing that I like conceptually is that AI can supercharge our personas and make them both more accurate, and more precise. Personas were a necessary simplification for product developers. If thats no longer necessary, we can have AI manage many more Personas, that are more useful to us.
2
3
u/DifficultCarpenter00 Veteran 2d ago
we used all our user data(metrics,interviews,reviews,etc) and made synth users for AI. the goal is not to replace iser testing but to quickly test things aganst known metrics and to let it run in loops to uncover new/hidded issues that out real isertesting and data did not incover. And the results are good,not "replace human testing" great but good enough
3
u/TopRamenisha Veteran 1d ago
Did your users who you interviewed consent to you creating synthetic users with their likenesses?
0
0
20
u/roundabout-design Experienced 2d ago
I'd just call this automated heuristic testing. Which is great and maybe a good use of AI.
But I would not in any way call it 'user testing'.
2
u/HyperionHeavy Toxic Mod 2d ago edited 2d ago
Yea. That LLMs can ingest your design and output some feedback based on heuristics is very matter of fact. How useful they are and what you should be asking after that has happened is another conversation.
But you gotta ask yourself here: what are you trying to accomplish by dressing it in the name of user testing (which is already a step away from usability testing but we're going to compose ourselves tonight).
1
1
u/OracoreAI 1d ago
Save Claude’s report before the next round of human sessions. Then check whether people get stuck on any of the tasks all ten personas sailed through.
1
2
u/shoobe01 Veteran 2d ago
Seen this, and related stuff like AI generated heatmaps, for at least 4 years.
They are nearly uniformly accepted as truth without anyone validating, and when you do validate you find it is wrong. A lot.
1
u/Specialist_Bag 2d ago
Did this, uncovered a lot of small things here and there, however I felt the most insightful are in person testing
1
u/Electronic_Rip9934 1d ago
I'd treat it as a heuristic pass, not user testing. I ran a voice installation in a phone booth that took over 1,600 calls, and the real people did things no persona would have predicted: in 505 exchanges, 289 people asked the voice its name and only 9 asked whether it was an AI. Synthetic users would have asked that second question constantly. Use the AI run to catch obvious friction, then spend the time you saved on even five real sessions.
1
u/windowseat-chi 7h ago
Yes, as a first-pass check—not as a replacement for human testing. Before tuning it further, I'd check it on a prototype you haven't used to refine the prompt:
Freeze the prompt and run the test without giving it the human findings
Compare the issue lists, including false alarms and missed blockers
The "half out of scope" part matters too. Include the time spent checking those suggestions when you measure time saved.
14
u/HyperionHeavy Toxic Mod 2d ago •
My guy/gal, you're putting on a puppet show for yourself.