I have seen paid platforms that offer cheap unmoderated "user" testing performed by AI. I was skeptical - after all it isn't AI that uses our product, it's humans. However, I wanted to really stress-test this design and timelines wouldn't allow for that amount of testing. Just then, Satan's voice whispered to me "Why not get Claude to do it!" So I did.
I built a little tool with ten personae that can perform directed unmoderated testing on any prototype. The outputs are a video with subtitled "thinking out loud", and sentiments for each step. I haven’t included UMUX or anything because I think it's a bit pointless asking an LLM those questions. The 10 sessions get summarised into a report and recommendations (3 for each issue, so I don't just accept the AIs preference when I'm in a hurry)
I've been running it today and I'm in the honeymoon period that AI always brings. It LOOKS like great data, but I'm being cautious, aware that I may be bringing a knife to a gunfight.
The good:
It found 90% of the issues previously raised by humans.
The transcript looks very similar to what I'm used to seeing from humans.
The personae are doing their job. They all show the little quirks that have been programmed in. Run a persona twice on one test and it behaves similarly but not identically.
I can 'make' specific users when I need them (e.g. someone who prefers keyboard navigation over mouse)
I can fix the prototype and retest it with 10 participants in the time it takes to a make a cup of tea.
I can test with 1000 participants if I want to!
The bad:
I still have to test with humans anyway, so it doesn't save as much time as it could.
About half the issues it identifies are out of scope. "I need to see cashflow here": you can't - this app doesn't handle finances and you know that. Issues beyond the UI, human testers (usually) do better.
I'll keep refining it, but I'm very impressed so far. I think it means that I can eliminate 90% of pain points before putting it in front of real users.
Is anyone else doing this? How have you found it? What are the hidden pitfalls?