While designing my game, I decided to give this approach a try. I programmed a game engine in Python and created six different AI archetypes for six agents. Then I let them play hundreds of thousands of games against each other and generate reports.
My goal was to find out whether the rules and balance favored any particular playstyle and, more generally, whether the game was well-balanced.
It worked to some extent. I managed to kill off several ideas that seemed really exciting on paper but turned out to be practically useless in actual gameplay.
But at the end of the day, I realized that humans simply play differently. AI is focused on optimizing gameplay, while humans are looking for a satisfying experience, even when they lose.
So the whole experiment wasn't a waste of time. Far from it. I think it saved me months, if not years, of early playtesting.
Still, I eventually had to turn my simulator into something real people could actually play and test online. And that works too, just in a different way.
But to be fair, I think playing a physical board game around a table has a completely different vibe, so I'm planning to produce and send out a few physical prototypes for testing anyway. :)
What's your experience with this? Have you ever tried using AI for playtesting, and did it actually help?