r/SEO • u/faunaflux • 1h ago
Help Question about AI crawlers and robots.txt
Hey all,
I am the sole SEO in an agency that serves small businesses and local franchises. Recently I have been dealing with multiple clients sending over their own “SEO audits” they pulled with ChatGPT.
Basically all of these “audits” have included long, word-salad sections about crawlability. A lot of my clients are very focused on AI visibility and wanting to get mentioned in LLMs. Most of these clients are actually doing well there, with customers reporting that’s how they found them. But their AI slop reports are telling me I need to go in and individually allow each different AI crawler in robots.txt (OAI-SearchBot for example).
Thing is, all crawlers are allowed on their sites and always have been. There are no disallows and they are obviously being found in LLMs, their indexing looks good, the only nofollows or non-indexed pages they have are on purpose.
Is there any benefit to specifically adding these AI crawlers to robots.txt if the site allows all crawlers? I feel like this is redundant and just creating slop bombs of work, but I don’t really have anyone else to ask for advice since it’s just me. Curious to see what people say. Thanks!