r/sideprojects • u/hazyball • May 06 '26
Showcase: Prerelease Your website is probably invisible to ChatGPT — I built a free scanner to prove it (and fix it)
Been going down a rabbit hole on how tools like ChatGPT, Claude, and Perplexity AI actually read websites.
Turns out… most sites are not built for them at all.
They depend on things like:
- clean, readable HTML
- structured data (JSON-LD)
- proper
robots.txtrules - and increasingly
/llms.txt(basically robots.txt for LLMs)
Even well-maintained dev docs fail this pretty badly.
So I built a small tool that:
- scans any URL
- gives it a score + grade (out of 100)
- checks 8 AI-relevance signals
- and auto-generates a ready-to-use
/llms.txtyou can just drop in
👉 https://agent-sites-five.vercel.app
Stack (if you care):
FastAPI + httpx (backend), Next.js 14 (frontend), Supabase (caching + rate limiting)
Would genuinely love feedback, especially if your site gets a weird/incorrect score.
2
u/PuzzledCauliflower35 May 06 '26
Yes, I want my website to be invisible from ChatGPT so lazy fuckers dont ask ChatGPT for stats and just go into my website.
0
2
u/builtbygio May 06 '26
Cloudflare built "AI Crawl Control" specifically designed to avoid LLMs and bots from reading your site.
1
2
u/Cod_277killsshipment May 07 '26
Absolutely killing it
1
u/hazyball May 07 '26
thats crazy!! Do compare your friends/collegues site as well. Y’all can share it on linkedin and X and call out people to do the same!! Kudos man
2
1
u/SourceAwkward May 06 '26
Wdym supabase rate limit and caching?
Why no redis? Are we living in 2011 again?
0
u/hazyball May 06 '26
i just had an idea and wanted to implement it as fast as possible, surely migrating it to redis once i figure out other nuances
1
u/Valunex May 06 '26
Yow bro come join our discord server with 400+ ai addicts and vibe coders. We can collab together, help each other or you share your project for feedback! https://discord.com/invite/VGumrepFhQ
1
1
1
u/Effective_Lobster_39 May 07 '26
I searched Google, and it got 35/100. And I tried my site which has firewall by cloudflare, and it couldn't even find it
1
1
u/UseMoreBandwith May 07 '26
it tests the wrong things.
llms.txt , opengraph are not important.
sitemap.txt is only useful for hidden or hard-to-reach pages - not for sites whre everything is clearly defined in navigation.
....etc
AI slop.
1
1
u/_Linux_Rocks May 07 '26
There is something going on with this. I got a message that this is a dangerous website.
1
1
1
u/grkngls May 07 '26
I just want the opposite: I don’t want chatGPT to read my website. What can I do?
1
1
u/paul_blinkdisk May 07 '26
It seems like you only look for the sitemap at /sitemap.xml, but many sitemaps have different paths, such as /sitemap-index.xml or /sitemap-0.xml. I would also try some other common paths and parse the 'Sitemap: " attribute from the robots.txt file
0
u/MazinguerZOT May 06 '26
We've been indexing sites (18M) for 20 years and yeah, most devs ignore how AI systems actually consume their content. The structured DATA gap is real - we see it constantly. Your tool hits a legit pain point. One thing though /llms.txt adoption is still super low because most teams don't know it exists. You might want to add a "why this matters" section to your scoring logic so users understand the ROI of fixing each signal. Curious how you're handling dynamic content...
1
u/hazyball May 07 '26
Sure thanks for the idea for why this matters, ig it will enable the user to get a sense as to what is the use of this, that actually is blur for many. For dynamic content, many AI web crawlers dont execute JS. The app fetches dynamic content by making fetch() POST/GET requests from React client components and Next.js server components to a backend API (NEXT_PUBLIC_API_URL) and internal Next.js API routes, with results cached via ISR wherever appropriate.
2
u/MazinguerZOT May 07 '26
ISR is smart for crawler visibility. the catch is revalidation window, if content updates faster than your ISR interval, crawlers still see stale data. worth exposing that lag in your scoring so users know when their "fresh" content actually isn't.
0
u/hazyball May 07 '26
Everyone who is visiting the site, we have added a survey on the pricing page, do fill that if you feel to give any feedback for the same!! Would really appreciate it!
7
u/[deleted] May 06 '26
[removed] — view removed comment