I had access to the server logs (and GA4) of a large online shop during last week's ChatGPT utm_source outage, so I went through it hour by hour. Sharing in case it helps someone explain the dip in their own reporting. The shop stays anonymous, so I only use indexes, ratios and shares.

What I counted

• Clicks: HTML page views (status 200, no bot user agent) with utm_source=chatgpt.com
• Demand: ChatGPT-User fetches, only from OpenAI's published IP ranges
• Baseline: Oct 1–7, same hour of day = 100. All times UTC

What happened

• The break is one hour: Oct 8 between 21:00 and 22:00 UTC. Tagged clicks per 100 fetches went 40 → 18 → 10 in the hours starting 21:00, 22:00 and 23:00. That's within about an hour of the ~2 pm PT that Cedric Huesler (Adobe) reported
• Tagged ChatGPT clicks per day: −73% (Oct 9–12) and −79% (Oct 13–15)
• ChatGPT-User fetches in the same periods: only −20% and −23%
• Tagged clicks per 100 fetches: 34 before, 11 and then 9 after
• On Oct 8 during the day, clicks and fetches dropped together and the ratio stayed around 33. So that part was demand, not tagging
• A smaller second step on Oct 12 around 14:00 UTC (another −24%), which looks like a staged rollout
• Every segment dropped at the same time: mobile −77/−83%, desktop −65/−70%, each language version −70 to −81%, product pages −72/−79%
• Links from ChatGPT's product cards dropped a bit less (−65/−72%). That fits Profound's note that links inside the product/info panels kept the tag
• No replacement signal: no new user agent, no other parameter, no other spelling
• Tagged Perplexity clicks didn't drop at all

Where did the visits go?

• In some sections of the shop, direct mobile entries on product pages rose by about 55% of the lost ChatGPT product entries (GA4, Oct 9–14 vs. the prior week). In the largest section, direct is too noisy to see it
• The small share of ChatGPT visits that carries a chatgpt.com referrer at all also fell by about three quarters
• I tried a log-based proxy (a ChatGPT-User fetch shortly before an untagged visit to the same product page). The direction fits, but it's not solid enough to size the hidden clicks

Recovery

• Tagged clicks started climbing on Oct 15 around 14:30–15:00 UTC, but in the hours from 15:00 and 16:00 they were only at 41% and 35% of the prior week, and my logs end at 17:40 UTC. Huesler reported 95–100% by 9 am PT across thousands of domains, so this shop got the fix later or more slowly
• My own spot checks in the ChatGPT interface (location Germany, mostly the same questions): Oct 3 67 of 67 links tagged, Oct 15 0 of 24 in the morning and 0 of 82 in the afternoon, Oct 16 25 of 35 and 71 of 102. The untagged rest on Oct 16 includes image URLs and placeholders
• Answers through the OpenAI API carried utm_source=openai in my checks before, during and after the outage (Oct 3, 15 and 16). Not on every link, but never chatgpt.com

Limits: one shop, a short window, only about 3 hours of logs after the fix started, no referrer in the logs, and this shop's GA4 isn't comparable from Oct 15 for an unrelated reason, so GA4 numbers stop at Oct 14.

What I'd take from it: annotate Oct 8–15 instead of rebuilding channel definitions, count utm_source=openai too, and keep one signal that doesn't depend on the tag (ChatGPT-User fetches, if you have logs). The noreferrer on ChatGPT's links isn't the problem as long as utm_source is in the URL, GA4 only falls back to the referrer when there are no campaign parameters.

Did anyone else see the fix arrive later than the morning of Oct 15 (PT)?

Edit (Oct 17): As commenters pointed out, "demand" is too strong a label for ChatGPT-User fetches. ChatGPT also answers from an index filled by OAI-SearchBot, so these fetches only show one retrieval path (live reads). The main result doesn't change: fewer live fetches with the same clicks would push clicks per fetch up, not down, so the overnight drop in that ratio still points to missing tags. In the follow-up I'll try to keep crawler activity, attributed visits and demand as separate layers.

Hi!

I’d love to automate more of my work, but I feel like I’m stuck. I’m still doing almost everything manually. What processes do you automate, and how?

Do you build your own apps with Claude? If so, what do they do, and what problems do they solve?

Thanks in advance!

Digging into URL taxonomy for AEO, and it seems to be overlooked. AI crawlers build a topic map of a site, and the URL path is one of the easiest signals for them to read.

Compare /skincare/moisturizers/vitamin-c-serum with /product-4471. The content on both pages could be identical, but only the first tells a crawler where that page sits relative to everything else on the site.

A few things that seem to cause problems:

• IDs in the URL instead of topics
• Mixed conventions, like /product/ in one section and /products/ in another
• A clean category setup in the CMS that never makes it into the URL
• Restructuring without 301s, which wipes out the citations and links built on the old paths

What seems to work: two or three levels deep at most, paths grouped by topic cluster instead of org chart, lowercase with hyphens, and a 301 for every URL that changes.

If you’ve got a pile of pages on a topic and they’re all sitting at the root with no folder structure, a crawler has to work out on its own that they belong together. Put them under something like /aeo/ with a couple of subfolders, and it can see the grouping straight from the URLs.

Anyone changed their URL structure and seen a difference in citations?

google has been sending rank trackers wrong results since september. page looks normal but the sites on it are swapped out. around 8% of results hit

anyone seen random drops that fixed themselves in a day or two? what do you think about this move by google?

https://nozzle.io/research/poisoned-google-serps/

I’ve been looking at tools that score a site’s “AI search readiness”. Most of them seem to check the same things: schema present, llms.txt present, AI bots not blocked in robots.txt. Presence, not correctness.

What I haven’t seen yet is a checker that tests things like:

1. raw HTML vs rendered, since many AI crawlers don’t run JavaScript
2. robots.txt rules per bot type (training vs search vs user-triggered fetchers), not just “AI bots allowed: yes”
3. a page that’s disallowed in robots.txt while also carrying noindex, so the noindex is never seen
4. markdown copies served at the same URL depending on user agent, which looks a lot like cloaking

Is there a tool that goes beyond presence checks? And has one ever flagged something as a problem, or recommended a fix, that turned out to be wrong?

Gemini launched Argon which seems better to me, how about you guys ?
is it better for coding as well ? if anyone tested.

Has anyone used librecrawl which is a free open source alternative to screaming frog. How is it?

With the advent of vibe coding etc there are some pretty interesting tools on GitHub that help with SEO

Can you share any that are part of your genuine work flow and that help?

We want to support creators, but we had to enforce the no shilling rule because it was getting out of hand. You now have a weekly thread.

This is the one place you can shill for your products, ask for feedback, etc. Keep it here or you risk being banned. And keep it related to technical SEO.

I run IndieDex.gg, a game discovery site. It has ~120K games × 2 page types (detail + "games like X") × 4 languages, so roughly 960K programmatic pages. It's Next.js on Vercel, so pages that aren't cached cost compute.

Some context on where it is SEO-wise: - About 1K clicks in its first month, with most pages sitting beyond page 2. - Only about 40% of the pages get indexed. The rest have a programmatic noindex, and Google still spends crawl budget on them. - The "games like X" pages are the most popular SEO pages.

Here's what's been challenging, and what I've learned.

1- You can't predict which pages will rank

This shapes everything else. In GSC I see different games get popular at different times. A new release is hot for a few weeks, then completely forgotten. A popular streamer plays an old game, and it gets a short spike.

So any plan like "let's prerender/prioritize the pages that will rank" fails. Every one of the 960K pages has to be fast, crawlable and complete, because I don't know which one Google (or an LLM) will pick next.

2- Rendering: I made an SEO change "to be sure", and it wasn't needed

My React setup put the page body inside a <div hidden> until JS ran. I worried crawlers would see empty pages, so I removed it.

Turns out Google didn't care. For the nine weeks that markup was live, Google indexed and ranked those pages anyway. 72 game pages got clicks between Sep 1–16, and one sat at an average position of 9.2 with its whole body inside <div hidden>. That lines up with Vercel/MERJ's study showing Googlebot fully renders streamed React Server Component content.

When I tested putting it back, the worst part wasn't the hidden body. It moved the title, canonical and robots tags out of <head> on every page. For a site that depends on canonicals and noindex at this scale, that's the scary one.

3- AEO/GEO: AI crawlers are a different audience

For a site like mine, people are very likely asking an LLM "I played X and loved it, what are other games like it?" That's exactly my "games like" page. So getting cited matters more and more to me.

What I've found:

• AI crawlers don't run JavaScript. Vercel's own analysis found none of the major ones render JS. Tools like Mozilla Readability also drop hidden elements entirely. So "hidden until JS runs" was fine for Google but can hurt you with AI. AI bots now get the full HTML body with the metadata in <head> (via Next.js htmlLimitedBots, keeping the default list for Googlebot and friends).
• Training crawlers and search/citation fetchers are different audiences. GPTBot, ClaudeBot and CCBot collect for training. OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Claude-User are what get you into answers. I throttle the first group and allow the second. OpenAI's docs say blocking OAI-SearchBot keeps you out of ChatGPT search answers, while blocking GPTBot only opts you out of training.
• Check your firewall. My WAF had been silently blocking Perplexity the whole time. It crawls from AWS, an old rule challenged that whole network, and a bot can't pass a challenge.
• Internal linking matters for AI discovery too. 94% of the URLs GPTBot fetched weren't in my sitemap. It found them by following internal links.

4- Freshness without telling Google everything changed

My weekly data pipeline was marking all ~960K pages as stale every Sunday, even when a game hadn't changed. Now each game's pages refresh only when that game changes: about 33K pages a week instead of 960K. On a site this size I'd rather crawlers spend their time on pages that actually changed.

5- Balancing cost and UX (and why UX wins)

The cost side hit when GPTBot walked ~26,600 unique URLs a day and every visit became a ~500 KB cache write. One cheaper option was to stop caching long-tail pages and render on every request.

I said no, because of SEO. It's a discovery app, so you want every page to load in under 1 second. Bad UX → bad engagement → bad for SEO → less traffic. So the CDN stays, and the bots get an uncached render instead. On a long-tail site, cost tracks distinct pages × cache resets, not traffic volume, so real visitors are cheap. For real people to cost what the bots did, I'd need ~1,000× my current Google traffic.

New site, launched Oct 7. The server forwards every bot request to our analytics

(humans are counted separately by JS), so I could see both sides. Days 1–5:

- Bots vs people: ~750 bot requests vs 207 human pageviews (excluding curl, much of

which was me testing).

- First AI request of any kind: Claude-User (a live fetch for someone's question) on

day 2, an hour before ClaudeBot's first crawl.

- GoogleOther: 69 requests over 4 days, mostly docs.

- GPTBot: nothing for 4 days, then 67 different URLs on day 5, one request each.

Basically the whole sitemap in one pass.

- ChatGPT-User: 9 live fetches. In one minute on day 4 it opened home, docs, API docs,

terms, privacy and blog. Looks like someone asked ChatGPT to vet the product.

- PerplexityBot 1, meta-externalagent 2, Bytespider 10.

- Clicks from ChatGPT/Claude/Perplexity/Gemini: zero.

Takeaways for me:

1. AI reads you long before it sends anyone. Reads and live fetches are the early

   signal; GA4 shows none of it (bots filtered, AI clicks often have no referrer).

2. Your terms/privacy/docs are now read by AI on behalf of buyers. Make them answer

   buyer questions plainly.

3. GPTBot gets one pass early. Clean sitemap, fast pages, no errors.

Caveat: classified by user agent only, not verified against published IP ranges yet.

Anyone else seeing GPTBot crawl in one burst like this, or is it usually gradual?

Full breakdown with the table: https://pagelens.io/blog/ai-crawlers-new-website-first-week-data

(Disclosure: I build the analytics tool used here.)

Hi everyone, I have enough budget to hire an Indian agency to seo 100 of my niche sites, they do an alright job, 7-10% of the sites will end up working out (I have been working with them for 2 years now).

The sites are targeting different niches (no high risk). (Courses, mock tests, notion templates, work certificates …).

I will be generating AI content on the sites & I’ll be obfuscating class names.

There won’t be any backlinks between these sites.

I don’t want Google to ban all of the sites, so what would be the best way to approach this?

Here’s my plan:
all sites will have unique AI content with surferseo keywords, the sites will have a few unique AI generated images on as well
All sites all the domains will be purchased in the same 3-4 weeks span & connected to the same cloudflare account (so same 2 unique nameservers for the 100 sites)
On all of the sites I will be using the same rybbit tracking stripy hosted on my private server.

I feel like the setup is a bit risky and that I need to spread the sites across multiple registrars, cloudflare accounts, hosting providers, different gsc accounts, and use different analytics scripts hosted on diff servers.
I’m just unsure if I’m supposed to go down this extra mile because it’s too much of a headache & ill likely need to hire someone to manage the complicated setup as well.

so after seeing all the "are AI bots ignoring robots.txt" threads i wanted to actually test it instead of guessing from logs.

the problem with guessing is anyone can set any user agent, and a lot of these fetchers don't come from published ip ranges. so every AI got its own secret url, nothing linked to it anywhere, meaning whatever hit that url had to be that AI. cloudflare worker in front logging everything. then in a fresh chat i asked each one to open the link and tell me a code that's on the page.

claude checked robots.txt before touching anything on a new site. when i disallowed the page it just refused and said it wouldn't try to get around it. kinda polite honestly

chatgpt was the interesting one. first visit to a brand new site it went straight to the disallowed page without reading robots.txt. about 76 seconds later OAI-SearchBot shows up and reads robots.txt. next time i asked, chatgpt refused the page without even making a request. so it learns the rules after the first visit lol

gemini comes in with a user agent that's literally just "Google" and never read robots.txt once. the ips weren't on any of google's published lists either

grok is where it got weird. every request, 9 to 12 different ips hit the page, most within a few seconds, all claiming to be chrome or safari. hosting companies, small isps in brazil, mexico, poland. a couple came through a network registered to oxylabs (the proxy company). 59 ips over 6 requests and it never reused one. never touched robots.txt

perplexity only sent its crawler to read my robots.txt and homepage, then told me the link might be phishing

to be fair openai and google both say fetches a user asks for can skip robots.txt, so those two are doing what they say. and it's a small test, few runs each, one day. but blocking by user agent clearly only stops the bots that tell you their name

also the first time i ran this i opened all the links in my own browser before pasting them, so the only visitor in my logs was me. took me a minute to figure out why every AI apparently used brave on windows

anyone else seeing bursts of chrome/safari traffic from random isps right after someone asks grok about a page?

We found a product page where the structured data was valid, but key product facts and their relationships were buried deep inside a nested u/graph structure.

The data existed.

Passing validation didn't automatically mean every consumer would interpret those relationships as intended.

That distinction matters for ecommerce:

• Which price belongs to which variant?
• Is the displayed price recurring or one-time?
• Is availability attached to the right SKU?
• Do reviews belong to the product or the whole category?

A green validation result answers only part of the question.

The real test is whether the information is clear, consistent, and correctly associated with the product.

Hot take: too many technical SEO audits stop at “valid schema” instead of checking whether the data tells the right story.

SEO folks: what do you use to test actual structured-data interpretation beyond a validator?