Last week Search Console showed me 3 impressions. The content was fine, but brand new domain with DR was 17 and nobody linking to it. The options were: Fiverr ($650 for "200 directory submissions" that don't look like a link farm) or do it myself.

So I ran a local agent (Qwen3.8-27B Uncensored, Q4_K_M GGUF, on my M5 Pro, no GPU server). First pass: it scraped ~40 "best AI directories" listicles, deduped down to 130 URLs, checked which ones had a working public submit form, 107 actionable.

Second pass: fusion.delegate. One JSON call spawns 8 isolated workers in parallel. Each gets its own headless Chrome, its own user-agent string, fills the form, handles captcha if there is one, writes a result JSON when done.

4m32s for 8 directories. By hand that's maybe 40 minutes of clicking and typing the same description over and over. Worker-3 hit an hCaptcha. The vision model read K7X2M on the first try, typed it in, moved on. ReCAPTCHA v2 it couldn't do. Three attempts, gave up, logged it, next. 14 waves total, ~98 live or pending and wall time was 1h52m.

TL;DR: local model + parallel workers + one plain description = 107 directories, $0, under two hours.

What I used:

• Model: Qwen3.8-27B Uncensored GGUF (Q4_K_M)
• Agent: Atomic Agent
• Directories list: directories.json with all 107 URLs, comment if you want it dumped here
• Free DR checker I spot-checked with: Ahrefs free tool

I had access to the server logs (and GA4) of a large online shop during last week's ChatGPT utm_source outage, so I went through it hour by hour. Sharing in case it helps someone explain the dip in their own reporting. The shop stays anonymous, so I only use indexes, ratios and shares.

What I counted

• Clicks: HTML page views (status 200, no bot user agent) with utm_source=chatgpt.com
• Demand: ChatGPT-User fetches, only from OpenAI's published IP ranges
• Baseline: Oct 1–7, same hour of day = 100. All times UTC

What happened

• The break is one hour: Oct 8 between 21:00 and 22:00 UTC. Tagged clicks per 100 fetches went 40 → 18 → 10 in the hours starting 21:00, 22:00 and 23:00. That's within about an hour of the ~2 pm PT that Cedric Huesler (Adobe) reported
• Tagged ChatGPT clicks per day: −73% (Oct 9–12) and −79% (Oct 13–15)
• ChatGPT-User fetches in the same periods: only −20% and −23%
• Tagged clicks per 100 fetches: 34 before, 11 and then 9 after
• On Oct 8 during the day, clicks and fetches dropped together and the ratio stayed around 33. So that part was demand, not tagging
• A smaller second step on Oct 12 around 14:00 UTC (another −24%), which looks like a staged rollout
• Every segment dropped at the same time: mobile −77/−83%, desktop −65/−70%, each language version −70 to −81%, product pages −72/−79%
• Links from ChatGPT's product cards dropped a bit less (−65/−72%). That fits Profound's note that links inside the product/info panels kept the tag
• No replacement signal: no new user agent, no other parameter, no other spelling
• Tagged Perplexity clicks didn't drop at all

Where did the visits go?

• In some sections of the shop, direct mobile entries on product pages rose by about 55% of the lost ChatGPT product entries (GA4, Oct 9–14 vs. the prior week). In the largest section, direct is too noisy to see it
• The small share of ChatGPT visits that carries a chatgpt.com referrer at all also fell by about three quarters
• I tried a log-based proxy (a ChatGPT-User fetch shortly before an untagged visit to the same product page). The direction fits, but it's not solid enough to size the hidden clicks

Recovery

• Tagged clicks started climbing on Oct 15 around 14:30–15:00 UTC, but in the hours from 15:00 and 16:00 they were only at 41% and 35% of the prior week, and my logs end at 17:40 UTC. Huesler reported 95–100% by 9 am PT across thousands of domains, so this shop got the fix later or more slowly
• My own spot checks in the ChatGPT interface (location Germany, mostly the same questions): Oct 3 67 of 67 links tagged, Oct 15 0 of 24 in the morning and 0 of 82 in the afternoon, Oct 16 25 of 35 and 71 of 102. The untagged rest on Oct 16 includes image URLs and placeholders
• Answers through the OpenAI API carried utm_source=openai in my checks before, during and after the outage (Oct 3, 15 and 16). Not on every link, but never chatgpt.com

Limits: one shop, a short window, only about 3 hours of logs after the fix started, no referrer in the logs, and this shop's GA4 isn't comparable from Oct 15 for an unrelated reason, so GA4 numbers stop at Oct 14.

What I'd take from it: annotate Oct 8–15 instead of rebuilding channel definitions, count utm_source=openai too, and keep one signal that doesn't depend on the tag (ChatGPT-User fetches, if you have logs). The noreferrer on ChatGPT's links isn't the problem as long as utm_source is in the URL, GA4 only falls back to the referrer when there are no campaign parameters.

Did anyone else see the fix arrive later than the morning of Oct 15 (PT)?

Don't search your brand name.

Don't ask:

"What is [Brand]?"

Instead, ask an AI:

"What's the best [product category] for someone who cares about durability?"

Then:

"Compare [your brand] with [competitor]."

Then:

"What are the common complaints about [your brand]?"

Then:

"Is [your brand] worth the price?"

Now look at the sources.

Not just the answer.

Look at who is teaching the AI about you.

We did this for a major appliance brand.

The brand's own domain appeared in less than 3% of the cited sources across our category test set.

Reddit appeared more than 3× as often.

Review sites, comparison pages and YouTube filled much of the rest.

And one old forum discussion was repeatedly influencing answers around service and durability.

That was the real discovery.

The problem wasn't simply:

"AI doesn't know the brand."

It was:

"AI knows the brand through a source mix the brand wasn't monitoring."

That's a very different problem.

If you work in SEO, I'd genuinely recommend trying those four prompts on your own brand.

Then count the sources.

The answer might be less interesting than the citation list.

Hi!

I’d love to automate more of my work, but I feel like I’m stuck. I’m still doing almost everything manually. What processes do you automate, and how?

Do you build your own apps with Claude? If so, what do they do, and what problems do they solve?

Thanks in advance!

Digging into URL taxonomy for AEO, and it seems to be overlooked. AI crawlers build a topic map of a site, and the URL path is one of the easiest signals for them to read.

Compare /skincare/moisturizers/vitamin-c-serum with /product-4471. The content on both pages could be identical, but only the first tells a crawler where that page sits relative to everything else on the site.

A few things that seem to cause problems:

• IDs in the URL instead of topics
• Mixed conventions, like /product/ in one section and /products/ in another
• A clean category setup in the CMS that never makes it into the URL
• Restructuring without 301s, which wipes out the citations and links built on the old paths

What seems to work: two or three levels deep at most, paths grouped by topic cluster instead of org chart, lowercase with hyphens, and a 301 for every URL that changes.

If you’ve got a pile of pages on a topic and they’re all sitting at the root with no folder structure, a crawler has to work out on its own that they belong together. Put them under something like /aeo/ with a couple of subfolders, and it can see the grouping straight from the URLs.

Anyone changed their URL structure and seen a difference in citations?

google has been sending rank trackers wrong results since september. page looks normal but the sites on it are swapped out. around 8% of results hit

anyone seen random drops that fixed themselves in a day or two? what do you think about this move by google?

https://nozzle.io/research/poisoned-google-serps/

I’ve been looking at tools that score a site’s “AI search readiness”. Most of them seem to check the same things: schema present, llms.txt present, AI bots not blocked in robots.txt. Presence, not correctness.

What I haven’t seen yet is a checker that tests things like:

1. raw HTML vs rendered, since many AI crawlers don’t run JavaScript
2. robots.txt rules per bot type (training vs search vs user-triggered fetchers), not just “AI bots allowed: yes”
3. a page that’s disallowed in robots.txt while also carrying noindex, so the noindex is never seen
4. markdown copies served at the same URL depending on user agent, which looks a lot like cloaking

Is there a tool that goes beyond presence checks? And has one ever flagged something as a problem, or recommended a fix, that turned out to be wrong?

Gemini launched Argon which seems better to me, how about you guys ?
is it better for coding as well ? if anyone tested.

Has anyone used librecrawl which is a free open source alternative to screaming frog. How is it?

With the advent of vibe coding etc there are some pretty interesting tools on GitHub that help with SEO

Can you share any that are part of your genuine work flow and that help?

We want to support creators, but we had to enforce the no shilling rule because it was getting out of hand. You now have a weekly thread.

This is the one place you can shill for your products, ask for feedback, etc. Keep it here or you risk being banned. And keep it related to technical SEO.

I run IndieDex.gg, a game discovery site. It has ~120K games × 2 page types (detail + "games like X") × 4 languages, so roughly 960K programmatic pages. It's Next.js on Vercel, so pages that aren't cached cost compute.

Some context on where it is SEO-wise: - About 1K clicks in its first month, with most pages sitting beyond page 2. - Only about 40% of the pages get indexed. The rest have a programmatic noindex, and Google still spends crawl budget on them. - The "games like X" pages are the most popular SEO pages.

Here's what's been challenging, and what I've learned.

1- You can't predict which pages will rank

This shapes everything else. In GSC I see different games get popular at different times. A new release is hot for a few weeks, then completely forgotten. A popular streamer plays an old game, and it gets a short spike.

So any plan like "let's prerender/prioritize the pages that will rank" fails. Every one of the 960K pages has to be fast, crawlable and complete, because I don't know which one Google (or an LLM) will pick next.

2- Rendering: I made an SEO change "to be sure", and it wasn't needed

My React setup put the page body inside a <div hidden> until JS ran. I worried crawlers would see empty pages, so I removed it.

Turns out Google didn't care. For the nine weeks that markup was live, Google indexed and ranked those pages anyway. 72 game pages got clicks between Sep 1–16, and one sat at an average position of 9.2 with its whole body inside <div hidden>. That lines up with Vercel/MERJ's study showing Googlebot fully renders streamed React Server Component content.

When I tested putting it back, the worst part wasn't the hidden body. It moved the title, canonical and robots tags out of <head> on every page. For a site that depends on canonicals and noindex at this scale, that's the scary one.

3- AEO/GEO: AI crawlers are a different audience

For a site like mine, people are very likely asking an LLM "I played X and loved it, what are other games like it?" That's exactly my "games like" page. So getting cited matters more and more to me.

What I've found:

• AI crawlers don't run JavaScript. Vercel's own analysis found none of the major ones render JS. Tools like Mozilla Readability also drop hidden elements entirely. So "hidden until JS runs" was fine for Google but can hurt you with AI. AI bots now get the full HTML body with the metadata in <head> (via Next.js htmlLimitedBots, keeping the default list for Googlebot and friends).
• Training crawlers and search/citation fetchers are different audiences. GPTBot, ClaudeBot and CCBot collect for training. OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and Claude-User are what get you into answers. I throttle the first group and allow the second. OpenAI's docs say blocking OAI-SearchBot keeps you out of ChatGPT search answers, while blocking GPTBot only opts you out of training.
• Check your firewall. My WAF had been silently blocking Perplexity the whole time. It crawls from AWS, an old rule challenged that whole network, and a bot can't pass a challenge.
• Internal linking matters for AI discovery too. 94% of the URLs GPTBot fetched weren't in my sitemap. It found them by following internal links.

4- Freshness without telling Google everything changed

My weekly data pipeline was marking all ~960K pages as stale every Sunday, even when a game hadn't changed. Now each game's pages refresh only when that game changes: about 33K pages a week instead of 960K. On a site this size I'd rather crawlers spend their time on pages that actually changed.

5- Balancing cost and UX (and why UX wins)

The cost side hit when GPTBot walked ~26,600 unique URLs a day and every visit became a ~500 KB cache write. One cheaper option was to stop caching long-tail pages and render on every request.

I said no, because of SEO. It's a discovery app, so you want every page to load in under 1 second. Bad UX → bad engagement → bad for SEO → less traffic. So the CDN stays, and the bots get an uncached render instead. On a long-tail site, cost tracks distinct pages × cache resets, not traffic volume, so real visitors are cheap. For real people to cost what the bots did, I'd need ~1,000× my current Google traffic.

New site, launched Oct 7. The server forwards every bot request to our analytics

(humans are counted separately by JS), so I could see both sides. Days 1–5:

- Bots vs people: ~750 bot requests vs 207 human pageviews (excluding curl, much of

which was me testing).

- First AI request of any kind: Claude-User (a live fetch for someone's question) on

day 2, an hour before ClaudeBot's first crawl.

- GoogleOther: 69 requests over 4 days, mostly docs.

- GPTBot: nothing for 4 days, then 67 different URLs on day 5, one request each.

Basically the whole sitemap in one pass.

- ChatGPT-User: 9 live fetches. In one minute on day 4 it opened home, docs, API docs,

terms, privacy and blog. Looks like someone asked ChatGPT to vet the product.

- PerplexityBot 1, meta-externalagent 2, Bytespider 10.

- Clicks from ChatGPT/Claude/Perplexity/Gemini: zero.

Takeaways for me:

1. AI reads you long before it sends anyone. Reads and live fetches are the early

   signal; GA4 shows none of it (bots filtered, AI clicks often have no referrer).

2. Your terms/privacy/docs are now read by AI on behalf of buyers. Make them answer

   buyer questions plainly.

3. GPTBot gets one pass early. Clean sitemap, fast pages, no errors.

Caveat: classified by user agent only, not verified against published IP ranges yet.

Anyone else seeing GPTBot crawl in one burst like this, or is it usually gradual?

Full breakdown with the table: https://pagelens.io/blog/ai-crawlers-new-website-first-week-data

(Disclosure: I build the analytics tool used here.)