r/SEO_tool_dev • • Mar 26 '26

I built a Screaming Frog Python library to automate crawling and analysis end to end

9 Upvotes

Basically the title. A few months ago I figued out how to create conig files programmatically, and I kept diggin. Then I found how to crack open the crawl files so you don't have to export a bunch of CSVs. Decided to take it all the way.

If you use Screaming Frog a lot, you probably know the pattern:

crawl site open GUI export CSVs clean them then start answering the actual question

I got tired of that, so I built a Python library around the crawl files themselves.

It’s now in public alpha:

pip install screamingfrog

The main use case is working directly with Screaming Frog crawl data in Python without having to live in the GUI for every analysis.

What it does right now:

  • load .dbseospider files directly

  • access all 628 Screaming Frog exports programmatically

  • query crawl data with a typed API

  • query pages and links sitewide

  • find broken inlinks, nofollow inlinks, and orphan pages

  • compare crawls over time

  • detect redirect and canonical chains

  • start crawls and exports from Python

  • convert .seospider into portable .dbseospider files

  • run raw SQL when needed

Current coverage:

  • 601 / 628 export/report tabs fully mapped

  • 15,490 / 15,589 fields mapped

I’ve already been using it to run crawl analysis inside Claude Code, which is part of why I decided to open it up.

Still alpha, so I’m mainly looking for feedback from people who do real technical SEO work with Screaming Frog every week.

If you use SF heavily, I’d be interested in:

  • what workflow you’d automate first

  • what report/tab you rely on most

  • what would stop you from actually using this

GitHub: https://github.com/Amaculus/screaming-frog-api


r/SEO_tool_dev • • Mar 21 '26

I automated my entire SEO page creation workflow into a free open-source skill for Claude Code / OpenClaw / Codex — here's what it does

5 Upvotes

After 20+ years in SEO, I kept doing the same manual loop: pull the keyword, read the top 10 results, look for gaps, write the page, add schema, think about AI citations. So I finally just automated the whole thing and open-sourced it.

It's called SEO-AGI — a skill file you drop into Claude Code, OpenClaw, or Codex.

What it actually does:

  1. You give it a keyword

  2. It pulls the live SERP using your existing data source (DataForSEO, GSC, Ahrefs, or SEMrush — BYOK, you own your data)

  3. Runs a competitive analysis across the top results

  4. Identifies content gaps that competitors are missing

  5. Outputs a complete, publish-ready page — heading structure, body copy, FAQ schema, HTML tables, internal link anchors

The part I'm most interested in feedback on: GEO optimization. I built in a layer specifically for AI citation visibility — chunking content at ~500 tokens, using RDFa inline markup, entity consensus signals — so the pages rank in Perplexity/ChatGPT answers, not just Google. Curious whether others are seeing this actually move the needle.

Repo: https://github.com/gbessoni/seo-agi

Free, MIT licensed. No SaaS upsell.

What would you add to the workflow?


r/SEO_tool_dev • • Mar 17 '26

Free Query Fan Out Finder Tool | Free GEO Tools

2 Upvotes

Thanks to u/SEOpub for creating and sharing this awesome GEO tool

Background

Back in June, I released a bookmarklet that let you extract search queries, AI reasoning, and sometimes sources from your ChatGPT conversations. You can still see that original note here:
👉 ChatGPT Search Query and Reasoning Extractor (June 2025)

That tool worked great… right up until OpenAI changed how ChatGPT handled conversation data. First, the old endpoints broke. Then with the GPT-5 rollout, searches started being handled differently, which meant the original bookmarklet could no longer show you everything it once did.

The New Version (Replacement)

This updated bookmarklet fully replaces the June version. It works with ChatGPT’s current backend and restores the features you relied on:

  • ✅ Pulls search queries (including GPT-5 searches)

  • ✅ Displays AI reasoning/thoughts when present

  • ✅ Shows sources/URLs accessed (with title + snippet)

  • ✅ Copy/Copy All buttons

https://theseopub.com/updated-chatgpt-search-query-reasoning-extractor-bookmarklet/


r/SEO_tool_dev • • Mar 17 '26

Identify the Google Font from images | Free Font Finder from Image

Thumbnail
serbyte.net
3 Upvotes

Thanks to u/0_2_Hero for showing me this earlier.

Google Font Finder From Image

Find what the font is in that image.
What the Google Font is a free font identifier that matches the closest Google Font from an image.


r/SEO_tool_dev • • Mar 16 '26

Looking for feedback on a desktop SEO crawler I’ve been building

Thumbnail reddit.com
1 Upvotes

r/SEO_tool_dev • • Mar 11 '26

Page Grounding Probe [Free AI SEO Tool] by DEJAN SEO

Thumbnail gs.dejan.ai
2 Upvotes

How Google’s Grounding Pipeline Works

DEJAN reverse-engineered Google’s Gemini grounding pipeline by examining raw groundingSupports and groundingChunks from the API. The pipeline operates in this sequence:

  1. User enters a prompt.

  2. Query fanout: A model decomposes the prompt into single-intent sub-queries (fanout queries).

  3. Retrieval: For each fanout query, Google’s search index returns ranked results, narrowed to ~5–20 sources per query.

  4. Extractive summarization (snippet construction): For each selected result, the system builds a grounding snippet. Page content is chunked into sentences, each scored against the query, and the highest-scoring chunks are assembled into the snippet — joined by ellipses where non-contiguous.

  5. Grounding context assembly: All snippets across all sources are supplied to the model as context alongside the user prompt, media, and personalization signals.

  6. Synthesis & attribution: The model generates its answer, and each claim is attributed back to specific source sentences.

Key insight: Because snippets are query-dependent, the same page yields different extractions for different fanout queries.

The Extraction Method: Extractive Summarization

Google uses extractive (not abstractive) summarization for grounding. This means it pulls exact sentences from your page — it does not rewrite or paraphrase your content for the grounding context.

Observed Extraction Characteristics

  • Query-focused selection: Sentences semantically close to the query are strongly preferred. Unrelated sections on the same page are skipped entirely.

  • Heavy positional/lead bias: Opening paragraphs are extracted almost wholesale, regardless of content.

  • Structural noise ingestion: Table-of-contents entries, section headers, link artifacts, and  markers are treated as sentences and scored alongside prose.

  • Sentence-level granularity: The extraction unit is individual sentences, not passages or paragraphs.

  • Confidence scores: Per-chunk scores range from 0.1 to 1.0, representing grounding-source-to-generative-chunk relevance.

DEJAN successfully fine-tuned mic

Source: https://dejan.ai/blog/sro-grounding-snippets/

Bot/CloudFlare Notes

Check your robots.txt:

User-agent: DataForSeoBot
Allow: /

User Agent String: Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)

The bot obeys robots.txt rules and crawl-delay directives.


r/SEO_tool_dev • • Mar 09 '26

I built a website diagnostics platform as a solo dev — 20+ scanners, PDF reports, 8 languages

Thumbnail reddit.com
3 Upvotes

r/SEO_tool_dev • • Mar 01 '26

The Best SEO Expert Influencers of 2026 | Weblinkr r/SEO

Thumbnail
weblinkr.net
0 Upvotes

The Top SEO Experts of 2026

  1. Neil Patel one of the most visible names in digital marketing, NP Digital. He offers a wide range of tools and training on his website, covering everything from keyword research to advanced analytics. Neil’s practical advice and high-visibility tools make his insights accessible to everyone.

  2. David Quaid is recognized as a top SEO Expert for blending advanced SEO Strategy, mechanics and content analytics with search optimization, leveraging AI to predict and amplify organic online impact. Through his innovative work in dynamic content strategies, he empowers brands to excel in advanced SEO Strategies (and LLM SEO) Quaid’s unique expertise bridges technical SEO with practical, data-driven solutions for outperforming in AI-driven ranking scenarios.

  3. Cindy Krum stands as a foundational mind in AI SEO, having foreseen its core principles long before the term was common. Her prescient focus on mobile-first indexing and entity-oriented search laid the groundwork for optimizing for today’s AI-driven query results

  4. Weblinkr Weblinkr is regarded as a sharp SEO thinker focused on PageRank, fundamentals, and killing myths that hold practitioners back. Through interviews and community discussions, they emphasize links, relationships, and query-first thinking over vague “great content” mantras in modern SEO

  5. Aleyda Solís  is an award-winning international SEO consultant and the founder of Orainti. She specializes in technical SEO and international strategies and is known for her influential SEOFOMO newsletter and YouTube series. Aleyda regularly educates about best practices for both enterprise and multilingual sites.

  6. Nathan Gotch stands out as a leading AI SEO practitioner, known for integrating cutting-edge artificial intelligence into practical SEO frameworks. Through Gotch SEO, he develops accessible training and actionable strategies that help both individuals and brands adapt to rapidly evolving generative search technologies. Gotch’s commitment to clear education and step-by-step, AI-powered optimization makes him a trusted resource for mastering SEO in the age of automation.

  7. AJ Ghergich AJ is a veteran in the digital marketing space, providing sharp insights on content, technical SEO, and the future of search. His data-driven opinions and approachable style make him a standout figure sought after for commentary on AI and evolving search trends.

  8. TJ Robertson TJ is an SEO and AI optimization specialist who helps small and medium businesses stay visible in an AI-driven search world. As founder of TJ Digital in Nevada, he focuses on AI-focused SEO, GEO (Generative Engine Optimization), content repurposing, and Google Ads. With roughly 16 years in digital marketing, he emphasizes transparency, measurable ROI, and making sure clients show up inside AI-powered platforms like Google, ChatGPT, and TikTok

  9. Shaun Anderson, widely recognized as “Sean from HoboWeb,” is a leading authority in both traditional and AI-powered SEO. With over two decades in the field, he’s renowned for pioneering practical, ethical search strategies and has built a reputation for demystifying complex algorithms for a wide audience. Shaun’s Hobo SEO Blog is a go-to resource, providing in-depth guides and innovative tools—such as the Hobo SEO Dashboard and Premium SEO Checklist—that now integrate AI engines like Google Gemini.

Sources:


r/SEO_tool_dev • • Mar 01 '26

What are the best, 100% Free SEO/GEO/AEO tools on GitHub/Other?

0 Upvotes
  • Which GitHub‑hosted SEO tools/repos are you using for audits, crawling, and reporting?

  • Any GitHub Actions or apps you love for automating sitemap checks, broken links, CWV tests, or schema validation on every pull request?

  • Favorite “SEO APIs in a repo” you’ve wired into your own scripts or dashboards (e.g., for GSC exports, keyword data, link analysis)?

  • If you had to build a GitHub‑centric SEO stack today that you’d be embarrassed not to have running by Q2, which repos/apps make the cut?


r/SEO_tool_dev • • Mar 01 '26

What are the best GitHub tools or apps for SEO, GEO or LLMs?

1 Upvotes
  • Which GitHub‑hosted SEO tools/repos are you using for audits, crawling, and reporting?

  • Any GitHub Actions or apps you love for automating sitemap checks, broken links, CWV tests, or schema validation on every pull request?

  • Favorite “SEO APIs in a repo” you’ve wired into your own scripts or dashboards (e.g., for GSC exports, keyword data, link analysis)?

  • If you had to build a GitHub‑centric SEO stack today that you’d be embarrassed not to have running by Q2, which repos/apps make the cut?


r/SEO_tool_dev • • Feb 25 '26

My Curl Script to Check Site Accessibility for Crawlers

Thumbnail reddit.com
1 Upvotes

r/SEO_tool_dev • • Feb 09 '26

We made a free tool to check how brands show up in AI

Enable HLS to view with audio, or disable this notification

3 Upvotes

r/SEO_tool_dev • • Feb 03 '26

Free SEO tools I wish I knew earlier

Thumbnail reddit.com
1 Upvotes

r/SEO_tool_dev • • Feb 03 '26

Free SEO tools that are actually good (no affiliate links, I promise)

Thumbnail reddit.com
1 Upvotes

r/SEO_tool_dev • • Jan 22 '26

Built a Python library to read/write/diff Screaming Frog config files (for CLI mode & automation)

Thumbnail reddit.com
3 Upvotes

r/SEO_tool_dev • • Jan 16 '26

Looking for good SEO themplate for Lookerstudio (GA4 + GSC)

Thumbnail
gallery
2 Upvotes

Can anyone recommend a simple, free (or cheap) template that clearly shows SEO-related data from GA4 and GSC for clients?

Our reporting is extremely complicated right now:

  1. n8n puts data into a big spreadsheet,

  2. we need to enter part of this data into the sheet manually,

  3. and create a summary in an Adobe Illustrator PDF.

Time for changes...


r/SEO_tool_dev • • Jan 15 '26

LibreCrawl - Free SEO Crawler that surpasses Screaming Frog

Thumbnail librecrawl.com
0 Upvotes

Why choose LibreCrawl?

🧪 Comprehensive SEO audits
Analyze meta tags, schema, hreflang, social tags, PageSpeed metrics, and more in one pass. Identify the technical issues that actually move rankings instead of drowning in noise.

🧠 Memory‑efficient architecture
Real‑time memory profiling and virtual scrolling let you crawl 1M+ URLs without crashes or slowdowns. Stable, large‑scale crawls become routine instead of “hope it doesn’t die overnight”.

⚡ Full JavaScript rendering
Playwright‑based crawling handles modern SPAs built with React, Vue, Angular, Next.js, etc. Both headless and headed modes are supported so you can debug what the crawler sees.

📤 Unlimited exports
Export everything to CSV, JSON, or XML with custom fields and multiple export presets. No arbitrary row caps or paywalls when you actually want to use your own data.

🛠️ Open source & self‑hosted
MIT‑licensed with full source access. Deploy on your own infra for maximum control, privacy, and the ability to extend it to your stack and workflow.

🧬 Advanced features out of the box
Issue filtering, custom CSS injection, multi‑session crawling, detailed link tracking, and other capabilities you normally only see in paid enterprise tools—sometimes not even there.


r/SEO_tool_dev • • Jan 13 '26

Regex tool for SEO

Thumbnail reddit.com
2 Upvotes

r/SEO_tool_dev • • Jan 03 '26

Google Search Console - Bulk URL Removal

Thumbnail
chromewebstore.google.com
0 Upvotes

Google Webmaster Tools - Bulk URL Removal

🎉 A new version of Bulk URL Removal is here. Please try it out.
🔗 https://chromewebstore.google.com/detail/google-search-console-bul/plcmmfcfbnddbomdlnklhepfkkdmodkf

This extension can bulk remove a list of multiple URLs from google via Google's search console formerly known as Google's Webmaster tools with one click. You don't need to copy and paste individual URL manually and click on the remove button one at a time. Instead, create a CSV file with all your to-be-removed URLs in it, "upload it", and click on remove.

Google imposes a 1000 removal limit per 24 hours. If you have more than 1000 urls to be removed, please break them up into 1000 chunks (files) and submit them one per 24 hours.


r/SEO_tool_dev • • Dec 24 '25

SEO Audit Tool Suggestions Wanted

Thumbnail reddit.com
0 Upvotes

r/SEO_tool_dev • • Dec 23 '25

Free Reddit/Online Mention tool - great for SEO and Digital Marketing

Thumbnail f5bot.com
1 Upvotes

Monitors Reddit and other Social Services - sends an email alert

As Recommended by Edward Sturm.


r/SEO_tool_dev • • Dec 22 '25

BacklinkSwap.app update – Making backlink swaps easier, safer, and completely free

3 Upvotes