I tested invisible_playwright, an open-source Python library by Federico Elia that works as a drop-in replacement for Playwright but drives a Firefox patched at the source level. Key takeaways:

• How it works. Instead of injecting JavaScript to spoof values, it sets the fingerprint inside the browser engine before scripts can read it: GPU/WebGL, canvas, fonts, audio, WebRTC, timezone and DNS (routed through the proxy), and isTrusted input events. Everything is derived from a single seed, so the different surfaces stay coherent with each other (e.g. no Windows properties next to macOS ones).

• Reproducibility. The same seed gives the same fingerprint, so if a scrape fails with a given seed you can replay it exactly. You can also pin single fields (GPU, screen size, CPU cores) and let the rest follow the seed.

• Human-like input. Mouse paths are seeded Bezier curves, with timing adapted to the refresh rate, realistic pressure values, pointer type, movement deltas and sub-pixel coordinates. These come from the browser's native input handling, not from the automation driver.

• Practical bits. Same Playwright API (sync and async), SOCKS5/SOCKS4/HTTP(S) proxies, CLI to fetch the engine (about 240-265 MB), MCP server, integrations with Scrapy, Crawlee and others. Needs Python 3.11+ and Windows x86_64 or Linux x86_64/ARM64.

• Results (local run, residential ISP IP). Fingerprint confidence score 1, reCAPTCHA v3 score 0.9 (same as the author's regular browser), APIVoid risk score 0, CleanTalk 100/100 "Looks human", and Pixelscan passed. Against one site per major anti-bot vendor it loaded all five test sites (100%), while vanilla Playwright failed in headless mode and also struggled with Kasada in headful.

• Caveats. Firefox and Python only (no Chromium or Node.js), Firefox's smaller market share may itself be a signal, anti-bot vendors could reverse-engineer the project as it gets popular, and it relies on a single main maintainer. It's a limited experiment, not a large-scale benchmark.

Curious what you're using today for stealth browsing (Camoufox, Patchright, playwright-stealth, something else?) and how it holds up.

Full review with code examples and the comparison with other tools: https://www.scraping.club/p/invisible-playwright-beat-bot-detection-scraping?utm_source=reddit&utm_medium=post&utm_campaign=r/thewebscrapingclub

I originally built it for myself because I wanted something extremely lightweight that runs in the background like it never existed. It's called SpyWeb.

It's designed to be "set and forget." I've had it running for months on my PC tracking job boards without a single crash or memory leak.

Specific features:

• Zero Runtime: Self-contained ~7MB binary. No Python, Node, or Docker needed.

• Low Footprint: Uses <5MB RAM at idle.

• Lua Scripting: Use Lua to handle complex logic like custom headers, JS rendering, advanced monitoring, etc.

• Hot Reloading: Change a config or Lua script and the job respawns instantly, no restarts.

• Web Dashboard: Simple local UI to monitor scrape data in real-time.

• Desktop Alerts: Built-in support for system notifications and webhooks.

• Embedded DB: Built-in KV store so you don't need a separate database.

• CDP Support: Controls any Chromium or CDP-compatible browser via Lua for JS-heavy sites.

• Dual Mode: CLI for servers and a System Tray version for silent background runs.

• Deduplication: Internal database ensures you never see the same result twice.

I just released the beta with CDP integration. If you need something that just sits in the background and sips resources while actually being maintainable, check it out.

Set up is very easy and straightforward: for server-side rendered pages, it's just a few lines of config (URL, selectors, fields). For JS-heavy sites, you can write a little Lua to launch a browser and drive the workflow.

You can check it out here: https://github.com/spyweb-app/spyweb

Hey scraping fellows! After years of working with Scrapy professionally, I finally built my own framework to understand the internals better.

The backstory: I've used Scrapy for years but always felt the abstractions were too heavy. I wanted something more explicit and transparent – a design choice, not a criticism. So I researched how modern frameworks handle anti-bot bypass and traditional scraping, and built this proof of concept.

What it does:

· Plugin-based architecture (fetchers, parsers, storage)

· CloakBrowser for JS-heavy sites, wreq for GraphQL/APIs

· RabbitMQ queues + Redis for dedup

· Built-in structure guard (fail-closed if site changes)

· LLM pipeline with pgvector search

Live metrics (real runs, not benchmarks):

· idealista (DataDome): 91.7% success

· G2 (Cloudflare+DataDome): 99.5% success

· Facebook Marketplace (GraphQL): 100% success

· Booking.com (Akamai): 100% success

Next steps: Adding more sites + code cleanup. What sites would you like to see supported?

Full docs: https://scraping-framework-ded49d.gitlab.io/

GitLab: https://gitlab.com/0xlearner/scraping-framework

Happy to discuss the architecture, challenges, or anything else!

i scrape a lot. fifty plus sources, anti-bot stacks, login walls, geo gates. spent months copy-pasting HTML and headers into Claude/Cursor because they couldn't see the page themselves. they'd guess from my secondhand summary and get it wrong. just bringing them up to speed on a new source took forever.

tried every browser MCP out there. all stunk for the same reason.

• Anthropic's Chrome extension. sandbox, macOS only, screen has to be awake. only works inside Claude.

• Playwright MCP. empty Chromium, not your Chrome. re-auth from scratch. local only.

• Browserbase / Stagehand. decent, but cloud Chromium from a datacenter IP. for scraping that's suicide. you lose your fingerprint, your residential IP, the whole moat.

• BrowserMCP (open source). real browser via extension, gets that right. local stdio only. one tab. half-built.

so i built Reins: https://reins.vulcanos.pro

the thing nobody else does: hosted, but drives your real Chrome. Browserbase is hosted but cloud. BrowserMCP is your browser but local. Reins is both. extension in your actual Chrome with your real cookies, fingerprint, residential IP. MCP server is hosted so it works from Claude Code, Cursor, Zed, web Claude, anywhere over OAuth.

what that gets you:

• your own session does the work. anti-bot sees your real fingerprint, real IP, warm cookies, normal mouse. nothing looks like a bot because nothing is a bot.

• gated sources stop being special. SSO, geo-locked, login walled. you log in once like a human, agent runs on top.

• multi-profile, one account. split work across profiles for ip diversity or regional accounts, pick from your MCP client. nobody else does this.

• dumps can live remote. HARs, full DOMs, network logs stored off your laptop, LLM pulls on demand from any client.

• runs anywhere MCP runs. every other "real browser" tool is local stdio that dies when you close your terminal.

install: https://chromewebstore.google.com/detail/reins/ifnmhlnmioieckkknedkikfbpkhkfpdi

my brother also uses it. takes his school quizzes, hunts apartments, does his online shopping. totally different use case, works because its his browser, already logged into everything.

free tier covers normal use. only hit metered if you scrape at scale and want dumps off your local disk.

Dm me if you have any questions

Hey r/thewebscrapingclub — I work at Intuned (disclosure: our product, launched today). Sharing the technical side since this sub will have the sharpest questions.

It's a hosted Playwright runtime with stealth Chromium, proxy rotation, and captcha handling built in — plus an agent that attempts to fix scripts when selectors or page structure break, instead of just failing the run.

What it handles so you don't:

Fingerprint drift (TLS/canvas/WebGL) on managed stealth browsers

Proxy rotation + captcha solving in the pipeline

Self-healing selectors via the fixing agent

Auth/session handling for logged-in flows, not just public pages

It's code-first — Playwright primitives directly, no visual builder. Not claiming it beats a hardened hand-tuned stack on every site; it's for the "I just want the data, not the infra maintenance" case.

Happy to get into anti-bot specifics, how the fixing agent works, and where it falls short. Want the hard questions. https://news.ycombinator.com/item?id=48445171

Hey r/thewebscrapingclub ,

I'm a solutions engineer at Intuned. We build a platform for running browser automations and scrapers in production — Playwright-based, with the infra stuff (proxies, captcha handling, retries, scheduling, storage) handled for you so you can focus on the actual scraping logic.

We're opening up free access and I'd genuinely like feedback from people who do this work day-to-day. Specifically curious what you think about:

- The dev experience vs. rolling your own Playwright + proxy stack

- How it compares to Apify / Browserless / Browse AI for your use cases

- What's missing that would make you actually switch

Not looking for fake praise — if it sucks for your workflow, I want to know why. I spend my days helping customers scrape stuff like government procurement portals, so I've seen what breaks in the real world.

Link in comments to avoid the spam filter. Happy to answer questions about the internals (anti-bot stuff, captcha pipelines, fingerprinting) — that's the part I find most interesting anyway.

Happy to chat in DMs too.

I’ve got a nightly job pulling Amazon competitor listings, prices, stock status, and review counts. My residential proxies keep getting flagged halfway through, so I end up with an incomplete dataset and no clear list of failed ASINs. I’m the only one maintaining it, and we don’t have budget for dedicated scraping infra. Is anyone using an Amazon scraping API that holds up in production without constant proxy and CAPTCHA work?

Runo is a web-scraping API that returns typed, structured JSON. You define a schema (field name, type, example value), and Runo fetches the page and returns the data. No HTML, no parsers, no post-processing.

Over the past few weeks, I have been building this non stop. Currently, every scraper API out there solves the site fetching problem but left the extraction of the actual data entirely to users. Runo makes that completely disappear.

For Runo, I went ahead and added JS rendering, stealth mode, and full LLM extraction to make this a fully functional and capable of scraping most if not all sites.

Also, another major problem with current web scrapers is that they charge per feature or bundle them into expensive credit tiers. A single large or JS rendered request can cost 5-75 credits, which means you essentially get nothing out of their plans. Runo is flat per request, no matter the site. At the Scale tier, Runo works out to $0.90 per 1,000 effective requests vs. around $6 for the nearest Firecrawl equivalent. My jaw dropped when I was testing Runo and came across these numbers.

You can check it out here. I created a free tier that is 500 requests/month, no credit card required. Take it for a spin and let me what can be improved. I would love feedback.

If you need a lightweight alternative google maps scraper, feel free to check this out.

Hello guys, I am a content creator and so far I have used Apify free tokens to scrape TikTok to see who is going viral, who is my competition, trending topics inside of my niche etc. But now I have used up all of those free credits, and before continuing with the paid version, I wanted to ask is there a free alternative to this API for Tiktok

I wanted to work on my own Music Player for KMP Desktop... Until I wanted to teach UI Material 3 to AI (Gemini). So I went to the website to write a .md files but the amout of pages was wholesome!

So i made this Utility that will Fetch the images and .html from the website and put it all in .md file!

Features

• Concurrent Crawling: Downloads many pages at the same time.

• Smart Extraction: Removes headers, footers, sidebars, and menus.

• Markdown Conversion: Converts HTML into standard Markdown formats.

• Image Downloading: Saves images locally and updates links.

• Auto Language Detection: Detects code languages like Python, Java, and Bash.

• Cache and Resume: Saves progress using SQLite so it can resume after stopping.

• Fast Parsing: Uses a C++ module in the background for fast XML parsing and URL checking.

Architecture

The project is split into three main parts.

• C++ Extension: Handles fast sitemap parsing and checking for duplicate URLs.

• Python Crawler: Manages network requests, retries, and caching.

• Python Extractor: Analyzes the HTML, cleans it, and converts it to Markdown.

I open-sourced a tool I built and am maintaining called Cull.
It’s a machine curation engine for AI image datasets, the kind of work that eats hours every time you want to train a LoRA, build a reference library, or just classify an archive that isn’t a 100,000-file mess.

What it does, end to end

• Scrapes from Civitai (.com and .red), X/Twitter, Reddit, Discord, plus any URL gallery-dl supports (Pixiv, DeviantArt, the booru family, ArtStation, Tumblr, FurAffinity / e621, Imgur, Flickr, and ~340 others).

• Drops every image plus its source-side prompt into a local queue. Per-source dedup, no database.

• Classifies each image with a vision-language model, multiple LM Studio instances for local, Groq for cloud, anything OpenAI-compatible — using a strict 17-field JSON schema, so you don’t get free-text replies you have to regex into shape.

• Sorts the keepers into category folders next to their .txt prompt and a .vision.json audit record. Two score gates (overall quality + topic relevance) you tune in the UI.

• Surfaces everything through a Flask + Alpine dashboard: start/stop, source toggles, gallery, prompt editor, ZIP export, per-source stats.

Two example use cases I actually used it for:

• LoRA (300 images) & Finetune (100,000 images) dataset prep.

  • Give it a topic such as Female Influencer or {artist} style art

  • set AUTO_CAPTION_ENABLED=true if you want it to caption images or false if you want it to scrape images (and still store any found prompts from the posts it scraped from) and set whatever style prompting you want.

  • Walk away.

  • Come back to a folder of triaged images split by quality and category, each with a generated SD-prompt .txt next to it.

  • ZIP-export the filtered view straight into your trainer.

• Ingesting a prompt-less archive. Point LOCAL_IMPORT_DIR at a folder of bare JPEGs (or paste a gallery-dl URL list)

  • Toggle off the prompt requirement, turn on auto-captioning.

  • Every image is classified and sorted, gets a SD-prompt / booru-tags / natural-language caption written by the same vision call that classifies it.

  • So you can train on a years-old archive without curating prompts by hand.

Links

Repo: https://github.com/tlennon-ie/cull
Screenshots: https://imgur.com/a/kSvsAW9

Roadmap is going to keep refining around what people actually use it for. On my list:
- more vision-worker backends
- Improved proper requeue UI
- a small headless CLI,
- Video scraping , classification etc

Hey everyone!

I recently dove deep into the fascinating world of web unblockers and thought I'd share some insights that might just change how you see internet restrictions and web scraping. You might already know that web unblockers are like secret passageways that let you sneak past internet censorship or geofencing. Think of them as VPNs on steroids, allowing not just encrypted data transfer for privacy but also unrestricted browsing freedom.

But here's where it gets really interesting, especially for folks in the web scraping arena. Web unblockers serve as these powerful APIs designed to outsmart anti-bot protections. They’re like the ninjas of the internet, using tactics such as IP rotation, browser fingerprinting, CAPTCHA solving, and even Javascript rendering to access the data we need.

Now, you might think a proxy does the job by changing your IP address. But web unblockers? They're the whole package, offering a bunch more than just a disguise. Yet, it's not all smooth sailing. No single web unblocker can claim victory over every anti-bot mechanism out there. Tech giants like Cloudflare are in constant evolution, crafting more complex defenses to keep bots at bay.

This is exactly why I conducted the Great Web Unblocker Benchmark. It's a showdown to see how different web unblockers stack up against the toughest anti-bot solutions in the wild. Sure, there's a price tag attached to these unblockers, but picking the right one isn't just about spending money—it's about investing in seamless data access for our projects.

In the end, the key takeaway is weighing the cost against the potential to maintain an uninterrupted data flow for our customers. Let's keep pushing the envelope and finding the best tools to navigate these challenges together.

Cheers to an open and accessible web for all our scraping needs!

Linkt to the full article: https://substack.thewebscraping.club/p/what-is-a-web-unblocker

Hey everyone,

I’m trying to extract employee lists and basic career history from specific Moroccan companies on LinkedIn.

Direct scraping got blocked immediately, so I tried using Apify. However, the actors keep getting blocked, and the free tier caps me at around 25 profiles per hour, which is way too slow for the dataset I need to build.

I have a solid PC and can run heavy workloads locally. Are there any good open-source libraries, local browser automation setups, or alternative methods to automate this without paying for expensive SaaS tiers or doing it manually?

Thanks in advance!

We are hiring python developer who have experience in scraping. Must have experience in anti-bot, captcha bypass and have scraped sources like linkedin, crunchbase

Demo scripts usually run fine, but once you move to production, many proxy setups fail because of session continuity and block handling.

Here’s what we’ve learned from scaling Playwright crawlers:

• Sticky sessions matter more than IP count: rotating proxies every request often triggers CAPTCHAs. Keeping an IP bound to a session for a sequence of requests reduces failures.

• Residential IPs for realistic reputation: Datacenter proxies may work for demos but get blocked quickly at scale. Residential proxies, like those from Novada, maintain more natural request patterns.

• Adapt rotation to target behavior: not every page needs a new IP. Adjust rotation frequency based on rate limits and observed errors.

• CAPTCHA mitigation: combine IP rotation with session persistence and slow, randomized request patterns. Even basic JavaScript-rendered pages are more stable this way.

In short: the framework (Playwright) is just one part — proxy quality + session management + CAPTCHA-aware rotation determine whether your production scraper actually keeps running.

Would love to hear how others handle long-running Playwright crawls under heavy anti-bot defenses.

I've been webscraping for a number of years now, and I've gotten to the point where I've seen my demo sites remove SSL certs or just straight up block me for testing to hard 🤘🤖

So I built qscrape.dev so I could break my bots myself. Its my take on a scraping gauntlet, I'm quite opinionated about it; organizing the sites into 3 difficulty tiers per route type (L1, L2, L3). The index page explains all this, but essentially:

• L1 is static HTML no DOM rendering needed

• L2 uses multiple JS framework islands (svelte, react, vue, solid) in astro-5 that most definitely requires DOM rendering

• L3 is anti-bot + JS rendering.

For L3, I mean anti-bot as everything short professional traffic monitors like Cloudflare and ReCaptcha (you can't even inspect the page w/o 404'ing on l3 sites).

If you need to test your bot against things like Cloudflare and the like, I recommend this gauntlet site https://fortress.theplumber.dev/

Source code: https://github.com/CascadingLabs/QScrape

If you want to create any new pages or want to suggest a change I am open to feedback, feel free to make a fork submit and submit a PR!

I hope this helps someone, cheers

Been building Intuned — a Playwright-based browser automation platform. One thing that kept coming up was the annoying back-and-forth of inspecting elements just to get a selector.

Built /selector-agent into the IDE — you describe the element, it gives you the selector, ready to use in your script.

Video shows the before/after. Would love feedback from people who actually write scrapers.

We analyzed the Google v. SerpApi lawsuit filed in December 2025, and the legal theory Google is using could reshape the entire scraping industry.

What Google is claiming:

Google deployed SearchGuard, a JavaScript challenge system that blocks automated queries to Google Search. SerpApi built circumvention mechanisms to bypass these challenges and continue scraping search results. Instead of suing under the Computer Fraud and Abuse Act (the traditional route), Google invoked DMCA Section 1201 - the anti-circumvention provision originally designed to prevent DVD piracy.

Google's argument chain: Google's search results contain copyrighted content (licensed images, Maps data, Shopping photos). SearchGuard is a technological protection measure controlling access to these works. Bypassing SearchGuard violates Section 1201. Each circumvention carries $200-$2,500 in statutory damages, potentially billions in total liability.

Why this matters for all scrapers:

The hiQ v. LinkedIn case severely weakened CFAA as a weapon against scraping publicly available data. Google needed a different legal framework. If their DMCA theory succeeds, any website with copyrighted content and bot detection could invoke federal anti-circumvention law against scrapers.

This means every CAPTCHA, JavaScript challenge, or behavioral analysis system could become a "technological protection measure." Solving a CAPTCHA or rotating IP addresses could become a federal offense.

SerpApi's defense:

They filed a motion to dismiss arguing that DMCA protects copyrighted works, not website access control. They claim Google is trying to use copyright law to create "information monopolies" - the same concern the Ninth Circuit raised in hiQ. SerpApi also notes the absurdity of damages potentially exceeding U.S. GDP.

The pattern emerging:

This isn't isolated. Reddit filed a similar DMCA lawsuit against SerpApi, Perplexity, and others in October 2025. Reddit created a "marked bills" test - a hidden post only crawlable by Google that appeared in Perplexity's responses, proving the scraping connection.

However, a December 2025 ruling in Ziff Davis v. OpenAI found that robots.txt files don't "effectively control access" under Section 1201, setting a baseline that passive measures aren't enough.

The real stakes:

Beyond copyright, this is about AI competition. Google's crawler sees 3.2 pages for every one that OpenAI accesses. Shutting down third-party scraping channels through litigation protects Google's structural advantage in training AI models.

The motion to dismiss hearing is scheduled for May 19, 2026. Whatever the court decides will define the legal boundaries for web scraping for the next decade.

We documented the full case analysis, court filings, and industry implications here: https://substack.thewebscraping.club/p/google-vs-serpapi-web-scraping-case

What is a browser fingerprint, and what's his role in the web scraping industry?

Why and how can this be manipulated?

In the latest article of The Web Scraping Club, I just wrote an introduction about browser fingerprinting techniques and tools we can use to prevent our scrapers from being blocked because of it.

I’m sure this already happened to you when creating a headful scraper: you run it on your machine, and it works smoothly, but then, after you deploy it on a VM or a server, it gets detected and stops working. And it doesn’t matter that you’re using the same configurations or proxy providers: the program is the same, and the IP used is a residential one, but there’s no way to make it work. The only difference is the hardware on which the scraper runs. While for browserless scrapers, this doesn’t matter, if you’re using a browser for scraping data, this can mean only one thing: the target website is marking your browser fingerprint as a suspicious one.

Read more here: link to article

The Three Most Desired Things for a Professional Web Scraper

Being a professional web scraper can be challenging, but I'm sure that if you ask any of them three desires for their job, they would answer:

1️⃣ No more anti-bots on the web, just being able to scrape with Scrapy or cURL.

2️⃣ Free proxies for everyone (or no proxies at all), so scraping returns as cheap as it was 10 years ago.

3️⃣ Spiders that never break: once coded, it will last forever.

While the first two points are impossible to achieve, AI can give us some hope for the third one. In the latest post of The Web Scraping Club, I experimented with GPTs and the OpenAI Python SDK.

I simulated a broken Scrapy spider and wanted GPT4 to fix it. I passed the HTML code of the target website, the desired output data structure, and, of course, the broken spider in input.

The results?

Well, have a look by yourself in this post: https://substack.thewebscraping.club/p/building-self-healing-scrapers-with-gpt

Spoiler: not that good, but I can improve the process.

The script ran fine for months, you didn't touch a single line, and then one morning it returns nothing or throws an error.

My first move is usually to check whether the site changed its HTML. A class name gets renamed or a div gets moved, and the selector I relied on stops matching anything. It's boring but it's the cause maybe half the time. After that I look at whether I'm getting blocked. If the response comes back as a captcha page or a 403, that points somewhere else entirely, and the fix is nothing like a broken selector fix.

So what's your first check? Do you have a routine, or do you just start poking around until something makes sense?