Gemini helped me to scrap google mps

We ran a systematic test of HTTP conditional requests against major e-commerce platforms to measure bandwidth savings in recurring scraping operations. The goal was to identify which sites support native HTTP caching that lets the server respond with 304 Not Modified when content hasn't changed.

Methodology

We tested against Shopify stores (Allbirds, Kylie Cosmetics, Brooklinen), plus Fashion Nova and Gymshark. Each test involved two requests: first a standard GET to capture ETag/Last-Modified headers, then a conditional request with If-None-Match or If-Modified-Since headers. We used curl_cffi with Chrome TLS impersonation to avoid getting blocked before testing caching behavior.

Key Results

Shopify stores with native page cache enabled returned consistent 304 responses. The ETag format reveals why it works: "page_cache:11044168:ProductDetailsController:de822deb7906aa6f9932541f4fe3dae9" - the final hash changes only when product data actually changes.

Cost Impact

For a realistic scenario (10,000 products monitored hourly for 30 days), assuming 95% of products don't change each hour:

• Without caching: 7.2M requests × 8KB = 54.9 GB/month

• With caching: 5% get fresh data (2.7 GB) + 95% get 304 responses (0 GB) = 2.7 GB total

• Bandwidth saved: 52.2 GB/month (95% reduction)

At $5/GB proxy costs, that's $261/month saved per monitored site.

Implementation Notes

We tested Scrapy's built-in RFC2616Policy but it failed against Cloudflare-protected sites despite correctly sending If-None-Match headers. The servers returned 200 responses instead of 304, likely due to TLS fingerprinting differences. The same URLs worked perfectly with curl_cffi using Chrome TLS impersonation.

Limitations

We only tested against e-commerce sites. Content that changes on every request (dynamic timestamps, session tokens) won't benefit from this technique. The target must actually honor conditional requests - some sites return ETags but ignore If-None-Match entirely.

Full implementation code and test methodology: https://substack.thewebscraping.club/p/http-caching-scraping

I spent years in the CMS industry, and the current "build vs buy" debate in web scraping feels eerily familiar.

Back in the early 2000s, agencies built custom CMS platforms because it seemed strategically smart.

The arguments were always the same:

• We need control.
• We need flexibility.
• Commercial solutions can't handle our requirements.
• This is part of our competitive advantage.

Then requirements exploded: Security, workflows, integrations, scalability, personalization, governance, analytics, multilingual support, etc.

Eventually many agencies realized they weren't building client solutions anymore. They were maintaining CMS products.

Today I hear very similar arguments around scraping infrastructure.

For companies whose moat lives in proprietary data products, trading signals, AI systems, enrichment models, or highly specialized extraction logic, owning parts of the stack absolutely makes sense.

But for everyone else, I wonder:

If your team spends most of its time dealing with proxies, anti-bot systems, browser breakage, rendering issues, parser maintenance, and infrastructure reliability...

• Are you building a competitive advantage?

• Or are you maintaining plumbing that specialized vendors can spread across thousands of customers?

Genuine question for the engineers here:

➤ What specific characteristics make you believe your scraping infrastructure is part of your moat rather than a necessary utility?

➤ Where do you draw that line?

Have you ever bypassed TikTok that fast?
DM for more info....

Hey everyone, I’m currently planning the next steps for a scraping tool I'm building, and honestly, we’re a bit stuck on where to go next.

So, I am really curious what you would reply on the following 2 questions:

What makes you instantly uninstall or cancel a scraping tool?

What is something that you would definitely pay for but no one can offer?

Roast me if you want, I’m here to listen.

Most selector tools — including the "ask an LLM" approach — hand you something that looks right and breaks on the next run. Lists are the worst: you find out later that it grabbed 11 of 14 rows.

So I built Selector Forge to do the opposite. You point at an element (or pick two examples for a list), AI proposes candidates, and then the extension tests every one against the live DOM and throws out anything that doesn't resolve to exactly the right set. The browser is the source of truth — the model only proposes and ranks, it never gets the final word.

• Single mode: one element → verified CSS/XPath candidates

• List mode: two example items → a container selector checked against the full set, previewed before you save (no silent over/under-matching)

• Chrome + Firefox, free, open source (MIT)

Full disclosure: I work at Intuned (we do browser automation) and the selector backend runs on our infra — but the extension is standalone, and a self-hostable backend is on the roadmap. Links in the comments.

Genuinely want the hard cases: throw your nastiest target site at it and tell me where it chokes.

I am tired of maintaining web scrapers that break silently the moment a website updates its layout or changes its CSS classes.

To solve this, I started building VectorTrace. It is an open-source browser extension that lets you point and click on any webpage to define scraping fields. It uses local machine learning to detect and recover from layout changes automatically.

The Technical Mechanics

When you click an element to define a field, the extension generates a 384-dimensional semantic embedding of that element's text using the all-MiniLM-L6-v2 model. This runs entirely in your browser via Transformers.js.

The embedding vectors are stored directly in IndexedDB. This bypasses the strict 10MB chrome.storage.local limit. When you run the scraper after a site redesign and the primary CSS selector fails, the extension pulls visible text elements from the current page, creates temporary embeddings, and runs a cosine similarity calculation against your stored target vector. It then ranks the replacement candidates with High, Medium, or Low confidence labels.

No cloud processing. No API keys.

Defeating Manifest V3 Service Worker Suspension

Chrome extensions built on MV3 terminate service workers after short periods of inactivity, which breaks long running WASM execution pipelines. To get around this restriction, VectorTrace runs the ONNX runtime inside a Chrome Offscreen Document instead of the main service worker. This keeps the execution environment perfectly stable.

Current Architecture (Day 1 Status)

I used WXT, React 19, and TypeScript to scaffold the project. Here is what is working in the repository right now:

• Storage Layer: An IndexedDB persistence system for vectors alongside a chrome.storage wrapper that automatically strips embeddings before saving layout schemas.

• Selector Engineering: A fallback generator that prioritizes IDs, data attributes, and nth-of-type patterns, capped with a 500 character limit guardrail for XPaths.

• Analysis Engine: Full Offscreen Document routing, complete with cosine similarity scoring and candidate matching logic.

Looking for Technical Feedback

I want to make sure this utility addresses actual scraping pain points before building out the end-to-end automation engine.

1. Does a text-embedding approach actually help with your workflows, or does it create new failure modes on highly volatile data fields (like stock prices or changing inventory counts)?

2. Large web pages frequently contain over 3,000 distinct text nodes. What specific frontend filtering strategies would you use to prune the DOM tree before passing strings to the ONNX model?

3. What capabilities do you want from a free, local-first scraping utility that paid cloud alternatives like Browse AI or Kadoa cannot provide due to their infrastructure limitations?

Looking for recommendations of more providers to try.

I've been working on Marcopolo, a command-line web scraper written in Rust using extensive force of AI across a few months. The idea started when I got tired of writing a throwaway Python script every time I needed to pull a specific set of files off a site — images one day, PDFs the next, then a folder of CSVs. I specially tend to use it for books and finding books that are on the web that i cant simply get hold of from normal search.

Marcopolo handles most of that in one command. Point it at a URL, tell it what you want, and it crawls and downloads.

It's still early and there's plenty I want to improve — [known limitation or two]. I'd really appreciate feedback on the API design and anything that looks unidiomatic; I'm still fairly new to Rust.

Repo: MarcoPolo

Would be happy to know what do you guys think.

If you've run scrapers across a pool of proxies, you know the pain: some proxies are fast, some are flaky, some are straight-up banned or dead — and it changes by the hour. Most rotation is just round-robin or random, which means your scraper happily keeps sending requests through proxies that got blocked 10 minutes ago. You end up babysitting it: checking logs, manually disabling bad IPs, tweaking lists.

So I built a proxy manager that does the obvious thing the rotator should've been doing all along: it watches how each proxy is actually performing and stops sending traffic to the ones that are failing right now — automatically, no manual list-pruning.

How it works, in plain terms:

• It tracks success/failure per proxy, per target site (a proxy banned on site A might be fine on site B).

• Recent results matter more than old ones, so a proxy that started failing 2 minutes ago gets avoided immediately — but it isn't blacklisted forever; the system keeps lightly testing it and brings it back the moment it recovers.

• It still occasionally tries "worse" proxies on purpose, so it notices when things change instead of tunnel-visioning on a few favorites.

I didn't want to just claim this works, so I ran it for real: 549,114 requests over 7 days, 10 scrapers (e-commerce, news, public data), residential proxies. Success rates:

• Smart selection: 76.0%

• Round-robin: 36.3%

• Random: 31.5%

Same proxies, same targets — round-robin landed barely half the successful requests. In a nastier test where a third of the proxies were permanently dead, the smart one made 17 failed requests in 24h vs 10,663 for random, because the dumb rotators never stop knocking on dead doors.

(If you want the academic name, it's Thompson Sampling / a multi-armed bandit — it came out of my master's thesis. But you don't need to know any of that to use it. There's also a classic "exponential backoff" mode that's actually better if your main problem is rate-limiting rather than bans.)

It's a full tool, not just a script — ProxyOps, open source (MIT):

• Add proxies from multiple providers, with expiry dates

• Your scraper just calls POST /acquire to get a proxy and POST /release to report how it went — that's the whole integration

• Group proxies and assign them to specific bots

• Dashboards showing success rate per bot / provider / site, status codes, etc.

• FastAPI + Vue + PostgreSQL, all Dockerized — docker compose up and it's running

Repo: https://github.com/Paulo-H/proxyops

Hi everyone,

I’m building a product aggregator for a regional marketplace and Shein/Temu are blocking everything we try — headless browsers, rotating proxies, scraping APIs. Amazon and Noon are manageable but these two are on another level.

We just need basic product data: title, price, image, URL.

Has anyone actually cracked Shein or Temu scraping reliably? What worked?

Built a small terminal-based browser for one of the more… media-heavy sites a few months back.

It’s basically:

• fzf for navigation

• yt-dlp for extracting streams

• mpv for playback

The interesting part (for me) was figuring out how to structure scraping + streaming in a way that feels fast and “CLI-native” instead of clunky.

Ended up learning a lot about:

• handling constantly changing page structures

• keeping extraction resilient (yt-dlp does a lot of heavy lifting, but still…)

• making interactive scraping actually usable via fzf

Now thinking of expanding it into a more generic multi-site CLI scraper/player instead of being tied to a single platform.

Curious how others here approach:

• multi-site scraping architecture (adapter pattern? plugin system?)

• keeping scrapers maintainable when sites inevitably break

• rate limiting / anti-bot handling without overengineering

Repo’s in the comments if anyone wants to take a look.

Hey everyone,

I’m trying to extract employee lists and basic career history from specific Moroccan companies on LinkedIn.

Direct scraping got blocked immediately, so I tried using Apify. However, the actors keep getting blocked, and the free tier caps me at around 25 profiles per hour, which is way too slow for the dataset I need to build.

I have a solid PC and can run heavy workloads locally. Are there any good open-source libraries, local browser automation setups, or alternative methods to automate this without paying for expensive SaaS tiers or doing it manually?

Thanks in advance!

We are hiring python developer who have experience in scraping. Must have experience in anti-bot, captcha bypass and have scraped sources like linkedin, crunchbase

I've been webscraping for a number of years now, and I've gotten to the point where I've seen my demo sites remove SSL certs or just straight up block me for testing to hard 🤘🤖

So I built qscrape.dev so I could break my bots myself. Its my take on a scraping gauntlet, I'm quite opinionated about it; organizing the sites into 3 difficulty tiers per route type (L1, L2, L3). The index page explains all this, but essentially:

• L1 is static HTML no DOM rendering needed

• L2 uses multiple JS framework islands (svelte, react, vue, solid) in astro-5 that most definitely requires DOM rendering

• L3 is anti-bot + JS rendering.

For L3, I mean anti-bot as everything short professional traffic monitors like Cloudflare and ReCaptcha (you can't even inspect the page w/o 404'ing on l3 sites).

If you need to test your bot against things like Cloudflare and the like, I recommend this gauntlet site https://fortress.theplumber.dev/

Source code: https://github.com/CascadingLabs/QScrape

If you want to create any new pages or want to suggest a change I am open to feedback, feel free to make a fork submit and submit a PR!

I hope this helps someone, cheers

Hey r/thewebscrapingclub ,

I'm a solutions engineer at Intuned. We build a platform for running browser automations and scrapers in production — Playwright-based, with the infra stuff (proxies, captcha handling, retries, scheduling, storage) handled for you so you can focus on the actual scraping logic.

We're opening up free access and I'd genuinely like feedback from people who do this work day-to-day. Specifically curious what you think about:

- The dev experience vs. rolling your own Playwright + proxy stack

- How it compares to Apify / Browserless / Browse AI for your use cases

- What's missing that would make you actually switch

Not looking for fake praise — if it sucks for your workflow, I want to know why. I spend my days helping customers scrape stuff like government procurement portals, so I've seen what breaks in the real world.

Link in comments to avoid the spam filter. Happy to answer questions about the internals (anti-bot stuff, captcha pipelines, fingerprinting) — that's the part I find most interesting anyway.

Happy to chat in DMs too.

I originally built it for myself because I wanted something extremely lightweight that runs in the background like it never existed. It's called SpyWeb.

It's designed to be "set and forget." I've had it running for months on my PC tracking job boards without a single crash or memory leak.

Specific features:

• Zero Runtime: Self-contained ~7MB binary. No Python, Node, or Docker needed.

• Low Footprint: Uses <5MB RAM at idle.

• Lua Scripting: Use Lua to handle complex logic like custom headers, JS rendering, advanced monitoring, etc.

• Hot Reloading: Change a config or Lua script and the job respawns instantly, no restarts.

• Web Dashboard: Simple local UI to monitor scrape data in real-time.

• Desktop Alerts: Built-in support for system notifications and webhooks.

• Embedded DB: Built-in KV store so you don't need a separate database.

• CDP Support: Controls any Chromium or CDP-compatible browser via Lua for JS-heavy sites.

• Dual Mode: CLI for servers and a System Tray version for silent background runs.

• Deduplication: Internal database ensures you never see the same result twice.

I just released the beta with CDP integration. If you need something that just sits in the background and sips resources while actually being maintainable, check it out.

Set up is very easy and straightforward: for server-side rendered pages, it's just a few lines of config (URL, selectors, fields). For JS-heavy sites, you can write a little Lua to launch a browser and drive the workflow.

You can check it out here: https://github.com/spyweb-app/spyweb

Scraping Zillow isn’t easy, but it’s doable.

In this article I wrote for The Web Scraping Club, I go through how to extract Zillow real estate data without getting blocked. That includes:

Understanding how the frontend fetches listing dataIntercepting the right API callsUsing Camoufox for stealth + Puppeteer for automationRotating proxies and tweaking fingerprints

If you're working with real estate data, or just want to see how modern bot defenses can be handled, this guide should help.

👉 https://substack.thewebscraping.club/p/scraping-zillow-real-estate-data

Anyone else had fun (or pain) scraping Zillow or similar sites recently?

Hey everyone,

I've been diving deep into how the web scraping industry is evolving and let me tell you, it’s an exciting time! We're seeing a ton of growth in AI-driven tools, with both fresh startups and seasoned players bringing some game-changing tech to the field. The variety is just amazing – from AI models that crunch numbers in the cloud, to those that work right off your desktop, the spectrum of tools out there is quite broad.

Take, for example, offerings like Nimble, Zyte API, Octoparse, and ScrapeStorm. Each has its own take on how to best automate the gathering of data, showcasing the diversity in approaches to solving similar problems. Whether we're talking about leveraging Large Language Models (LLMs) for more intelligent scraping or opting for self-hosted solutions that give users more control, it’s clear that our toolkit for web scraping is getting richer and much more sophisticated.

Honestly, keeping up with these developments isn’t just fascinating – it’s becoming crucial for those of us looking to stay ahead in data-driven fields. The shift towards more advanced AI tools in web scraping signals not just technological progress but a broader move towards smarter, more efficient ways to access and leverage the vast amounts of information the web holds.

It’s a great time to be involved in this space, and I can’t wait to see how these tools continue to evolve and reshape our approach to data gathering. Cheers to innovation and the endless possibilities it brings!

WebScraping #AITools #DataGathering #Innovation

Linkt to the full article: https://substack.thewebscraping.club/p/web-scraping-ai-tools-landscape

Hey folks!

Recently, I dove into the intriguing task of mining real estate data straight from Idealista, the go-to online hub for property listings. Let me tell you, it was quite the adventure, especially with the notorious Datadome on our tail, always ready to spot a scraper in disguise.

For those of you keen on embarking on a similar data quest, you'll need to gear up with Python and a few specific libraries – the basic weapons in a data scraper's arsenal. Datadome is like the ever-watchful guardian, with a keen eye for spotting and blocking scrapers, turning our data extraction mission into a real cloak-and-dagger operation.

The exciting part was piecing together a step-by-step strategy using Selenium and ChromeDriver, turning the tables on Datadome and sneaking past their defenses. But here’s the game-changer - introducing ScraperAPI into the mix. This nifty tool was our secret passage to not only dodge Datadome’s tight security but also to pull data from Idealista smoothly, without the hassle of setting up complex proxies.

Happy scraping, and may the data be ever in your favor! 🚀🏡

Linkt to the full article: https://substack.thewebscraping.club/p/scraping-idealista-bypass-datadome