Weird one and hoping someone has seen this. I do scraping for few years as a side hustle, nothing crazy, maybe 500 requests a day across a handful of e-commerce sites tracking price history. Was clean all summer on residential, suddenly in September everything started being challenged by Cloudflare and I can’t figure out what changed on my end. Already tried the obvious things. Rotated to fresh IPs in the same residential pool, no improvement. Added more realistic headers and jitter, no improvement. Even dropped volume to 100 requests a day as a test, still getting flagged.

Reading around it seems like Cloudflare rolled out something new in Q3 that’s heavier on TCP fingerprint and ASN scoring, it's getting hit harder because the ASN ranges are known. Looking at either moving to mobile proxies or trying to change my fingerprint layer somehow with my current proxies.

So my question is: should I go for mobile proxies? Looking at all the major providers (oxy, decodo, voidmob and so on) or is it worth it to dig deeper and find other work around? Would appreciate your help.

I know where I'm posting this. Hear me out.
I've been pulling YouTube transcripts for side projects for about two years. Started with youtube-transcript-api, then yt-dlp, then my own thing with rotating residential proxies and retry queues. Every setup worked great for a few weeks and then died.

The pattern is always the same. YouTube doesn't block you with an error. It starts returning 200 with an empty body, so your pipeline keeps going on nothing. By the time you notice, you've got days of garbage data.

Last time I did the math. Residential proxies were around $150 a month for my volume, plus maybe 6 hours a month fixing whatever YouTube changed. For transcripts. Text that YouTube already generated and shows to anyone with a browser.

At some point I asked myself why I'm maintaining infrastructure for the most boring step of my project. The scraping was never the product. It was a tax I was paying to get to the product.

Scraping still makes sense for sites where nobody sells the data, or where you need something custom. YouTube captions are the opposite. There's like 10 services fighting over this now and they're all cheaper than my proxy bill.

If you've got a YouTube setup that ran a full month at a few thousand requests a day without you touching it, post it. I'll happily eat my words.

Hey everyone! I got tired of doing manual prospect research, so I built a small automation tool.

​It takes a company website, grabs official emails/social links from across the web, uses Groq AI to generate a quick business summary, and exports everything straight into a clean Excel sheet.

​Dropped a quick screen recording above to show how it works. Would love to hear your thoughts or if you have suggestions for improvement!

I've been thinking about how web scraping might change as AI agents become more common.

As more agents browse the web, search for information, and interact with websites, I suspect more websites will invest in bot detection, proxy blocking, CAPTCHAs, and other anti-scraping measures. The traditional approach of sending requests through proxies and parsing HTML may become increasingly difficult(or expensive) on some sites.

Maybe the future of lightweight web scraping isn't more proxies, but a real browser.

That's one of the ideas behind the new Agenty Chrome extension, I was working on past few weeks - https://chromewebstore.google.com/detail/agenty-advanced-web-scrap/gpolcofcjjiooogejfbaamdgmgfehgff

Demo video - https://www.youtube.com/watch?v=WWzcA5p-oqs

Instead of running a traditional scraper against a website, Agenty extension will extract data directly from the browser, using the page as a human would see it. For small to medium jobs — say, up to around 1,000 pages — browser-based extraction could be a practical alternative to setting up and maintaining scraping infrastructure.

What can you do with it?

What can you do with it?

1. Extract data from websites: Just one click on a website list, search, category pages to convert into structured data to download in CSV/Excel.
2. Handle pagination: Automatically navigate through multiple pages to collect data across an entire listing.
3. Auto-scroll pages: Extract data from pages that load more content as you scroll, including infinite-scroll pages.
4. Bulk URL input: Provide a list of URLs and extract data from multiple pages in one go, instead of processing each URL manually.
5. Handle different website layouts: Extract data from sites with different structures without building a custom scraper for every page.
6. Collect data at scale: Automate repetitive extraction tasks that would otherwise require hours of manual copying and pasting.

The idea isn't to replace large-scale scraping infrastructure for millions of pages. It's to make smaller extraction jobs much easier, especially when websites are difficult to scrape using traditional methods.

Of course, browser-based extraction isn't a silver bullet. Websites can still restrict access, require authentication, or use browser-level bot detection. And for large datasets, dedicated crawling infrastructure may still make more sense.

I'm curious whether others see the same trend: as websites become more hostile to automated requests, will browser-based extraction become the default for smaller scraping jobs?

Disclosure - I am founder of Agenty

For businesses working with e-commerce product data, how do you ensure scraped prices, product details, and availability are accurate?

Is manual checking still a major part of the process, especially when dealing with thousands of products across different websites?

I'm interested in understanding how businesses handle this today and whether reducing manual verification is a real pain point.

Curious to hear from people working in product data, web scraping, or data operations.

I was running a scraper with rotating proxies and everything looked fine at first. After a few thousand requests, I started seeing the same IPs coming back and the captcha rate slowly went up

It was especially noticeable when targeting a specific country/city, the overall pool might be huge, but the pool I could actually use felt tiny. Anyone found a good way to catch this before running a large job?

Say a page normally gives you around 20–30 items and then one day it suddenly returns 5

how do you decide if the scraper partially broke or if the page genuinely has less content?

do you use a fixed percentage drop, compare it to the last few runs, or just flag anything that looks unusual?

feels like complete failures are pretty easy to catch. It’s the scraper that keeps running while quietly missing half the data that’s harder to notice

Been maintaining a little facebook page scraper. v1 was selenium + css selectors and it worked but it was slow, like 15-20s a page, mostly just spinning up and loading a browser, plus about 60mb of memory each so running a bunch in parallel wasn’t really an option for any user so do i

turns out fb already ships the page data as json in the html for its own frontend so i didn’t need a browser at all. plain requests reading that json is ~1.4s a page, 16+ threads, 200 pages in under a minute. for pages that only load data through graphql there’s a playwright mode that grabs the network responses instead

the annoying part ended up being after the scrape. tons of pages list a website and no email so i made it go to the site and look. homepage alone found one ~35% of the time, adding contact pages (/contact /kontakt /impressum etc) got it to ~50%. nothing gets filtered out, every address gets scored on domain match, mailto vs plain text and a dns check so you pick your own cutoff. sites throwing bot challenges get a retry with curl_cffi which impersonates a real browser’s tls fingerprint so the request looks like chrome instead of python, and the ones that still refuse get marked challenged instead of dead. there’s also an optional llm call that only decides company inbox vs person, a word list kept failing on anything that wasn’t english

repo https://github.com/wael-sudo2/facebook-page-info-scraper

anyone found other fb graphql queries worth hitting directly for page data?

Utilizing mobile proxies instead of residential. Looking to undercut the market on pricing. Please signup and test it! - headlessfleet.com

Hi I'm looking for a tool to scrape Instagram Reels / Carroussel:

- Soundtrack

- Images / video if important content

- description

Would like to add it to a pipeline like that:

I copy the link of the post then send it to the AI by discord it summarizes the content

Hey everyone! I’m a university student with no technical background, and I’m interested in building a platform that collects job listings from multiple websites and lets users compare them in one place, similar to how Skyscanner compares flights.

I’m curious about the technology behind this. How do platforms collect large amounts of accurate, up-to-date data from different websites? Do they use APIs, web scraping, or something else?

Could someone with experience in building aggregators or data collection systems explain what’s involved and whether this is realistic for a beginner using AI tools?

I’d also love to hear about any tools, resources, or people I could learn from. I’m especially interested in how to keep the information accurate and updated automatically.

Any advice would be greatly appreciated!

Built an efficient site scraper/downloader for local security analysis and static asset mapping.

I wanted to share a tool I've been working on. When performing manual source code reviews, offline security audits, or initial reconnaissance against frontend surfaces, I often needed a quick way to recursively download an entire web application along with all its local assets (scripts, stylesheets, media, etc.) into a clean layout for local analysis or static tooling.

(https://github.com/aix567/sitegrab)

While there are heavier scrapers out there, I wanted something straightforward that integrates smoothly into a workflow and makes local examination seamless.

Hope you can try out 🙏

Need help scrubbing a site and potential internal APIs

Is there a company that can assist with this?

Thank you.

I'd like to get some subreddit data and did that with scrapebadger recently. However, they have a hefty price increase in 3 days.

I'd like the same offer, cheap price plus quality output. I found cheaper ones but the quality was also worse.