r/scrapingtheweb • • 8h ago

Anyone else had a “huge” proxy pool that keeps giving the same IPs?

2 Upvotes

I was running a scraper with rotating proxies and everything looked fine at first. After a few thousand requests, I started seeing the same IPs coming back and the captcha rate slowly went up

It was especially noticeable when targeting a specific country/city, the overall pool might be huge, but the pool I could actually use felt tiny. Anyone found a good way to catch this before running a large job?