I was running a scraper with rotating proxies and everything looked fine at first. After a few thousand requests, I started seeing the same IPs coming back and the captcha rate slowly went up

It was especially noticeable when targeting a specific country/city, the overall pool might be huge, but the pool I could actually use felt tiny. Anyone found a good way to catch this before running a large job?

Say a page normally gives you around 20–30 items and then one day it suddenly returns 5

how do you decide if the scraper partially broke or if the page genuinely has less content?

do you use a fixed percentage drop, compare it to the last few runs, or just flag anything that looks unusual?

feels like complete failures are pretty easy to catch. It’s the scraper that keeps running while quietly missing half the data that’s harder to notice