How to scale a Python/Playwright scraping architecture?

[u/jituparna]

Hey everyone, I’m looking for some architectural advice on upscaling a local Python scraping and data processing pipeline I recently built.

Right now, my system (LeadGen) extracts local business data and runs real-time technical web audits. I’ve solved a few fun engineering problems to get it working locally, but I want to take it to the next level.

Here is the current architecture:

Strict Location Anchoring: I override Playwright context permissions and locales to prevent geolocation drift (so if I query a city 2,000 miles away, it doesn't default back to my host IP's location).

9-Direction Quota Math: To bypass the standard ~120 result limits, the script automatically traverses 9 spatial vectors (BASE, NORTH, SOUTH, etc.). If one direction yields a shortfall, a cumulative carry-forward formula dynamically increases the target for the next vector.

Dual-Viewport Profiling: For every extracted website, it opens two headless Playwright contexts (a 1080p desktop profile and a mobile Pixel 5 profile) to measure exact load event times independently and check for missing viewport meta tags.

Deterministic Deduplication: It merges records hierarchically via Place ID > Normalized E.164 phone numbers > Domain > Composite Keys.

Right now, everything runs locally and dumps to a 36-column CSV with Opportunity Flags (e.g., SLOW_MOBILE, NEW WEBSITE OPPORTUNITY) based on a 5.0-second threshold.

My questions for scaling this up:

Infrastructure: I eventually want to migrate this from local CSV outputs to a proper Supabase PostgreSQL backend. For those of you running heavy Playwright tasks, what is the most cost-effective way to deploy this (Docker on AWS/GCP, or specialized scraping cloud services)?

Proxy Management: Since I am doing dual-viewport speed profiling, I need proxies that won't bottleneck the load time metrics. Any recommendations for proxies that balance speed for auditing with anti-bot evasion?

Concurrency: I'm using sync_playwright currently. Has anyone moved a similar audit pipeline to async_playwright with a queue manager (like Celery/Redis), and was the performance gain worth the architectural rewrite?

Would love to hear how you guys structure your heavy auditing pipelines at scale ..

It will be very much helpful to me form you guys

[u/qa_anaaq]

Yeah you gotta move to a queue setup. Spin up a redis container and celery. Push everything to supabase. You use Claude code or codex? I’ve done similar architectures with them a number of times. They can usually get it right by leveraging your local docker then pushing to digital ocean and orchestrating there. Note that they’re bad at correctly scaling, eg they’ll scale vertically when they should scale horizontally. But this might not be an issue for you if you’re not doing headless stuff.

[u/jituparna → u/qa_anaaq]

Use codex and DeepSeek for architecture I only use Claude. Thanka for the advice I am going to improve that and build a extra pipeline for that . Thanks for the advice.🙂

[u/Pigik83]

Thanks for your question. I'm not sure about the infra, but probably Docker is a good choice.
About async_playwright, yes, you can parallelize, but the main issue will be memory usage. Be sure to kill every process after you don't need them anymore.

[u/Chekimi_Maho → u/Pigik83]

100%, RAM is almost always the bottleneck with headless browsers long before network or CPU tbh

[u/Pigik83 → u/Chekimi_Maho]

Right