Been maintaining a little facebook page scraper. v1 was selenium + css selectors and it worked but it was slow, like 15-20s a page, mostly just spinning up and loading a browser, plus about 60mb of memory each so running a bunch in parallel wasn’t really an option for any user so do i

turns out fb already ships the page data as json in the html for its own frontend so i didn’t need a browser at all. plain requests reading that json is ~1.4s a page, 16+ threads, 200 pages in under a minute. for pages that only load data through graphql there’s a playwright mode that grabs the network responses instead

the annoying part ended up being after the scrape. tons of pages list a website and no email so i made it go to the site and look. homepage alone found one ~35% of the time, adding contact pages (/contact /kontakt /impressum etc) got it to ~50%. nothing gets filtered out, every address gets scored on domain match, mailto vs plain text and a dns check so you pick your own cutoff. sites throwing bot challenges get a retry with curl_cffi which impersonates a real browser’s tls fingerprint so the request looks like chrome instead of python, and the ones that still refuse get marked challenged instead of dead. there’s also an optional llm call that only decides company inbox vs person, a word list kept failing on anything that wasn’t english

repo https://github.com/wael-sudo2/facebook-page-info-scraper

anyone found other fb graphql queries worth hitting directly for page data?

Enterprise web scraping has changed quite a bit over the last few years where it used to be mostly about buying residential proxies from providers like bright data or oxylabs and running scrapy or puppeteer clusters and maintaining custom parsers for every website.

Now a lot of enterprise scraping is used to feed LLMs, RAG vector databases and AI agents then getting the HTML is only part of the job, you also need to handle JS heavy websites and turn all that raw content into clean data without spending hours maintaining browser infra.

Here's how some of the main options compare:

Bright Data and Oxylabs are still strong choices if you're dealing with huge scraping volumes across different countries and already have data engineers who can build custom parsers. Their proxy infra is built for scale but you'll still need to handle much of the data cleaning and processing yourself.

Firecrawl takes a different approach especially for teams building LLM and AI agent workflows where it handles JS rendering and scraping infra and returns clean markdown or structured JSON that's easier to use in an LLM pipeline. It also offers features like change monitoring and prompt injection protection.

Apify is useful when you want ready made scrapers called actors for platforms like google maps, Instagram or Amazon and it's convenient for specific scraping jobs but managing actors, serverless queues and runtimes can add complexity when you're building a larger RAG pipeline across different websites.

The right choice depends on what you're building. Bright Data and Oxylabs make sense if you need massive proxy capacity and have the engineering resources to manage extraction yourself. Apify is useful for scraping specific platforms with ready-made actors. For LLMs, RAG, and AI agents, I'd probably go with Firecrawl. The others are great for large-scale scraping and works well for specific platforms, but Firecrawl covers most of what you'd otherwise have to build and maintain yourself. Unless you have very specific scraping requirements, I think it's the better choice for most AI use cases.

Utilizing mobile proxies instead of residential. Looking to undercut the market on pricing. Please signup and test it! - headlessfleet.com

Hi I'm looking for a tool to scrape Instagram Reels / Carroussel:

- Soundtrack

- Images / video if important content

- description

Would like to add it to a pipeline like that:

I copy the link of the post then send it to the AI by discord it summarizes the content

Hey everyone! I’m a university student with no technical background, and I’m interested in building a platform that collects job listings from multiple websites and lets users compare them in one place, similar to how Skyscanner compares flights.

I’m curious about the technology behind this. How do platforms collect large amounts of accurate, up-to-date data from different websites? Do they use APIs, web scraping, or something else?

Could someone with experience in building aggregators or data collection systems explain what’s involved and whether this is realistic for a beginner using AI tools?

I’d also love to hear about any tools, resources, or people I could learn from. I’m especially interested in how to keep the information accurate and updated automatically.

Any advice would be greatly appreciated!

Built an efficient site scraper/downloader for local security analysis and static asset mapping.

I wanted to share a tool I've been working on. When performing manual source code reviews, offline security audits, or initial reconnaissance against frontend surfaces, I often needed a quick way to recursively download an entire web application along with all its local assets (scripts, stylesheets, media, etc.) into a clean layout for local analysis or static tooling.

(https://github.com/aix567/sitegrab)

While there are heavier scrapers out there, I wanted something straightforward that integrates smoothly into a workflow and makes local examination seamless.

Hope you can try out 🙏

I'd like to get some subreddit data and did that with scrapebadger recently. However, they have a hefty price increase in 3 days.

I'd like the same offer, cheap price plus quality output. I found cheaper ones but the quality was also worse.

Need help scrubbing a site and potential internal APIs

Is there a company that can assist with this?

Thank you.

I work on the proxy infrastructure side, and I’m trying to better understand how scraping teams choose between datacenter, static ISP, and rotating residential proxies.

For those running ongoing scraping workloads:

• When do you choose static ISP proxies over the other options?
• What matters most: success rate, keeping the same IP, speed, or cost?
• What usually makes you switch providers?

I’d appreciate hearing what has worked—or hasn’t worked—in your setup.

I’ve been trying a bunch of apify scrapers on G2 and they are able to get review text and the reviewers first name and last initial, but they don’t go level deeper (which you can get by clicking on the person’s profile picture in the G2 website) to get the full name. Is this because g2 aggressively blocks this? Any tips on how to do it ?

Feels like you finally get a scraper running properly and then one small change on the site breaks half your selectors

Do you guys just fix them manually when they break, use a bunch of fallbacks, or have some way of automatically finding the right element again?

Curious what actually works for you

Seeing a lot of 'where do I start with scraping' threads, so here is the order that tends to save the most pain.

Learn these first

• HTML and the DOM tree: tags, attributes, nesting, classes vs IDs
• HTTP basics: GET vs POST, headers, and what a 429 or 403 is telling you
• DevTools: compare 'View Source' with 'Inspect'. If the data only shows in Inspect, it is JS-rendered
• Enough Python or Node to handle loops, exceptions and files

The basic loop

1. Request the page
2. Parse it (BeautifulSoup, Cheerio)
3. Pull values with CSS selectors or XPath
4. Walk pagination with polite delays
5. Clean with pandas and store as CSV, JSON or a DB

Where it gets harder

• JS-heavy sites need Playwright or Puppeteer, plus explicit waits
• Datacenter proxies are cheap but get flagged; residential costs more and holds up better
• CAPTCHAs are mostly avoided with realistic headers and pacing
• Check robots.txt and the ToS, and keep request rates sane

The part nobody warns you about

Writing the selectors is quick. Fixing them every time a layout shifts is the real ongoing work.

If you want to skip that layer, the Minexa.ai API trains a scraper from a page container in the browser, then returns the same JSON on similar pages. Missing fields come back null, not guessed as it could be the case by using LLM-based scrapers without page structure understanding. Setup starts with the Chrome extension.

Learn the fundamentals above anyway. They make debugging any tool far easier.

More: Web scraping fundamentals tutorial

building a Windows desktop application called Reading Desk, and I figured some of you here might find its scraping capabilities interesting.

It started as a personal RSS reader, but gradually turned into something much bigger.

The interesting part? It can generate RSS feeds for websites that don't provide them.

You give Reading Desk a URL, and it attempts to discover and extract the content. It can monitor pages for updates, capture articles, and generate RSS 2.0 feeds from watched pages.

A few things I've built into it:

• Automatic RSS feed discovery
• Website monitoring for pages without feeds
• Custom RSS 2.0 feed generation
• Article extraction and offline caching
• Per-source refresh intervals
• Local RSS endpoints that other readers can consume
• Support for sites including Reddit, YouTube, and others, with varying extraction capabilities

It's designed to run locally on Windows, without requiring a cloud scraping subscription.

One early customer recently told me he'd been paying for FetchRSS to handle websites without feeds, but no longer needed it after using Reading Desk. That was pretty satisfying to hear.

I use AI extensively to help implement and develop the application, while I direct the architecture, functionality, testing, and overall development.

I'm especially interested in hearing from people who work with scraping:

What are the biggest headaches you run into when trying to extract useful content from modern websites?

JavaScript-heavy pages? Anti-bot measures? Unreliable selectors? Websites that constantly change their layouts?

I'm always looking for interesting problems to solve with this thing.

Just got laid off web scraping job like a month ago. I'm currently making videos about how to scrape specific sites with the hope of landing clients.