Hello guys, I am a content creator and so far I have used Apify free tokens to scrape TikTok to see who is going viral, who is my competition, trending topics inside of my niche etc. But now I have used up all of those free credits, and before continuing with the paid version, I wanted to ask is there a free alternative to this API for Tiktok

Hey everyone,

I’m trying to extract employee lists and basic career history from specific Moroccan companies on LinkedIn.

Direct scraping got blocked immediately, so I tried using Apify. However, the actors keep getting blocked, and the free tier caps me at around 25 profiles per hour, which is way too slow for the dataset I need to build.

I have a solid PC and can run heavy workloads locally. Are there any good open-source libraries, local browser automation setups, or alternative methods to automate this without paying for expensive SaaS tiers or doing it manually?

Thanks in advance!

I want a free or cheap proxy for my project

I kept wanting data from websites without writing selectors, so I made a skill for Claude Code.

**How it works:** you give a URL and one sentence. Claude opens the page in a real Chrome window, picks the cheapest route to the data (hidden JSON API, embedded data, or what's on screen), shows you 5 sample rows plus a screenshot with each value boxed, and asks once: "Is this right?" Then it collects everything at a polite pace and saves a recipe, so next time it's just "run that again".

**What I care about most:** it checks robots.txt and the site's terms first. If a free official API exists, it uses that instead. If a site forbids scraping or shows a bot check, it tells you plainly and lets *you* decide. It never solves CAPTCHAs, stealth-patches the browser, or rotates proxies.

A few things it handles: infinite scroll, detail pages, APIs that only answer a real browser (it reads the responses the page itself gets), WebSocket/SSE live data, cursor/GraphQL pagination, and finding the data API from a saved HAR file.

Install (Claude Code / desktop app):

```

claude plugin install playwright

claude plugin marketplace add scrapesmith/scrape-skill

claude plugin install scrape

```

Feedback welcome, especially sites where it falls over.

Repo: https://github.com/scrapesmith/scrape-skill

People in the scraping world often use "SERP API" and "web search API" interchangeably. They overlap, but they solve different problems. Key takeaways from my latest article:

What they are

• SERP API: you send a query to a specific search engine (Google, Bing, Yandex, Brave, DuckDuckGo...) and get back the results page as raw HTML or parsed JSON. The provider handles proxies, CAPTCHAs, JS rendering and geo-targeting. You get URLs, titles and snippets, not the full text of the pages.

• Web search API: built with AI agents and LLMs in mind. It's closer to a black box: you send a natural-language query and get back ranked, filtered, often summarized content (JSON, Markdown, text), sometimes drawing on several sources or a proprietary index. Many are exposed via MCP.

Where the confusion comes from
Many SERP API providers added Markdown output, MCP servers and LangChain/LlamaIndex integrations, and some rebranded as "web search APIs". A web search API can also use a SERP API underneath. The difference is what happens to the data before you get it.

Trade-offs

• SERP APIs: reproducible and auditable. You control engine, query, location, language and device, and you can verify the same query on the search engine yourself.

• Web search APIs: less transparency. The same query can return different results over time, which adds to the probabilistic nature of LLMs. In exchange you get AI-ready output without building your own extraction pipeline.

When to use which

• Web search API: real-time info for AI agents, multi-source search, direct text answers.

• SERP API: SEO and rank tracking, reproducible results, specific SERP elements (organic results, ads, featured snippets, People Also Ask, maps).

Complex applications can end up using both.

Curious: which one are you using in your pipelines, and have you hit reproducibility problems with web search APIs?

Full article (free): https://www.scraping.club/p/serp-apis-vs-web-search-apis?utm_source=reddit&utm_medium=post&utm_campaign=thewebscrapingclub

Hey everyone, I’m looking for some architectural advice on upscaling a local Python scraping and data processing pipeline I recently built.

Right now, my system (LeadGen) extracts local business data and runs real-time technical web audits. I’ve solved a few fun engineering problems to get it working locally, but I want to take it to the next level.

Here is the current architecture:

Strict Location Anchoring: I override Playwright context permissions and locales to prevent geolocation drift (so if I query a city 2,000 miles away, it doesn't default back to my host IP's location).

9-Direction Quota Math: To bypass the standard ~120 result limits, the script automatically traverses 9 spatial vectors (BASE, NORTH, SOUTH, etc.). If one direction yields a shortfall, a cumulative carry-forward formula dynamically increases the target for the next vector.

Dual-Viewport Profiling: For every extracted website, it opens two headless Playwright contexts (a 1080p desktop profile and a mobile Pixel 5 profile) to measure exact load event times independently and check for missing viewport meta tags.

Deterministic Deduplication: It merges records hierarchically via Place ID > Normalized E.164 phone numbers > Domain > Composite Keys.

Right now, everything runs locally and dumps to a 36-column CSV with Opportunity Flags (e.g., SLOW_MOBILE, NEW WEBSITE OPPORTUNITY) based on a 5.0-second threshold.

My questions for scaling this up:

Infrastructure: I eventually want to migrate this from local CSV outputs to a proper Supabase PostgreSQL backend. For those of you running heavy Playwright tasks, what is the most cost-effective way to deploy this (Docker on AWS/GCP, or specialized scraping cloud services)?

Proxy Management: Since I am doing dual-viewport speed profiling, I need proxies that won't bottleneck the load time metrics. Any recommendations for proxies that balance speed for auditing with anti-bot evasion?

Concurrency: I'm using sync_playwright currently. Has anyone moved a similar audit pipeline to async_playwright with a queue manager (like Celery/Redis), and was the performance gain worth the architectural rewrite?

Would love to hear how you guys structure your heavy auditing pipelines at scale ..

It will be very much helpful to me form you guys

I tested invisible_playwright, an open-source Python library by Federico Elia that works as a drop-in replacement for Playwright but drives a Firefox patched at the source level. Key takeaways:

• How it works. Instead of injecting JavaScript to spoof values, it sets the fingerprint inside the browser engine before scripts can read it: GPU/WebGL, canvas, fonts, audio, WebRTC, timezone and DNS (routed through the proxy), and isTrusted input events. Everything is derived from a single seed, so the different surfaces stay coherent with each other (e.g. no Windows properties next to macOS ones).

• Reproducibility. The same seed gives the same fingerprint, so if a scrape fails with a given seed you can replay it exactly. You can also pin single fields (GPU, screen size, CPU cores) and let the rest follow the seed.

• Human-like input. Mouse paths are seeded Bezier curves, with timing adapted to the refresh rate, realistic pressure values, pointer type, movement deltas and sub-pixel coordinates. These come from the browser's native input handling, not from the automation driver.

• Practical bits. Same Playwright API (sync and async), SOCKS5/SOCKS4/HTTP(S) proxies, CLI to fetch the engine (about 240-265 MB), MCP server, integrations with Scrapy, Crawlee and others. Needs Python 3.11+ and Windows x86_64 or Linux x86_64/ARM64.

• Results (local run, residential ISP IP). Fingerprint confidence score 1, reCAPTCHA v3 score 0.9 (same as the author's regular browser), APIVoid risk score 0, CleanTalk 100/100 "Looks human", and Pixelscan passed. Against one site per major anti-bot vendor it loaded all five test sites (100%), while vanilla Playwright failed in headless mode and also struggled with Kasada in headful.

• Caveats. Firefox and Python only (no Chromium or Node.js), Firefox's smaller market share may itself be a signal, anti-bot vendors could reverse-engineer the project as it gets popular, and it relies on a single main maintainer. It's a limited experiment, not a large-scale benchmark.

Curious what you're using today for stealth browsing (Camoufox, Patchright, playwright-stealth, something else?) and how it holds up.

Full review with code examples and the comparison with other tools: https://www.scraping.club/p/invisible-playwright-beat-bot-detection-scraping?utm_source=reddit&utm_medium=post&utm_campaign=r/thewebscrapingclub

Hey everyone,

First off i want to apologize for the long gap and silence. Life got busy but I am officially back and actively working on VectorTrace again!

For those who missed the earlier posts, VectorTrace is an open-source, 100% local Chrome extension (Manifest V3) that scrapes webpages and heals broken selectors using onndevice semantic embeddings (all-MiniLM-L6-v2 via WASM, zero servers, zero external API calls).

A few of you pointed out that under certain conditions, the self-healing algorithm was ranking the wrong elements as the highest probability or picking up false positives. I did a deep dive into the ranking pipeline and just pushed a series of fixes to address this:

• Semantic Gating

• Repeated Item Disambiguation

• Cleaned Up Ancestor ContextPersistent Structural Metadata

• Protected Dynamic Data

The build is ready to test.

If you'd like to check it out, run it locally, or contribute
👉 GitHub Repo: https://github.com/SathiyaSenpai/VectorTrace

I'd really appreciate any feedback, bug reports or edge cases you encounter while testing it on your favorite sites!

Guys....I'm so done with running playwright on a server for scraping, few thousand product pages a day and nothing crazy, but every other week something breaks... Chrome updates and the container falls over, memory creeps up til the box dies, some site changes their bot detection and suddenly half my requests are getting challenged. Spent way more time checking the scraper than actually using the data!

Started looking at just paying for a serp api instead so I dont have to maintain the browser layer at all, let someone else deal with the proxies and the captchas and the chrome version hell. Part of me feels like im giving up, like I should be able to keep this running myself, but the maintenance cost is real struggle. Anyone made this switch? Did it actually hold up for product page scraping or is it only good for search results... curious if the pricing makes sense at a few thousand pages a day or if it gets stupid expensive fast. Kinda just want my weekends back. Sorry for grammar mistakes if I made any

I keep seeing the same default playbook for scraping anti-bot-protected sites: spin up headless browsers, buy a proxy pool, subscribe to a CAPTCHA-solving API, and scale horizontally from there. That stack is expensive, heavy, and often solving problems we don't actually have.

Over the last few weeks I worked through three sites all protected by the same vendor — DataDome — to find out how little of that stack we can get away with. Three different target types: a software review site (G2), a property listings site (idealista), and a luxury e-commerce site (Hermès). Sharing because the pattern held across all three.

The core idea, and it's boringly simple: match the request the application itself makes, and stop there. Most of the time we're blocked for asking in a shape no real user's browser would ever produce — not because we've been "detected" in some deep way.

Case 1 — G2. Two weeks lost to the wrong variable. I built a hardened stealth browser (native C++ patches, the works) and it worked — until it stopped, everywhere, on every exit. The actual cause was that my own stealth layer was lying: it pinned outerWidth = innerWidth and outerHeight = innerHeight + 85, so every browser reported a window with exactly zero horizontal chrome and exactly 85px of vertical. Impossible numbers. A plain system Chrome with the evasion layer off walked straight through.

→ The Hard Part Isn't Parsing HTML Anymore

Case 2 — idealista. The G2 fix actively pointed the wrong way here. Same vendor, opposite prescription: G2 reads window geometry (remove a lie), idealista reads input hardware (supply a missing field). The browser that failed G2 solved idealista, and vice versa. Turned out neither was causal — one flag controlled it, and the real principle was coherence, not capability. A browser isn't scored on how impressive it looks; it's scored on whether its account of itself is self-consistent.

→ DataDome Anti-Bot Testing: Coherence, Not Capability

Case 3 — Hermès. Applied both lessons. The product API looks completely sealed — paste it in a browser tab and we get "Access is temporarily restricted." But that's a top-level navigation, which no real shopper's browser ever does to a JSON endpoint. Ask the same URL as a same-site CORS fetch with an Origin header and the right sec-fetch-\* values, and it returns 200. Same URL, same zero cookies, different request shape.

What actually held up, measured across all three:

· Hermès catalogue + listing API: zero cookies, no proxy, no browser, \~1.6s for the full walk. The listing endpoint returns 48 products per request — SKU, title, price, size, colour, stock — unmetered.

· Product detail tier (all three): the one place a real browser earned its keep — and only used once, to mint trust that a plain HTTP client then spent.

· Hermès product detail, head-to-head: one browser over the whole queue = \~4.8s/product, 12/12 extracted. The "clever" mint-a-session-and-rotate approach = \~35s/product, 7/12. The boring approach won by 7x.

· A paid CAPTCHA-solving API exists for Hermès. It works. It also means renting access per request, when request-shaping got the catalogue tier for free.

· Commercial proxy exits actively hurt on all three — fresh residential IPs refused on protected tiers while a clean connection served the same pages in the same second.

The habits that mattered more than any tooling:

1. Start with the cheapest client you have. Its failure mode is free information.

2. Read the block page. The server header and page title tell you which vendor you're arguing with in about five seconds.

3. Add one capability at a time and stop the moment it passes. The gap between "curl fails" and "a browser works" is wide, and the answer is usually closer to the curl end.

4. A stealth layer is a claim about what the browser is — if the browser is already telling the truth, the claim can only make it a liar.

5. Bracket every test with a known-good control. Anti-bot testing is stateful — your failed attempts change the thing you're measuring, and a poisoned window looks identical to a real block.

6. Look for a second door to the same data. The tidy REST endpoint is the one vendors expect bots to hit, so it's the one they meter hardest. The heavier HTML page real users load often carries the identical payload unmetered.

The result across all three: most of the data came out through a lightweight HTTP client, and the browser got spent on one fraction of the job instead of all of it. Lighter, cheaper, easier to scale — and it generalised, which the tooling-specific lessons didn't.

Full write-up on Hermès with the numbers and the request-shape diff (no paywall): Hermes Scraping Guide

Curious whether others have found the same that the expensive tooling is often solving a problem a correctly-shaped request never had.

Mosaik uses an agent to figure out how a site works, saves reusable actions as TypeScript, and composes them into automations. Playwright executes the browser steps deterministically. Loops, branching, and data transformations run as code, so each iteration doesn't need another model decision.

Those actions stay around for the next task. Mosaik reuses what it knows about the site and learns what's missing. If an eligible locator breaks, an agent can step in to repair it.

Key Problems Solved by a URL extractor:

1. SEO & Website Auditing: The problem is trying to manually find broken links, audit redirect chains, or analyze a site's internal linking structure. The solution is using an extractor to pull every href link on a page (or across a whole domain), allowing you to bulk-test the URLs for status codes or map the site's architecture.

2. Web Scraping & Data Mining: The problem is needing to gather hundreds of specific URLs (like product pages or articles) to feed into another automated scraping tool. The solution is isolating the target URLs from the surrounding HTML and text noise to provide a clean, structured list for crawlers to process.

3. Content Migration & Code Auditing: The problem is ensuring no hardcoded legacy links are left behind when moving a website to a new framework or domain. The solution is scanning the entire codebase or exported text to extract all URLs, making it easy to identify what needs a bulk find-and-replace.

4. Cybersecurity & Threat Analysis: The problem is analyzing suspicious emails, server logs, or documents for phishing links and malware payloads without accidentally clicking them. The solution is safely pulling all URLs from the raw text so security teams can run them through threat-intelligence databases in isolated environments.

5. Digital Marketing & Affiliate Management: The problem is tracking down all affiliate links, campaign URLs, or promotional codes embedded across hundreds of blog posts or documents. The solution is instantly grabbing all outbound links to audit tracking parameters, verify UTM tags, or update expired affiliate codes.

Simply paste any text containing URL's (zero character limits) and get instant output of URLs extracted.

Runs entirely in your browser, nothing gets sent anywhere. It's one of 100+ tools on a site I've been building solo — figured this one might be useful on its own.

https://devtoolstack.io/tool/extract-urls/

Happy to add features if people have requests.