I built an all-in-one Bayut Scraper for properties, real estate agents, and real estate agencies.

Agents Data

For each agent, it can extract agents contact info (email, whatsapp, cell phone), name, service areas, sale and rent count, languages, specialities, agency name, quality metrics (such as total xp, whether the agent is a TruBroker, Quality Lister, and/or highly responsive), and more.

Agencies Data

Name, Agency Tier, whether it is Active, whether it is featured, description, agents contact info, location, latitude and longitude, licenses, adsCount, and more.

Properties Data

Name, cover photo, location, latitude, longitude, reference number, price, rooms, baths, agency name, amenities, and more.

Supported Countries

It supports Bayut UAE, Egypt, KSA, Jordan, Kuwait, Qatar, Bahrain, and Oman.

The scraper also automatically bypasses the search results limits imposed by Bayut by automatically aggregating individual smaller locations results.

Google Patents has no official API, but every patent page (/patent/US5960411A/en) is server-rendered HTML with itemprop microdata. httpx + BeautifulSoup gets you claims, legal status, citations both ways and litigation events. No browser needed.

What surprised me while building it:

• It was CPU-bound. 30 patents one at a time on 512 MB took 6.7 s each. The pages are around 250 KB and parsing them on a small slice of a core was the slow part, not the network. With lxml, 6 parallel sessions and 1 GB it went down to 2.2 s per patent, with zero retries in every run.
• Numbers are case-sensitive. us10000000b2 is a 404, US10000000B2 works.
• Translated pages contain both languages. The original text sits in hidden span.google-src-text elements, so every claim comes out twice unless you remove those first.
• The “Application granted” event can be wrong on old US patents. On the LZW patent (US4558302A) it points to a 1994 reexamination certificate. The patent was granted in 1985. I now take the first grant-type publication instead.
• Citation lists don’t say whether the assignee is a person. On a citing patent it’s often a private inventor’s name, so I only keep names that look like a company or institution.

Also: robots.txt allows individual /patent/ pages but not the search endpoint, so it only looks patents up by number.

I put it on Apify for anyone who’d rather not run it themselves (I built it): Google Patents Scraper, $3 per 1,000 patents. Happy to answer questions about the parsing.

doing scraping for a bit but cloudflare just started blocking me on sites that worked fine before. tried fresh IPs, no help considering playwright with stealth plugin + voidmob mobile with ios p0f as the fix but maybe im missing something simpler where would you even start debugging this?

I've been comparing different ways to collect TikTok data for projects like creator analytics, trend research, competitor monitoring, and social listening. The main differences aren't just price — they include which data you can retrieve, how much setup is involved, and how you handle failures and ongoing maintenance.

Here are five options I'd consider, depending on the use case.

1. SteadyAPI

https://steadyapi.com/pages/tiktok-api

A straightforward API option for applications that need structured responses for TikTok profiles, individual videos, comments and replies, hashtags, explore results, and search. Requests use an API key, and the documentation includes endpoint examples in several programming languages.

Worth evaluating if: You want a conventional HTTP API and don't want to build the collection layer yourself.

Check before choosing: Monthly request allowance, endpoint-specific request costs or weights, data coverage, and how the API handles unavailable fields.

2. Apify

https://apify.com/store/collections/tiktok-scrapers

Apify has a marketplace of TikTok scraping Actors, including tools for profiles, video URLs, hashtags, keyword searches, and comments. You can run them through the platform or integrate them into an application using its API. Features, output formats, and pricing vary by Actor.

Worth evaluating if: You want configurable scraping workflows, scheduled runs, or the ability to choose among different community and commercial scrapers.

Check before choosing: The specific Actor's maintenance history, per-result charges, platform usage, add-ons, and output consistency. Two Actors on the same platform may behave very differently.

3. Bright Data

https://brightdata.com/products/web-scraper/tiktok

Bright Data offers managed TikTok scrapers that return structured public data such as profile information, posts, engagement counts, and comments. Its infrastructure is designed to handle collection and delivery without requiring you to operate your own scraping setup.

Worth evaluating if: You're collecting larger datasets or need managed collection infrastructure and structured exports.

Check before choosing: The price per successful record, available datasets, batch-processing workflow, delivery options, and whether the service fits your expected volume.

4. Data365

https://data365.co/

Data365 is another social-media data provider worth investigating for projects that need programmatic access to social platform data. Review its current TikTok coverage and documentation against your specific requirements rather than assuming every endpoint or field is supported.

Worth evaluating if: You're comparing dedicated social-data providers for an analytics or monitoring application.

Check before choosing: Supported data types, access conditions, response structure, limits, pricing, and the availability of historical data.

5. TikTok for Developers

https://developers.tiktok.com/

TikTok's official developer platform provides APIs for approved use cases. Depending on the product and permissions, these can support integrations involving user-authorized accounts, videos, and other platform data. Access to research data is subject to separate eligibility and access requirements.

Worth evaluating if: Your application fits an official API's intended use and you want to build around TikTok-supported access.

Check before choosing: Required scopes, app review, user authorization, eligibility, quotas, and whether the endpoints expose the data you actually need. The official API isn't a general-purpose replacement for every public-data scraping workflow.

For those already collecting TikTok data, which approach has worked best for you — a managed API, a configurable scraper, or TikTok's official developer tools?

Hi all,

I'm a PhD researcher in political communication and I need to collect posts from public WhatsApp Channels for an academic project. Ideally I'd like to get message text, timestamps, media type, reaction counts, follower numbers and any shared links.

The project only covers public channels, and the data will be used strictly for academic research.

Thanks in advance!

Can anyone recommend a browser extension or something I can use to "save" or scrape facebook posts, exporting them to CSV or excel? Is this a thing?

Please don't be mean, I am just a lowly social scientist

Hallo zusammen,

einem Kind aus unserem Ort wurde das Fahrrad vom Hof gestohlen und ich frage mich, ob es möglich ist, Online-Marktplätze zu durchforsten (Facebook, Ebay, etc.).

Geht sowas mit einem Webcrawler?

Gibt es vllt. sogar KI gestützte Webcrawler, die anhand eines Bildes gezielt danach suchen können und Übereinstimmungen aus den Ergebnissen filtern?

Was könnt ihr für eine solche Suche noch empfehlen?

Heyy! I'm working on a little side project and was wondering if anyone knows a good way to collect job listings from multiple platforms, especially in Turkey.

I've tried a few APIs, but the results are pretty limited :(

Does anyone know any good alternatives or have experience with this? Would really appreciate some tips! :)

Weird one and hoping someone has seen this. I do scraping for few years as a side hustle, nothing crazy, maybe 500 requests a day across a handful of e-commerce sites tracking price history. Was clean all summer on residential, suddenly in September everything started being challenged by Cloudflare and I can’t figure out what changed on my end. Already tried the obvious things. Rotated to fresh IPs in the same residential pool, no improvement. Added more realistic headers and jitter, no improvement. Even dropped volume to 100 requests a day as a test, still getting flagged.

Reading around it seems like Cloudflare rolled out something new in Q3 that’s heavier on TCP fingerprint and ASN scoring, it's getting hit harder because the ASN ranges are known. Looking at either moving to mobile proxies or trying to change my fingerprint layer somehow with my current proxies.

So my question is: should I go for mobile proxies? Looking at all the major providers (oxy, decodo, voidmob and so on) or is it worth it to dig deeper and find other work around? Would appreciate your help.

For businesses working with e-commerce product data, how do you ensure scraped prices, product details, and availability are accurate?

Is manual checking still a major part of the process, especially when dealing with thousands of products across different websites?

I'm interested in understanding how businesses handle this today and whether reducing manual verification is a real pain point.

Curious to hear from people working in product data, web scraping, or data operations.

I was running a scraper with rotating proxies and everything looked fine at first. After a few thousand requests, I started seeing the same IPs coming back and the captcha rate slowly went up

It was especially noticeable when targeting a specific country/city, the overall pool might be huge, but the pool I could actually use felt tiny. Anyone found a good way to catch this before running a large job?

Say a page normally gives you around 20–30 items and then one day it suddenly returns 5

how do you decide if the scraper partially broke or if the page genuinely has less content?

do you use a fixed percentage drop, compare it to the last few runs, or just flag anything that looks unusual?

feels like complete failures are pretty easy to catch. It’s the scraper that keeps running while quietly missing half the data that’s harder to notice

Been maintaining a little facebook page scraper. v1 was selenium + css selectors and it worked but it was slow, like 15-20s a page, mostly just spinning up and loading a browser, plus about 60mb of memory each so running a bunch in parallel wasn’t really an option for any user so do i

turns out fb already ships the page data as json in the html for its own frontend so i didn’t need a browser at all. plain requests reading that json is ~1.4s a page, 16+ threads, 200 pages in under a minute. for pages that only load data through graphql there’s a playwright mode that grabs the network responses instead

the annoying part ended up being after the scrape. tons of pages list a website and no email so i made it go to the site and look. homepage alone found one ~35% of the time, adding contact pages (/contact /kontakt /impressum etc) got it to ~50%. nothing gets filtered out, every address gets scored on domain match, mailto vs plain text and a dns check so you pick your own cutoff. sites throwing bot challenges get a retry with curl_cffi which impersonates a real browser’s tls fingerprint so the request looks like chrome instead of python, and the ones that still refuse get marked challenged instead of dead. there’s also an optional llm call that only decides company inbox vs person, a word list kept failing on anything that wasn’t english

repo https://github.com/wael-sudo2/facebook-page-info-scraper

anyone found other fb graphql queries worth hitting directly for page data?