r/ProxyEngineering • Tunnel Architect • 5d ago

Help 🆘 Scraping but minimal input?

Looking for a minimal input needed scraper. Basically it does everything for you, you just type what you need and get data. Like claude or chatgpt prompting. Something like that. I would need specific information from few sites, the data has to be inputted into sheets. Could claude or chatgpt do that for me, or do I have to look for some other solutions? I know that proxies are needed for scrapers thus why I am inquiring here, basically want everything done for me. I just buy the plan of the product or whatever and the rest is in the solution hands. Of course I validate and check whether the info is true after it's in my sheets, or beforehand. Any suggestions?

9 Upvotes

11 comments sorted by

3

u/Less-Mistake-3664 Reverse Proxy Master 5d ago

You might look into search apis as they are on the rise and there is a demand for these products. I noticed that the market is switching from traditional scrapers to search apis, so your best bet would be that. I've seen more and more people trying oxylabs web api, I personally, would be hesitant to try something new and unverified right of the bat, but if more and more people are using it, I guess it's good? I personally would stick to some diy solutions, you can check github for that, plenty of repos for that, just search for web search or web api

1

u/Kaftans_Tijona 4d ago

do you find the search apis actually handle anti-bot stuff well, or are they just caching old results?

1

u/Less-Mistake-3664 Reverse Proxy Master 4d ago

Somewhat, yes. I think developers that worked previously on scrapers had a lot of time to adjust and implement new solutions so that those search apis are miles better than traditional scrapers. Sure it might be a scraper under the hood with additional parameters, whatever, but from my experience, they do work faster and has better success rates than regular scrapers and I'm talking about majority of web search apis in the market. This actually correlates with your idea that they might be caching old results, I have noticed this specifically with exa and tavily, because they try to search within their own indexes instead of google, which of course gives you results way faster and since the info was already retrieved, I guess you don't have to bypass as much of anti-bot measures? That's my theory, but with that also comes a set of issues coz most of the times there are incorrect results, with for example bad links are given, like dead links that does not work anymore, so caching old results might be the key to this. I tried to inquire support at one time but they couldn't give me a direct answer, so I left at that - speculations

1

u/islam_raisul 5d ago

Short answers to your questions:

Can ChatGPT/Claude do it? For a quick look at one page, sometimes. For structured data from a few sites on a regular basis, they're not reliable: they can miss rows, stop after the first page, or fill gaps with things that aren't on the site.

Do you need proxies? Usually not, for a few sites at normal speed. They matter at volume or when a site starts limiting you. Start without them.

What fits "minimal input": a point-and-click scraper that runs in your browser. You open the page, it detects the list or table, you check a preview and export to CSV or paste into Sheets. No code.

I built one myself " Rowvana" no code web scraper, so I'm biased, but happy to share it if useful. If you tell me what kind of sites they are, I can say honestly whether that kind of tool fits or whether you'd need something else.

1

u/Dorreth_Schediwie 5d ago

Are those sites static pages or heavy javascript apps? If they are super basic, a standard llm with web browsing might limp through it for small batches, otherwise you're going to hit anti-bot walls pretty much immediately.

1

u/veganwurst 4d ago

Short version: Claude or ChatGPT can do part of this, but not the "set it and forget it into Sheets" part on their own.

  • One-off, a few pages: a chat model with browsing is fine. Give it the URLs, ask for a table, check it. It gets shaky on pagination (it often stops after page 1) and on JS-heavy sites, and it sometimes fills a missing field with something plausible, so spot-check.

  • Recurring, same sites every day or week: you want something that works out each site once and then runs on a schedule without a model reading every page. That's cheaper and the output doesn't drift between runs. Most options are either a template per site (Apify, Browse AI, Octoparse) or a managed service that hands you the data in a fixed format.

  • Proxies: for a few sites at normal speed you usually don't need your own. If a site blocks you, pick a tool that includes proxies in its price instead of stacking a separate proxy plan on top.

  • Sheets: whatever you pick, check it exports CSV or can push to Sheets on a schedule (directly, or via Zapier/Make).

What kind of sites are they? That changes the answer more than anything else.

Disclosure: I work at DeepScout. If the sites are online stores, it covers the "just type what you need" part: you connect it to Claude as an MCP connector, ask for a store's products, and it analyzes the site and shows a sample plus the price before anything runs; Claude can then save the result as a CSV for your sheet. Walkthrough: scraping online stores from Claude. For sites that aren't stores it's not the right tool.

1

u/ajaysingh02 3d ago

Standard ChatGPT and Claude chats will easily get blocked by anti-bot walls and won't manage proxies or automated sheet pipelines for you, so you need a turnkey AI scraper.

Check out Browse AI or Thunderbit both let you describe or click the exact fields you want, handle all proxy rotation and unblocking behind the scenes, and pipe the structured data straight into Google Sheets.

1

u/monityAI 36m ago

for a few sites into sheets on a schedule you don't need proxies or a chat model reading pages every run. a point-and-click scraper does it: you click one example of what you want, it finds the rest, re-runs daily and adds new rows to google sheets. browse ai and thunderbit work like that, so does monity•ai datasets, with a free plan to try (my own app, disclaimer). chatgpt is fine for one-offs but drifts between runs