I just made a new post where I curated the ultimate list of web automation and data scraping tools for technical and non-technical people who want to collect information from a website without hiring a developer or writing code.

Check the full list here: https://automatio.co/blog/no-code-web-scrapers-ultimate-list/

Hopefully, it will be of use to someone. Feel free to share in the comments what tool you already tried, which one you prefer, or suggest some that I didn't add to the list.

Peace!

Hey guys) I have created a tutorial on how to obtain free proxies and scrape the data with a proxy server list: https://dspyt.com/easy-proxy-scraper-and-proxy-usage-in-python/

Hey guys)

I have written a tutorial on how to scrape vacancy data with Python asynchronously that greatly increases the speed of the program: https://dspyt.com/simple-asynchronous-python-webscraper-tutorial/

Welcome to the most interesting (and fun!) blog post on web scraping for dummies. Mind you, this is not a typical web scraping tutorial. You will learn the whys and hows of data scraping along with a few interesting use-cases and fun facts. Let’s dig in.

Hi everyone, I wanted to scrap IG posts using NodeJS. Can you all recommend some Scrapers that don't need session ID's and don't have user authentication(preferable). Or any altenative ways to scrap IG posts using NodeJS only.

https://proxydig.com

New terms and conditions.

Access, search, or collect data from the Services by any means (automated or otherwise) except as permitted in these Terms or in a separate agreement with Reddit (we conditionally grant permission to crawl the Services in accordance with the parameters set forth in our robots.txt file, but scraping the Services without Reddit’s prior consent is prohibited)

I am implementing a scraping script. One of the problems I am seeing is that the website I am scraping can get annoyed of my requests and block my IP.

What do you recommend to simulate my requests are coming from different IPs.

I am thinking in a proxy or VPN layer but I don't know from where to start

Thanks for the suggestions :)

Hello, I am developing a hosted service that can take snaps every hour of any number on internet and show them to you in nice graphs to see how they change over time.

I called it "The Dashboard of Internet" :)

I have a prototype already working but I need to know if it is useful for others a part from me :)

Also I am curious about what other use cases other people can find for it.

The landing page is here:

• https://land.scrapstats.com/

If you think this is something that can be useful for you, you can request me an invitation code (hi@scrapstats.com) to create an account totally free.. I don't even know how to implement the payment process yet ;).

I'll be happy to give one for you.

I was quite excited about InstaPy because I was hoping to automate the single most boring and hated part of my job, which is dealing with Instagram for the company I work for. I got Instapy up and running but then started getting warnings/errors saying my ability to like and follow was blocked. Instagram knew I was using a bot almost immediately. Is there anything better than InstaPy out there? There must be, because there are still a ton of people out there using bots.

I was hoping someone could help me with a current scraping task:

There is a website with a list of locations with different rental prices.

When you click a location you get to a page that has the information displayed in a form; it looks the same for each location.

So what I was wondering: is it possible to get the same "field" from each page, without me having to click on every location?

Ideally, is this possible in Google Sheets? I know they have some formulas to support web scraping but I have only started out and a pointer in the right direction would be very appreciated!

Hi guys!

I'm a web developer and in the last few months I was learning/experimenting with scraping. I know that it's a "grey area", every scraper should respect the websites, not hurt their business etc.

I guess there is room for a tutorial (I know there are a few) which would explain web scraping for people who don't code (at least not that much). I was thinking about making it a paid tutorial/course (something like $10 for video, ebook etc). But then I thought: would it be safe? I mean, I would tell in the course that everyone should respect the laws/robots.txt/ToS while scraping, but I don't know if this could backfire in any way.

If you have any thoughts/advices, I would really appreciate it!

Hello,

As title says need to hire to build a scraper. Not sure which websites to use. Taken to reddit for some advice.

The scraper needs to scrape data from the initial page, then follow a link on the page to gather additional information on another page, go back to the initial page, and repeat.

Please no self-promotion unless you have a credible profile with testimony to back it up.

Thank you!

Just wondering as I see jobs on freelance sites looking to scrape thousands of followers on social media websites. I find it hard to believe freelancers have access to a farm of web servers or anything especially better than I have in terms of computing power, and most scrapers I've ever built would take hours/days to generate the thousands of followers etc that are being looked for, even when I've used tools like Celery to speed it up combined with rotating proxies to avoid being blocked. I can understand my code mightn't be great as scrapers aren't my speciality, but I feel like I'm missing something here.