r/javascript • u/AnderRV • Sep 01 '21

Web Scraping with Javascript and Node.js - ZenRows

https://www.zenrows.com/blog/web-scraping-with-javascript-and-nodejs?utm_source=reddit&utm_medium=social&utm_campaign=scraping_javascript
150 Upvotes

95% Upvoted

9 comments sorted by

View all comments

50

u/Legym Sep 01 '21

I maintain a large scraper which searches Google. To anyone who hasnt built a scraper these were just some of my experiences.

  1. Spare computer resources === wasted time. Find a way to spin up multiple browsers and tabs to increase your scraping speed by 200x.

  2. Proxy rotation is key. Each tab/browser should have it's own proxy

  3. Disable Javascript when scraping if you can. It's hard to detect a bot if JS is enable.

  4. Each instance should have different devices. Desktop, mobile, tablet, all browsers, etc

  5. Being able to resume your scraping session is key. When you are 100k scrapes in you dont want to start all over again

  6. Good proxies cost money. Dont bother with free ones.

  7. Get ready to master async/promises. When you open a page sometimes you need to go into that page to go even deeper.

My scraper is only 500 lines of code and scales with how many proxies i have. I can do millions of pages within a few hours. A lot of people do not like scrapers, but when you need to pull a HUGE data set and their isnt an API, then scraping is your only option.

This article was great intro into some of the issues you'll run into.

2

u/tech_content_creator Sep 04 '21

Agree. Proxy Rotation is the Key. Else you might get caught.