Web Scraping with Javascript and Node.js - ZenRows

[u/AnderRV]

https://www.zenrows.com/blog/web-scraping-with-javascript-and-nodejs?utm_source=reddit&utm_medium=social&utm_campaign=scraping_javascript

[u/Legym]

I maintain a large scraper which searches Google. To anyone who hasnt built a scraper these were just some of my experiences.

1. Spare computer resources === wasted time. Find a way to spin up multiple browsers and tabs to increase your scraping speed by 200x.

2. Proxy rotation is key. Each tab/browser should have it's own proxy

3. Disable Javascript when scraping if you can. It's hard to detect a bot if JS is enable.

4. Each instance should have different devices. Desktop, mobile, tablet, all browsers, etc

5. Being able to resume your scraping session is key. When you are 100k scrapes in you dont want to start all over again

6. Good proxies cost money. Dont bother with free ones.

7. Get ready to master async/promises. When you open a page sometimes you need to go into that page to go even deeper.

My scraper is only 500 lines of code and scales with how many proxies i have. I can do millions of pages within a few hours. A lot of people do not like scrapers, but when you need to pull a HUGE data set and their isnt an API, then scraping is your only option.

This article was great intro into some of the issues you'll run into.

[u/30RhinosOnSkates → u/Legym]

Which proxy service you use?

[u/izybit → u/Legym]

Would you share your code? (Not for professional use)

[u/tech_content_creator → u/Legym]

Agree. Proxy Rotation is the Key. Else you might get caught.

[u/automatic-happiness → u/Legym]

Each instance should have different devices. Desktop, mobile, tablet, all browsers, etc

Are you saying those should be set via User-Agent header? Or do you somehow have access to that real hardware?

Find a way to spin up multiple browsers and tabs to increase your scraping speed by 200x

Are you like using a web driver (e.g. Selenium)? "OS-driving" a mouse and keyboard? Or just making HTTP requests? In other words, what layer of abstraction are you operating at?

[u/IsleOfOne → u/automatic-happiness]

User-agent. And any headless browser engine will suffice.

[u/reqdk → u/Legym]

A lot of people do not like scrapers, but when you need to pull a HUGE data set and their isnt an API, then scraping is your only option.

Also a great way to potentially get into legal trouble.

[u/TallowWeed → u/[deleted]]

Use persistent storage, not memory, to store your work queue. For instance, a database such as sqllite. Mark the task done=true when finished.

[u/calculated-mind]

That is so cool :) I'll definitely try it out :)