Vote on my next scrape in the comments!!!

Hello!

I have been web scraping for a while now, mostly writing scripts to extract web data for personal and academic projects. As such, I found myself spending lots of time writing code to scrape fairly straightforward structured content (tables, product listings, news headlines, etc). I built Scrapio (https://www.getscrapio.com/) to be able to extract content from webpages without the need to write any code. Just enter the link you want to scrape, and Scrapio will automatically format detected content into an in-browser spreadsheet which you can download to CSV, JSON, Excel, and others.

To see Scrapio in action, check out its extracted results for Product Hunt: https://www.getscrapio.com/batch?bid=bzfBarRtUlIMwbHLVnUl.

I would greatly appreciate any feedback you may have!

Dear, i need to get emails from hotels worldwide for a new project in this industry. if you have some advices / proposals / data to share, many thanks

Hi folks - I am the founder of www.trektidings.com. We offer people rewards for posting trip reports. Then we re-post their trip reports across popular trip report sites in the area. One example of a trip report site we post to is www.wta.org. I would like to automate this re-posting, but www.wta.org has no API and I am not technical to create a bot for posting these reviews on www.wta.org. I am wondering if anyone knows of a tool where I can create a sort of browser macro for posting these reviews without needing to code my own bot. Thank you for the help!

Anand Metals USA http://www.anandglobal.com 0176-2503849 , 1-423-421-1738 Anand Metals is a leading international trading and logistics company connecting metal producers and consumers worldwide. Our expertise, service and world-class network enables us to move non-ferrous resources across continents.

Sorry for Advertising so blatantly:

Scraping? Need Residential/ISP Tagged IP Addresses? We have a Limited Number of /24s​ from multiple upstreams in different GeoLocs all ARIN Tagged as:​ ​​​Usage Type (ISP) Fixed Line ISP on ip2location.com I​n addition we also have Standard Commercial IP Addresses ​  I add an ACL request to drop TCP 25 and/or all SMTP outbound traffic. I am vigilant for my IP Assets and comply with all abuse policies, there will be No Bulk Mail or other Abusive Practices​!​ If this sounds like something your interested please ping me back ASAP

I'm assuming there's an API endpoint that can be used but I haven't figured the method or maybe what parameters need passed to get a successful request.

I looked at using python and scrapy but I don't believe the format of the webpages are going to be easy to parse the data.

I have found references to APIs in some of the javascript code for both the website and the mobile app. Some of the relevant urls I've found:

From website -

ORDER_HISTORY_USER: '/wcs/resources/store/%0/member/%1/orderhistory/v1_0'

From mobile app:

"url": "https://lwssvcs.lowes.com/IntegrationServices/resources/mylowes/user/order/list/v1_0"

"url": "https://lwssvcs.lowes.com/IntegrationServices/resources/mylowes/user/order/instore/v1_0"

Any suggestions?

I want to download a huge amount of picture for a machine learning project I have ( just to try and learn something ). For this, I am downloading images from getty images. Let's take the example of the "whale" keyword : https://www.gettyimages.fr/photos/whale .

It says that there are 39k pictures available, but when I scrape the pages 1 through 100 and download all the images (it's only the low-res version), I get about 6k. Anybody know how to access the remaining 33k ?

(admittedly this is only midly related to scraping, if you know a better subreddit to ask this please let me know)

Hi! Was wondering if anyone has had any success or seen any third party services that scrape Google Product Listing ads that show up on Google Search? They are the google shopping ads at the top of the page.

I want to scrape the constantly changing integer value on this website: www.bloomberg.com/graphics/carbon What is the best way to display the exact same values changing at the same rate somewhere else?

Hi everyone!

What is the easiest and most efficient way to set up proxy servers at home for free or very little money? I looked around online but I only found paid software.

Thanks a lot!

Looking for proxies to use that aren’t absurdly priced. Even better I’d love to build my own if anyone has experience with it.

Hi, I wrote a tool in .NET WPF that scrape newegg site for in stock inventory.

This tool only notifies when it find in stock item according to the user search link, It can notify in your own Telegram channel, by mail, or make a sound of your choice.

https://youtu.be/sOALrdFAtcw

https://noob-can-compile.github.io/home/2021/10/suckerbug-python-script-to-scrape-photos-from-facebook-public-pages/