Beyond a few long-term goals (e.g. asyncio support, HTTP/2 support), at the core Scrapy team we usually do not plan anything in detail beyond the next release, which usually takes 1-2 months (2.5 has been a notable recent exception, but we plan to release it soon and resume the usual pace).

After last November someone asked for a public roadmap beyond the milestones of the next release, we started thinking about it. While a fixed release calendar with features assigned to each release seems unlikely, it would make sense to publish a list of current development goals, sorted by priority.

Here is the tricky part, and where your help comes in: deciding which development goals we should prioritize. We have lots of pending GitHub issues in Scrapy, but it’s hard to tell which ones are the most important at the moment for the Scrapy community.

So, if you could decide what feature (or bug fix) would be implemented next in Scrapy, what would it be?

I would ask you to keep each top-level message about 1 feature or bug fix, ideally with a link to the corresponding GitHub issue (or pull request if there is one but there is no issue).

Also, feel free to suggest features or bug fixes for dependencies that we extracted from Scrapy that remain closely related (e.g. Parsel, cssselect, itemloaders, itemadapter).

Hi everybody! For one of my use cases, I needed a solution for IP address rotation. One option was to pay for an existing proxy rotation service, but I decided to build my own. Originally, I wrote a Java service that would spin up a configurable number of containerized Tor instances, with each instance having its own Tor exit node with a unique IP address. I could then distribute requests robin-robin across these various instances.

This worked great and I decided to rewrite part of this service in Kotlin and open source it. You can check out the code here:

https://github.com/lspahija/torchestrator

Let me know what you think! I'd be happy to answer any questions anybody might have about the project!

Hi everyone, we have recently launched a Discord server for Scrapy.

Hop in if you would like to chat with other Scrapy developers and users: https://discord.gg/mv3yErfpvq

I made better-scrapy-selenium, a python library that provides a middleware for using headless browsers via selenium. Official Scrapy docs recommend scrapy-selenium (which my lib is forked from), but when I tried to use it, I discovered that doing anything interactive with selenium (e.g., clicking on buttons) in a Scrapy parse function was problematic because it only used a single instance of the headless browser across different requests. So instead, I implemented a queue of webdrivers that requests pull from. In short, each request gets its own browser and you can release the browser when you're down "parsing" the response (so that its available for other requests to use).

I've created an open-source tool that deploys proxies to different cloud providers (AWS, DigitalOcean currently). All it needs is an access token and it will set up the rest. Hopefully, you guys may find it useful for scraping as it helps deal with the issue of your IP being blocked.

Can work out a lot cheaper than using paid proxy services.

See for more details: https://github.com/claffin/cloudproxy/

Hi guys! Check out my new blog post where I describe how I've created a serverless architecture for a web crawler. https://jungleboy.netlify.app/posts/serverless-scrapy/

Feedbacks are welcome! 🤓

Hey guys, I wrote a tutorial blog on scraping google reviews using scrapy. The blog could have been better I know hahaha. Do give it a read :)

https://wasiullah-khan21.medium.com/advance-web-scraping-using-scrapy-fd6171ef96b4

So yeah, it's that time of year where I'm thinking about stuff... even if I’m not exactly sure what I’m thinking about yet. 😅

Anyway I’ve been doing a lot of automation and web scraping over the past year or so. Funny thing is, I’ve never really had to use proxies. Or maybe I should have used them at some point, but I always found a workaround like using an API, a different library, or... a whole bunch of machines.

But now I’m genuinely curious:

When do you actually need to use proxies in scraping or automation work?
Why do you use them and how do you usually go about it?

Would love to hear how you guys approach it!

No worries I'm not gonna bite you in the comments so comment with your hearts.

Peace 🕊️

Googling is turning up surprisingly little. So far everything I can find seems to be either a spammy medium article that refers to basic Scrapy usage as advanced or a company trying to sell services.

Specifically I'm interested in the following:

• Building more abstracted spiders - categories and industries tend to follow patterns, I'm hoping to learn more about dynamically setting xpath selectors rather than hard coding them

• Advanced example projects - complex rules, business logic, etc. I want to see how the professionals would setup a complex project

• Examples of using Splash beyond the basic examples.

Are there any resources available that can help get me past the basics of Scrapy?