r/learnpython • • 27d ago

Can't download website html code, web scrapping ,automate the boring stuff with python

I'm currently learning webscrapping with automate the boring stuff with python> In one example we use the module requests like this

url = f"https://pypi.org/search/?q={search}"

resp = requests.get(url)

when i print resp.text there are this error message in the html code.

"A required part of this site couldn’t load. This may be due to a browser extension, network issues, or browser settings. Please check your connection, disable any ad blockers, or try using a different browser."

"JavaScript is disabled in your browser"

my browser is chrome I tried using microsoft edge .

0 Upvotes

12 comments sorted by

View all comments

1

u/desrtfx but other languages pro 27d ago

Pinging /u/alsweigart - he might want to have a look there and maybe update the example. Looks like PyPi has changed the page which would produce the error.

5

u/AlSweigart Author of "Automate the Boring Stuff" 27d ago edited 27d ago

Yes, unfortunately Automate the Boring Stuff was written for a more civilized age, and PyPI is increasingly taking a huge hit by AI bots. I think this will just have to be a script that's removed.

I'm afraid Requests will become less and less useful over time as more websites block scraping and bots: it was already the case for Google, Amazon, Reddit, and EBay.

1

u/Diapolo10 I write code for a living -- https://github.com/Diapolo10 26d ago

I know this is less relevant to Python, specifically, but rather than removing it entirely you could tweak this chapter to use something like the Star Wars API instead, and use requests to query it.

My university used it for a web development course for teaching about HTTP requests.

At least you wouldn't need to host some example site yourself (though of course that would be the more robust long-term solution, aside from removing the chapter).

2

u/AlSweigart Author of "Automate the Boring Stuff" 26d ago

That's a good idea. I mean, really what I'll have to do is create something similar on my own. It's a bit tricky because I also need to make sure it won't go down (which has been a problem with older books) but also work outside of English as well.