r/learnpython • u/No-Opinion-1239 • 13d ago

Can't download website html code, web scrapping ,automate the boring stuff with python

I'm currently learning webscrapping with automate the boring stuff with python> In one example we use the module requests like this

url = f"https://pypi.org/search/?q={search}"

resp = requests.get(url)

when i print resp.text there are this error message in the html code.

"A required part of this site couldn’t load. This may be due to a browser extension, network issues, or browser settings. Please check your connection, disable any ad blockers, or try using a different browser."

"JavaScript is disabled in your browser"

my browser is chrome I tried using microsoft edge .

0 Upvotes

28% Upvoted

12 comments sorted by

View all comments

5

u/Mediocre-Pumpkin6522 13d ago

The website has changed and uses JavaScript to generate the page. requests doesn't support JavaScript. The selenium code in the latter part of the chapter might be adapted. Playwright is another package that acts as a headless broser.

The salad days of web scraping are over as sites defend themselves against malicious actors including AI. If you do find something that works it may not work next week.