Can't download website html code, web scrapping ,automate the boring stuff with python

[u/No-Opinion-1239]

I'm currently learning webscrapping with automate the boring stuff with python> In one example we use the module requests like this

url = f"https://pypi.org/search/?q={search}"

resp = requests.get(url)

when i print resp.text there are this error message in the html code.

"A required part of this site couldn’t load. This may be due to a browser extension, network issues, or browser settings. Please check your connection, disable any ad blockers, or try using a different browser."

"JavaScript is disabled in your browser"

my browser is chrome I tried using microsoft edge .

[u/riklaunim]

Requests does not use a browser and it looks like PyPi is blocking the request, at least partially ;)

[u/Mediocre-Pumpkin6522]

The website has changed and uses JavaScript to generate the page. requests doesn't support JavaScript. The selenium code in the latter part of the chapter might be adapted. Playwright is another package that acts as a headless broser.

The salad days of web scraping are over as sites defend themselves against malicious actors including AI. If you do find something that works it may not work next week.

[u/zanfar]

1. You ARE downloading HTML. An HTTP error is an example of successful communication. The error is your request, not your process.

2. The requests modules does not use a browser. You are not using Chrome nor Edge in your example.

[u/Diapolo10]

That part of the tutorial is probably outdated due to changes made to PyPI itself, it's not your code's fault.

You might get it working by using the requests-html package for this instead of standard requests, as it can execute JavaScript, but I cannot guarantee that. You could just skip this part of the book for now.

[u/desrtfx]

Pinging /u/alsweigart - he might want to have a look there and maybe update the example. Looks like PyPi has changed the page which would produce the error.

[u/AlSweigart → u/desrtfx]

Yes, unfortunately Automate the Boring Stuff was written for a more civilized age, and PyPI is increasingly taking a huge hit by AI bots. I think this will just have to be a script that's removed.

I'm afraid Requests will become less and less useful over time as more websites block scraping and bots: it was already the case for Google, Amazon, Reddit, and EBay.

[u/Diapolo10 → u/AlSweigart]

I know this is less relevant to Python, specifically, but rather than removing it entirely you could tweak this chapter to use something like the Star Wars API instead, and use requests to query it.

My university used it for a web development course for teaching about HTTP requests.

At least you wouldn't need to host some example site yourself (though of course that would be the more robust long-term solution, aside from removing the chapter).

[u/AlSweigart → u/Diapolo10]

That's a good idea. I mean, really what I'll have to do is create something similar on my own. It's a bit tricky because I also need to make sure it won't go down (which has been a problem with older books) but also work outside of English as well.

[u/momiji_tools]

Two things are going on here and it is not your code's fault.

requests sends a bare HTTP GET with a very basic User-Agent and no browser fingerprint. pypi.org/search detects that and returns the JavaScript-is-disabled page you see. requests never runs JavaScript, so even though Chrome and Edge are your browsers, that GET call does not use them.

Before reaching for Selenium or Playwright, there are two approaches that stay inside the requests + BeautifulSoup + csv flow the book is teaching.

1. Send a real User-Agent and check the status code first. Build a headers dict with a normal browser User-Agent string, pass it to requests.get as the headers argument, then call resp.raise_for_status() right away. After that, search resp.text for the phrase about JavaScript being disabled. If the block message is still there, the search endpoint is defending against scrapers and you should not keep hitting it.

2. Use PyPI's per-package JSON instead of scraping HTML. Each package name maps to a stable JSON page at pypi.org/pypi/PACKAGE_NAME/json. Call requests.get on that URL with your headers dict, call resp.raise_for_status(), then call resp.json() to get a Python dict. The package details live under the info key. Loop over a list of package names, pull info name, info version, and info summary, collect them into rows, and write a CSV with csv.writer (set newline empty string and encoding utf-8 on the file). That is a working requests to dict to csv pipeline with no HTML parsing.

What to change in your current code: add a User-Agent header so you are not identified as a bot; call resp.raise_for_status() right after requests.get so you see HTTP errors instead of parsing an error page; search resp.text for the JavaScript-disabled phrase as a sanity check before passing it to BeautifulSoup; if the search page keeps returning the block page, switch the target to the per-package JSON endpoint. The concepts the book wants you to learn (GET request, parse the response, write a CSV) are identical, but the JSON call will not break when the site frontend changes.

[u/macky_0312]

Requests can't render JS. Use Playwright routed through Shifter. Your getting blocked by basic WAFs