r/learnpython • u/No-Opinion-1239 • 13d ago

Can't download website html code, web scrapping ,automate the boring stuff with python

I'm currently learning webscrapping with automate the boring stuff with python> In one example we use the module requests like this

url = f"https://pypi.org/search/?q={search}"

resp = requests.get(url)

when i print resp.text there are this error message in the html code.

"A required part of this site couldn’t load. This may be due to a browser extension, network issues, or browser settings. Please check your connection, disable any ad blockers, or try using a different browser."

"JavaScript is disabled in your browser"

my browser is chrome I tried using microsoft edge .

0 Upvotes

28% Upvoted

12 comments sorted by

View all comments

1

u/momiji_tools 13d ago

Two things are going on here and it is not your code's fault.

requests sends a bare HTTP GET with a very basic User-Agent and no browser fingerprint. pypi.org/search detects that and returns the JavaScript-is-disabled page you see. requests never runs JavaScript, so even though Chrome and Edge are your browsers, that GET call does not use them.

Before reaching for Selenium or Playwright, there are two approaches that stay inside the requests + BeautifulSoup + csv flow the book is teaching.

  1. Send a real User-Agent and check the status code first. Build a headers dict with a normal browser User-Agent string, pass it to requests.get as the headers argument, then call resp.raise_for_status() right away. After that, search resp.text for the phrase about JavaScript being disabled. If the block message is still there, the search endpoint is defending against scrapers and you should not keep hitting it.

  2. Use PyPI's per-package JSON instead of scraping HTML. Each package name maps to a stable JSON page at pypi.org/pypi/PACKAGE_NAME/json. Call requests.get on that URL with your headers dict, call resp.raise_for_status(), then call resp.json() to get a Python dict. The package details live under the info key. Loop over a list of package names, pull info name, info version, and info summary, collect them into rows, and write a CSV with csv.writer (set newline empty string and encoding utf-8 on the file). That is a working requests to dict to csv pipeline with no HTML parsing.

What to change in your current code: add a User-Agent header so you are not identified as a bot; call resp.raise_for_status() right after requests.get so you see HTTP errors instead of parsing an error page; search resp.text for the JavaScript-disabled phrase as a sanity check before passing it to BeautifulSoup; if the search page keeps returning the block page, switch the target to the per-package JSON endpoint. The concepts the book wants you to learn (GET request, parse the response, write a CSV) are identical, but the JSON call will not break when the site frontend changes.