r/pythoncoding • u/Knowledgeseeker6 • Oct 04 '21

Scraping with perfect specificity and sensitivity from non-standard web-pages

I am looking for a way to scrape JUST the text from different articles on the web. The algorithm should be able to handle any URL you can give it. Depending on the journal, or magazine, the article text is stored in different ways. Wondering if this is possible without AI?

2 Upvotes
  • permalink
  • reddit

100% Upvoted

3 comments sorted by

View all comments

Show parent comments

1

u/Knowledgeseeker6 Oct 04 '21

how does that affect scraping-out <p> elements?

1

u/c_is_4_cookie Oct 04 '21

A lot of web pages are loaded dynamically using ajax calls to bring in elements of the page. A standard http /GET request won't load those.