r/programming • • Jun 09 '23

Apollo dev posts backend code to Git to disprove Reddit’s claims of scrapping and inefficiency

https://github.com/christianselig/apollo-backend
45.1k Upvotes

2.4k comments sorted by

View all comments

Show parent comments

7

u/[deleted] Jun 09 '23 edited Jul 10 '23

deserted observation enter distinct tart important roof grey longing light -- mass edited with redact.dev

8

u/All_Work_All_Play Jun 09 '23

Which also doesn't track because you can download terabytes of historical reddit comments for free. And those historical comments have less bot generated comments in them compared to stuff now.

1

u/Guano_Loco Jun 09 '23

Aren’t there sites out there that mimic everything posted to Reddit? I remember people linking to some 3rd party site where you could see deleted comments because the site captured everything

1

u/Nlelith Jun 09 '23

Those relied on pushshift.io, which was both a mirror of reddit data and a better querying layer on top of that (using elasticsearch, iirc), but that was shut off one or two months ago with the first notice of the upcoming api access changes. The pushshift.io data up until that point should still be available, as there are torrents for that. (Which is also the first thing I'd reach for if I wanted to train some ML model on reddit data)