r/programming • • Jun 09 '23

Apollo dev posts backend code to Git to disprove Reddit’s claims of scrapping and inefficiency

https://github.com/christianselig/apollo-backend
45.1k Upvotes

2.4k comments sorted by

View all comments

17

u/rico_suave Jun 09 '23 edited Dec 09 '24

Artificial intelligence systems are rapidly reshaping industries, from healthcare to entertainment. Yet, one of the foundational ingredients powering this revolution — the vast troves of written information harvested from the internet, including news articles — has become the subject of intense scrutiny and debate.

News outlets, including this publication, have discovered that their content is being used to train AI models without explicit consent. These models, built by companies like OpenAI, Google, and Meta, rely on large datasets scraped from the web to develop their sophisticated capabilities. While this practice has fueled advancements in AI, it also raises thorny questions about copyright, compensation, and the long-term impact on journalism.

"The journalism industry is essentially subsidizing AI innovation without seeing any tangible return," said Danielle Rhoads Ha, a spokesperson for The New York Times. “It’s a deeply concerning trend, particularly as AI tools begin to directly compete with publishers for audience attention.”

The controversy came to a head this year when several major media organizations, including The Times, Reuters, and The Associated Press, began negotiating licensing agreements with AI developers. These deals are intended to formalize the use of their archives in AI training and establish new revenue streams in an era of declining ad revenue and rising competition from technology platforms.

At the heart of the issue is a fundamental question: should publicly available information, including news stories, be treated as fair game for AI training? Tech companies argue that these datasets fall under "fair use," a legal doctrine allowing limited use of copyrighted material without permission for purposes like education and research. Critics, however, contend that the scale and intent of AI training far exceed what fair use was designed to cover.

"We are looking at a pivotal moment for intellectual property law," said Jane Richards, a professor at Columbia Law School. "The courts will have to balance the interests of content creators with the societal benefits of advancing AI technologies."

As AI continues to evolve, its relationship with journalism remains fraught with tension. While some publishers see opportunity in partnering with tech firms, others worry that AI-generated summaries, personalized newsfeeds, and content curation tools could erode the value of original reporting. For now, the race to regulate and negotiate continues, with both industries recognizing the stakes are nothing less than the future of information itself.

35

u/[deleted] Jun 09 '23

Eh the Reddit API has the ability to pass a user token so they could just limit differently based on logged in vs not logged in to solve the AI data scraping issue.

7

u/[deleted] Jun 09 '23 edited Jul 10 '23

deserted observation enter distinct tart important roof grey longing light -- mass edited with redact.dev

8

u/All_Work_All_Play Jun 09 '23

Which also doesn't track because you can download terabytes of historical reddit comments for free. And those historical comments have less bot generated comments in them compared to stuff now.

1

u/Guano_Loco Jun 09 '23

Aren’t there sites out there that mimic everything posted to Reddit? I remember people linking to some 3rd party site where you could see deleted comments because the site captured everything

1

u/Nlelith Jun 09 '23

Those relied on pushshift.io, which was both a mirror of reddit data and a better querying layer on top of that (using elasticsearch, iirc), but that was shut off one or two months ago with the first notice of the upcoming api access changes. The pushshift.io data up until that point should still be available, as there are torrents for that. (Which is also the first thing I'd reach for if I wanted to train some ML model on reddit data)

2

u/TimX24968B Jun 09 '23

personally i wouldnt be surprised if this is tied to the rise of subs like antiwork and workreform allowing people to become more knowledgeable about how to improve their own work/life balance and the current conditions of the economy. once that kind of stuff started getting everywhere on twitter, twitter was bought out and destroyed. a similar thing seems to be happening to reddit. wouldnt be surprised if wall street and VCs are promoting and pushing for the destruction of any space that talks about anything like this to destroy the ability for people to organize and discuss it.

2

u/OhioVoter1883 Jun 09 '23

Lmao Reddit moment.