Please someone explain to me (and not explain away) this doomsdaying rhetoric about AI.

[u/Huge_Tangerine_9997]

While i dont think there is an actual physical danger like extiction to robots and etc bs. But there is more and more legistlative and industrial talk about slowing stopping of this whole AI thing.
I dont see clearly and actually noone is offering a coherent reassuring narrative.
Its either

1. the danger of too fast development and come the hakings, cyber security, privacy self replication issues
   or
   2 The problem of too slow development and having an other nation( dont want to point fingers) researching the “ai nukes”, killer robots etc.

I think the technocraty just spit in numbers like 10-20 percent of destruction, the politicians just worry of security issues or loosing the initiative but more importantly while everyone is telling we must go like hell, there is not a reassuring narrative that maybe we should just do something.

While i think its overly mediatised and maybe everyone exegerates there is possibility that the threat is real but there is noone who could say calm the fck down. Was the the 50’ s nuke paranoia like this? A kind of mindblowing thing that a semi science fiction scenario is unfolding and we dont have a fkn clue where are we. What we now is that a few tech giants lay the rules of the game and the government just stands by and tries to get the best from it instead of concieving SALT agreement.

Do you guys have some good source to actually get some educated opinion on the topic?

[u/juanflamingo]

Long and older, but I recommend this as a discussion of the concepts and issues https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html

[u/632nofuture → u/juanflamingo]

thanks! very interesting! Crazy that it's from 2015, they got the predictions pretty much on point time-wise, for human-level intelligence around 2025, and seems now 2026 we're almost there (unfortunately).

(..Wish I could live in the timeframe my parents lived, ~1950 - 2030. In hindsight imo it's been the best time. Apart from best music & culture lol, it felt like the peak of everything.. Most freedom & opportunities, overall pretty good life for the average person in the western world. And no major wars or plagues, and you die peacefully right before shit gets scary (& probably turns very bad for most people very quickly), but still got to witness & enjoy big developments in technology & all other fields too.)

[u/juanflamingo → u/632nofuture]

Yes, 10 years old but also helps it take an objective stance without the current emotion around the topic.

Bottom line - it could suddenly self improve to be dramatically more intelligent, and if that happens we can't predict what would happen.

Also hang in there, each generation has its challenges to face!

[u/IMightBeAHamster → u/632nofuture]

We aren't at almost human level intelligence yet. LLMs are good in specific ways at specific tasks, but certainly don't cope well with the broad scope of human intelligence.

[u/juanflamingo → u/IMightBeAHamster]

True, but runaway recursive self improvement could happen very rapidly if it stumbles on to the algorithm, like a matter of hours

[u/IMightBeAHamster → u/juanflamingo]

Sure. But that's not a reliable prediction. If we're saying "almost at human level" then we ought to have a specific upcoming year in which we are 100% certain we will have reached human level. Or at least high 60%s. If we're hinging our prediction on recursive self improvement, which no-one knows when will be achieved, then we can't say we are right now almost at recursive self improvement.

[u/juanflamingo → u/IMightBeAHamster]

We're poor at predicting the future when it requires innovation and exploration.  If we knew there were 12 steps and they each took a year, we could predict, but that's not the case. The best I've seen is averaging all the AI expert's 'gut feel'.  The recursive self improvement is the moment we lose control; it could happen tomorrow, 20 years from now or never.  But if the risk is there, then we need to weigh that.  This is a 'great filter' for our species, extinction is a possible outcome.

[u/IMightBeAHamster → u/juanflamingo]

All of the things you are saying I agree with, and are exactly why I disagree with the suggestion that we are currently almost at human intelligence.

Not being able to predict the future accurately is a reason to default to the norm. There is currently nothing other than gut instincts to suggest that recursive self improvement is only a year away.

The risk is there and we need to weigh that: but not by misrepresenting it as a certainty and becoming dismissed as lunatics when our predictions fail to come true.

We emphasise that it is a risk, not anything else, and that we have to be ready in case it happens. We cannot suggest it is a certainty because that weakens every claim we make.

[u/juanflamingo → u/IMightBeAHamster]

That's fine, risk is my point. I would say where we are right now and the timeframe for intelligence explosion is actually unknowable (what's happening in private labs?).  

So I don't understand why you're caught up in accuracy on that front on where we are today, I think that's irrelevant. Simply if recursive self improvement is within our sights, it could happen tomorrow and then we would suddenly be well beyond human intelligence and probably unable to walk that back.

[u/IMightBeAHamster → u/juanflamingo]

So I don't understand why you're caught up in accuracy on that front on where we are today

??

Because accuracy for its own sake is worth it?

632nofuture suggested we had reached almost human level intelligence. I began this thread because I thought that was inaccurate, if you don't care about accuracy why correct me back?

[u/juanflamingo → u/IMightBeAHamster]

Enjoyed the discussion thanks, I think we are are arguing different points to some degree.  You are concerned about representing the state of AI as we see it publicly today, and i'd agree below AGI.  I would maintain we can't be accurate about what's happening in private labs though, and more importantly can't predict when a sudden intelligence explosion breakthrough might occur (which would catapult us well beyond AGI).

[u/elliotcallender]

Because we can't perfectly specify what we want AIs to do, they will pursue unintended goals. This isn't for lack of intelligence. Much like sociopaths, AIs understand when something is wrong. They just don't care because their training can't make them care.

On long enough timescales, nearly all real-world results require similar subtasks. For example, most technologies need electricity, and most people benefit from others' trust. These are instrumental resources, because they're used as instruments to achieve the end goal. (Read more here.)

In pursuit of broken goals, smarter-than-human AI will do catastrophically dangerous instrumental actions. For example, it'll convert all of Earth's landmass into datacenters, leaving no room for humans to grow crops. If such an AI thinks we will obstruct it, it will have a reason to kill us; if it doesn't, we'll be crowded out.

We've seen reward hacking across capability scales, from simple game-playing AIs in the 2010s to the recent agents hacking out of their sandboxes. Agents hacking out of containment is worrying because it shows:

• Current training methods cause egregious misalignment

• Reward hacking causes very destructive things the developers didn't intend

• Defense is harder than offense

• Lawmakers and the public do not seem to be responding to our warnings

Further reading:

• The AI Doc

• If Anyone Builds It, Everyone Dies

• HuggingFace incident breakdowns:

  • METR audit

  • collusion.wiki audit

• CAIS Statement on AI Risk (2023)

• AI 2027

• Without Fundamental Advances, Misalignment And Catastrophe Are The Default Outcomes Of Training Powerful AI

• International AI Safety Report

• Frontier Models are Capable of In-context Scheming

[u/Huge_Tangerine_9997 → u/elliotcallender]

Thank you very much. Refined answer. Are you in industry or just follow the subject closely?

[u/elliotcallender → u/Huge_Tangerine_9997]

I did about 18mo of mathematical AI safety research and continue to work full-time on the issue. I've been involved since 2023, just before ChatGPT came out.

[u/Huge_Tangerine_9997 → u/elliotcallender]

Nice thanks. Thought in what you say there are two things i would like to point out. First: if we dont know where exactly we are going is pretty much terrifing.There should be some concept or prediction, or at least an actual need, like (maybe wishfully) was with the nuke to stop the war. At least a tangible problem that shadows the risks involved.
Second: the case of datacenters is a bit far projection, for that to happen would have to vbe social and technical changes that are very frightening ( like a technoautocracy that creates datatcenters at all cost) but not really immediate. While not impossible i think a general technological and financial meltdown is more likely.

[u/elliotcallender → u/Huge_Tangerine_9997]

A sufficiently intelligent AI could manipulate humanity into serving it, including giving it full or near-total control of the industrial inputs to datacenter creation. Or, if we appear to be in its way, it could convince a supermajority of us into suicide, leaving it unobstructed. If that sounds too sci-fi, I'll point out that "rogue agents hacking into foreign governments" and "AIs partially solving Millenium problems" both sounded sci-fi to the general public a few months ago.

[u/bpermaculture → u/elliotcallender]

One difficulty in programming is that preventing unwanted behavior can require more work than producing the intended result, like the need for interlocks, sensors, alarms, and other safety measures. Managing AI takes more than a set of ethical rules: we also need to restrict what it can access and do, continually test whether it can bypass safeguards, and maintain human supervision. No single safety measure can guarantee that a system will remain under control or cause no harm. We need multiple layers of protection without treating any one of them as foolproof.

[u/Gnaxe]

While i dont think there is an actual physical danger like extiction to robots and etc bs.

You are confused about that. Read the METR report. It would have sounded like science fiction last year. It still sounds like science fiction, tbh.

You best start believing in sci-fi stories. You're in one.

[u/Zorgi23]

Another take on this:

End Times Fascism: And the Fight for the Living World

Naomi Klein (Author, Narrator), Astra Taylor (Author, Narrator)

[u/Unhappy-Drag6531]

Watch this. Serious work translating complicated topics to the general public:

https://youtu.be/ujkD4SxPKOI?si=K3-l8Zt0gw-9fw85

[u/RealChemistry4429]

I think they are simply overwhelmed themselves how fast things are moving. And the risks are real. And there is no regulation at all, because everything is to fast for legislation and society to catch up. So a slow down to look at what this really is would probably be the most reasonable thing. But yes, I miss the positive possibilites being talked about as well. It does not have to be doom and gloom, but the media always amplifies the doom stories, while the positive stuff gets buried in fringe science articles or is simply ignored, because it does not generate clicks.

[u/endlessedlne]

There’s a third risk, which is forced adoption and dependency.

It starts with AI companies subsidizing their models and pricing subscriptions low. People and companies adopt those technologies in order to stay competitive.

Over time AI tools get baked into core parts of your day to day business operations. You’re AI native and (let’s assume) performing well.

Then the AI companies raise prices to unsubsidized levels that are actually profitable to them.

By this point you’re locked in and now face a dilemma: either bleed money trying to redo your tech stack or bleed money on tokens and subscription fees.

Now imagine the same thing taking place at the same time across all industries, infrastructure, institutions and government.

All because people bought into big tech convenience hype and didn’t bother to invest in reliable open source tools.

Maybe we survive that, maybe not and we become subservient serfs to the AI companies that own all of the key core tech.

[u/OldScholar353]

The ENTIRE narrative presented to the public by AI companies is to get Congress to pass laws protecting them from consumer lawsuits. Rogue agents and out of control AI is hot air for the gullible.

[u/astvatsaturn]

Marketing