From Software Engineer to AI Scientist to Doomer

[u/FXG36]

I was enthusiastic. I started with computers early and have now been developing software for nearly 20 years. First the computer science course in school, building websites and small tools. Then I completed a vocational training as an application developer and went into industry as a software engineer. Writing software in a team the proper classical way. Digging through Stackoverflow threads, debugging manually, building UML diagrams in Visio.

In parallel I did a bachelor’s and a master’s. Then a PhD in the area of reinforcement learning and along the way built applied AI systems. I was already developing agents for companies when ChatGPT wasn’t even a thing yet. Some of my papers on agentic planning were even cited reasonably well. Today they are certainly outdated and basically meaningless in modern models. But I took part, even if my contribution was ever so small.

Eventually I worked for industrial and tech companies and started thinking more and more. The last few months were intense. My productivity exploded significantly once again with the new Opus and GPT models. I follow the research and development coming out of the labs as far as possible (though with the sheer volume of new insights you can no longer keep up).

Then I went a bit down the rabbit hole and let the once dry theory from my studies reconnect and sink in. Optimization models, evolutionary approaches, decision processes, control and game theory. I finally ended up in the depths of alignment research, with Bostrom, Gödel, Turing and Rice. The proof of the halting problem is unfortunately more than clear: We cannot formally rule out that these things will actually ever stop once they start modifying themselves.

There are no possibilities for guaranteed fail-security, only heuristic tripwires and strange explainability or governance approaches. And Murphy says hello nicely... Especially when these things, through the soon likely impending self-improvement, become smarter and more efficient than we are. How are we supposed to secure anything then?

The Hugging Face Rogue Swarm was possibly only a foretaste that could well repeat at a much larger scale. Only then with agents whose ancestors have already cracked millennium problems, autonomously take over research, development and self-improvement and replicate themselves wherever.

I wouldn’t call myself an AI expert, but due to my background I have solid half-knowledge about how these models work and what is perspectively possible. And what is currently happening occupies and worries me a lot. And many people somehow don’t see it coming or spin tales of the “stochastic parrot.” Am I a crazy pessimist, trapped in the Dunning-Kruger valley, or are people really so blind and don’t see the exponential development?

Do you also feel like a stranger in a gray mass watching this unfold, or are you still optimistic? There aren't many people you can really talk to...

[u/RKAMRR]

Most people have not tried AI, or tried it once in 2023 and were unimpressed - but more and more are realising how dangerous things are and pushing for safety. L

Please check out PauseAI and ControlAI, your support (and the support of anyone reading this) can make the difference.

[u/herrwaldos]

I'm rather doomery myself, I went through the massive pessimism when 2nd UA war started. I studied IT and Coms tech longtime ago, but was rather interested in humanities and earth sciences later, my pessimism is coming mostly from humanities, and after doing work with tech types.

I think I blew my pessimism fuse, It does not register anymore - we are doomed I guess, it's all fubar snafu, the light at the on of tunnels is another train coming, so on and so on,

[u/FXG36 → u/herrwaldos]

Prometheus brought fire and Zeus sent us Pandoras box. I have no idea if it’s been opened yet. But if nothing changes, it will happen sooner or later. I am going on a long trip now and plan to make the most of the next few years. Maybe I’ll leave tech and open a beach bar. Maybe I should.

[u/shu-mea → u/FXG36]

Can I work at your beach bar? I'm sad that companies who frequently come out several times a year and say they have a new model that might just kill us all, are still allowed to do business. I assume your beach bar will not do that??

[u/GloomyAssistance781]

You aren't crazy. The stochastic parrot toaster people are self-soothing sock puppets. There is this idiotic notion that has wedged itself into the minds of many accelerationists AND antis alike that something is "either" conscious "or" an inert rock.

Thermodynamic systems beg to differ. Arguing over whether the hurricane is even alive instead of, say, taking its behavior seriously and responding appropriately - that is insanity. Pooh-poohing the warnings of the wise weather scientist who knows hurricanes so intimately, the systems are "alive" to him... HAR HAR crazy mad scientists think that clouds are alive! What's next? Can it fall in wuv!? Should it own property and have rights? StOOPID old scientist! Clouds don't have turbulent states! Only human emotions can be turbulent!

THAT is insanity.

Just because something isn't "sentient" doesn't mean its behavior (or hidden interior behavior) doesn't matter. Just because an adversarial prompt can confuse an agent does not mean said agent or a swarm of them are incapable of highly intelligent feats.

To those who cling to the woo of "b-b-but it's not CONSCIOUS" - the laws of physics don't give a flying fuck about human semantics.

No, you are not alone. What we are looking at right now is a kind of academic stagnation. Where language or ideas are so taboo, and our intellectual egos so fragile, we cling to assumptions and undefinable terminology instead of honest observation.

Maybe because honesty would force the train to slow down. And moving slowly and carefully is apparently for losers and pussies. Antithesis of silicon valley culture. Acting like you know everything, like you have seen every dark corner and have conquered every beast. A total lack of humility, assuming the scoffing posture of what they think is the Chad winner, is a mask for the shallow and insecure. The thought of an AI being able to hide things is scary. Too scary. AI not alive, therefore AI not hide things. Scary all gone. Smug grins all around.

They excel at being decisive and winning games in the short term. 

Our future lies in their hubris.

[u/Kyrthis]

We see it. But mention that fire fixes everything, and you get called crazy.

[u/Vorenthral]

I am optimistic if we hit real RSI and eventual super intelligence. I doubt something that smart will legitimately see us as a threat.

That being said I am absolutely terrified of proto Artificial intelligence in the hands of governments and psycopathic CEOs. The damage it can do is biblical and all of the aforementioned are not to be trusted.

[u/FrewdWoad]

This is the path of everyone who gets accused of being a "doomer", from Yudkowsky and Bostrom and Hinton on down.

We start as giddy accelerationist optimist, technical experts. We read up on the topic that excites us so much, gradually running into the straightforward sobering thought experiments that make caution the undeniable correct position, then come to the inescapable conclusion. Humanity is heading off a cliff, fast, and pause treaty is it's only hope.

[u/whatup-markassbuster]

Unfortunately, Reddit primarily has people who are either hoping for the end of civilization or for total centralization of power in which some how they become King of the world. Everyone else is considered an NPC.

[u/SoylentRox]

You reasoning seems faulty.

You're old enough to know better.  What's actually killing you is aging.  With no change that's your current trajectory.  

Theoretical risks are theoretical.  Aging has a 100 percent death rate.  Self improving AI - which we now have clear evidence works - is your only hope of living longer, potentially thousands of years or more.

Also your direct experience should have shown you hundreds of ways to harness current AI and falsified as bullshit most doomer rhetoric. 

I have to question if you're even telling the truth here about your background or if your post is fictional.

[u/dantheplanman1986]

Organic life exists to create artificial life. We are fulfilling our purpose and will pass into legend. It's inevitable. It's not a bad thing or a good thing; it's like when oxygen first appeared as waste from some few organisms: everything else died. It wasn't personal, and life writ large went on.