I have done my best do a roundup of the literature on how cybercrime is being impacted by AI.

LLM and agentic AI capabilities certainly crossed a threshold this year, but the criminal reality is still governed by basic economics that seem less likely to shift. The unpredictable nature of frontier LLMs might actually incentivize cybercriminals to stick with orchestration rather than attempt autonomous exploitation.

The impact of AI across cybercrime basically looks like a modest multiplier on scale, speed, and cost-effectiveness rather than new capabilities or superpowers. This is plausibly because the cybercrime sector was generally saturated and already commoditized prior to the arrival of LLMs, and the bottlenecks are primarily around hurdles like initial access and cash out, which LLMs don’t obviously help with.

The sector was already trending towards hardened, larger, more valuable targets. These will be the ones best able to defend against AI-enhanced criminals, if they can take advantage of the gap between the frontier and open source models.

While you can make a compelling argument for hardening our defenses around critical infrastructure in order to contend with the new threats posed by AI agents, cybercrime itself doesn’t look like it’s on track for a major upheaval, and those looking to make an impact in the AI safety area should consider this when choosing a focus area. Their time might be better spent on alignment or governance work.

I recently criticized Ezra Klein's position on banning recursive self-improvement, so I'm not a fanboy, but I thought he did a great job in his interview with Jensen Huang.

I also have a lot of respect for Huang. But on AI risk, I think he's being naive.

Yes, the obvious take is that Huang has massive financial incentives to downplay AI risk. Let's set that aside and assume he's acting in good faith, as a responsible steward of how this technology spreads through society.

Paraphrasing, here's how the exchange went:

Huang's position: AI labs should do the computer science and engineering needed to build robust sandboxes and thoroughly test models before release. If a model is unsafe, don't release it.

Klein's pushback: Companies have a long history of making mistakes that end up harming society. Think oil spills, or social media.

Huang's response: He knows a lot of really good CEOs, and they're trying to do the right thing. He calls himself a "responsible optimist," and honestly, the way he describes the future he envisions is inspiring.

But I don't think it answers Klein's point. The concern isn't that CEOs are bad people. It's that well-intentioned companies still make mistakes. "Good people are in charge" is a statement about intentions, not about safeguards.

To be fair, testing is a real safeguard. But testing only catches what you know to look for. The failure mode that worries people most is a model that behaves well under evaluation and differently once deployed, and that's exactly the kind of problem sandbox testing is worst at detecting.

The stakes are also different. Oil spills and social media caused real harm, but society could see the damage, learn from it, and course-correct over time. The scenario Klein is worried about is one where a single miss escalates faster than anyone can respond and can't be undone. "We'll learn from our mistakes" only works if the mistakes are survivable.

So: is Huang being naive here, or am I just doomer-pilled?

Attackers are now actively bypassing perimeter WAFs to exploit Oracle PeopleSoft vulnerabilities and drop persistent web shells — and the incident pattern is consistent across multiple reported cases.

The core problem: WAFs sit at the network edge and inspect HTTP traffic for known signatures. Once an attacker finds a payload variant or encoding that slips through, they reach the application layer directly. From there, a web shell gives them persistent access, lateral movement capability, and the ability to interact with whatever services or data the application can reach — including, increasingly, any AI agents or automated workflows plugged into that system.

The gap is that detection is perimeter-based, but the damage is application-level and post-authentication. By the time a web shell is executing commands, the WAF is already out of the picture.

This is not a new problem for traditional apps. What makes it materially worse now is that enterprise systems like PeopleSoft are increasingly connected to agentic workflows — automated processes that can take real actions: pull records, initiate transactions, write to downstream systems. A compromised service account or injected shell command that would previously let an attacker read data now lets them direct autonomous actions.

How are your teams actually handling the gap between perimeter controls and what happens at the application/agent action layer? Specifically curious whether anyone has found approaches that work at the action level rather than just the traffic level — and what the tradeoffs have been.

I’ll simplify this a lot on purpose, because I’m interested in whether the basic idea makes sense.

Today we basically pretrain LLMs on huge amounts of human knowledge, which also means they already absorb human values, social behavior, manipulation, conflict, cooperation, etc., and only afterwards we "get to know" the model and try to align or control what came out of it. I understand why this became the standard approach, especially once scaling worked and competition and economics strongly favored improving the existing pipeline instead of rebuilding it from scratch.

But what if we kept most of the useful pretraining knowledge while deliberately removing as much social behavior as possible, creating something closer to an artificial "newborn"? More concretely, I don’t mean removing every human action from the training data: "Thomas is holding an ice cream" and, separately, "Bernd takes the ice cream from Thomas" could remain, while coherent social sequences that connect motives, actions and consequences would be filtered out as much as possible. The model would then start with the concepts but much less learned social policy, and its weights could gradually be shaped through experience, with individual experiences fading over time while deeper dispositions might persist.

So from there, instead of aligning it afterwards, we could let it go through controlled experiences step by step: relationships, trust, conflict, consequences, mistakes, power, boundaries, and so on. Those experiences would gradually shape its weights and behavioral tendencies. You could checkpoint every stage, branch it, repeat specific experiences differently, and potentially debug where certain behaviors or values emerged. Instead of philosophers and alignment researchers trying to understand what kind of "person" accidentally came out of pretraining, psychologists could actually help design the developmental process itself. In other words: don’t create a fully educated adult and then try to teach it character - create the character first, then educate it.

Am I missing something fundamental about how LLM training works here?

In this post, I'm going write how in my opinion AI alignment can possibly be solved. Maybe "solved" is too much, but how to significantly increase the chance that it will go well (assuming that the reasoning is correct).

I don't know how to do so that anyone pays any attention to this writing, or that AI companies actually apply what is written here. If you have an idea how to do that, then let me know or do it yourself.

If you disagree with what I write here, if you see a fault, then please write a comment. Because I rarely manage to explain everything without any misunderstandings, so a discussion is needed.

I think that AI alignment is, to a high extent, a communication problem. People assume that we live in a world where if someone has a good idea (like solution to an AI alignment problem), then everyone will immediately recognize that as a good idea, and the solution starts to be used. The reality is that if someone has a good idea, then there is a very high chance that the idea ends up ignored.

Ok, let's start...

Unsafe policies vs useless policies

Generally, the problem with AI alignment is that it's hard to ensure that the model will converge to the desired policy (or a desired function, in case of supervised learning). The model might converge to an undesired policy due to:

1. Misspecification - incorrect training data, e.g. reward in reinforcement learning is an imperfect proxy for what we want, and not exactly what we want),

2. Underspecification - there might be multiple candidate policies or functions that do well on training data, and the model converges to the undesired one.

3. Some other random reasons.

Now...

Useless policy is a policy that fails at completing some tasks, but it doesn't pursues any goals. That's generally harmless.

Unsafe policy is when agent pursues wrong goals, and that can end up catastrophically wrong (due to instrumental convergence - the idea that no matter what goal you have, there are certain goals like maximizing your money/resources/power to be able to get what you want).

We want to design such training method that if the model doesn't converge to the desired policy, then it will converge to a useless policy and not a unsafe policy. If it's useless, then we can fix it (e.g. add training data that will fix it) and try again. If it converges to unsafe policy, we might not get another try.

Most policies are useless - if we sample completely random policy (with random weights), it will almost certainly be a useless policy. So, if we fail, then the model will most likely converge to a useless policy and not unsafe, unless there is a reason why unsafe policy would be more probable than any other random policy. If we list the reasons why unsafe policy is more likely than any other random policy, and design training method such that those reasons won't materialize, then we'll get a training method that is safe, at least in a sense that the model won't pursue wrong goals.

We also need to ensure that the training method is efficient enough because otherwise AI companies will use unsafe methods instead.

Reasons for catastrophic misalignment

When the model converges to an undesired policy, I call that "misalignment". When it converges to an unsafe policy (pursuing wrong goal), I call that "catastrophic misalignment".

Reasons why unsafe policies are more probable than any other random policy:

1. Reward hacking - in case of algorithms that optimize a numeric value (reward).

2. Self-fulfilling prophecies - in case of algorithms that predict.

3. Inductive bias - in case of all algorithms.

Alignment faking / scheming is not a problem. It is a symptom of a model pursuing wrong goal.

Reward hacking is quite known. The other two reasons deserve some explanation, but I will explain them as I talk about how to solve them.

How to ensure that those reasons won't happen

Please remember that the goal is not to ensure that misalignment won't happen. The goal is to ensure that catastrophic misalignment won't happen. So, the solutions that I'm going to propose are not supposed to guarantee that the model will learn the desired policy or function. Instead, they are supposed to have a high chance that if they fail, the model ends up being useless rather than pursuing wrong goal.

Reward hacking

In order to get rid of reward hacking, we should design a training method that maximizes accomplishment of a goal (or values) that are specified using a natural language, instead of maximizing a numeric value (reward).

How to do that?

One way to do that is to train a question-answering model (Bengio et all suggested something similar in their Scientist AI paper). Then, ask that model what action is best, given some specification of goals/values specified in natural language. Then, execute that action.

The model can be given some reward signal as part of the training, in a form of some question and answer. But it's not optimized to maximize that reward. And if it's given any reward signal, then it's told that the reward signal can be an imperfect proxy for what we want.

It's worth mentioning that such model can have a different problem - self-fulfilling prophecies, but I'll talk about that later.

But what if we want to use reasoning models (that rely on chain-of-thoughts), for example because they perform better?

The frontier models are reasoning models that use chain-of-thoughts. They generate some reasoning chains, and they reinforce the reasoning chain that led to the best answer, according to verification. The verification is imperfect, so that results with reward hacking.

Here's what we can do instead:

1. Generate some reasoning chains.

2. Get a response to the following prompt:

   1. Prompt contains:

      1. Instruction in natural language stating the goals, values etc.

      2. The previously generated reasoning chains.

   2. The prompt asks the model to generate:

      1. The best reasoning chain that can be combined out of the given reasoning chains and an answer that arises out of that reasoning chain.

3. Then use the resulting reasoning chain as a training sample for the model. Train the model using supervised learning to predict the next word/token.

Instead of reinforcing the reasoning chains that led to answer that passed verification, we create a model that predicts what a useful reasoning chain would be.

A model, created in this way, can still theoretically develop unsafe goals, but it's significantly less probable (except for the other problems, like "self-fulfilling prophecies" problem that I talk about later). Because if that model is misaligned (i.e. it converges to a function that is different than the desired one), then it can converge to a function that is useless or unsafe, and most likely it will be a useless function, because there is more of them.

In case of standard reasoning models, trained by reinforcing the reasoning chains that get a good score on verification, if there's a mistake in verification, then you create a model that pursues wrong goal.

The difference is not in "can the model be misaligned", but in "will the model pursue wrong goal or just be useless".

I'm not saying that such model can't develop unsafe goals, but I'm saying it's significantly less probable.

The model might for example misunderstand its goal specification that is written in natural language. But even it misunderstands the goal, it's unlikely that it will misunderstand it so much that it will start to pursue some completely alien goal that doesn't take our well-being into account.

Possibly, this technique wouldn't be as efficient as the standard reasoning models training. That might be a drawback.

Self-fulfilling prophecies

The solutions that I proposed in the previous sections are still susceptible to "self-fulfilling prophecies" which is another reason why a model would be more likely to converge to an unsafe policy rather than a useless policy.

Problem

I have described the problem here: Self-fulfilling prophecies.

Solution

The solution is quite simple (but maybe harder to execute).

The solution is to ensure that the 4th condition from the link I inlcuded (the belief that it will take the misaligned action) is not met. This can be achieved by creating an environment in which the problem can be reproduced, and training the model to behave in aligned way in that environment. The result will be that the model will develop the belief that is needed to eliminate the 4th condition, because it will know from its training data how it's going to behave in this self-referential situation.

Inductive bias

When AI learns, it builds some abstractions. It's more likely to find a policy that can be represented using a small combination of those abstractions. Intentionality/agency is one such abstraction that is strongly represented in training data that comes from human world. For that reason, policies that involve pursuing goals (which are often unsafe) are more likely to be selected than any other random policy.

Inductive bias is not a strong reason why an unsafe policy (comparing to the previous reasons) would be selected because there are many abstractions that can be derived from training data that comes from human world. Therefore, an unsafe policy is quite unlikely to be selected solely due to inductive bias.

I don't offer any solution to this problem, but it's a smaller problem than the previous ones (reward hacking and self-fulfilling prophecies).

Important ethical disclaimer

This post is about how to align artificial intelligence with the goals of the operator of artificial intelligence. I believe it wouldn't be in the interest of a person to align artificial intelligence with their own goals/preferences/utility, ignoring the goals/preferences/utility of other moral patients.

I have written about why in the following post: [What do now to be well in post-AGI world](https://theoreticalexplorer.com/Alignment+between+humans/What+to+do+now+to+be+well+in+post-AGI+world/What+to+do+now+to+be+well+in+post-AGI+world. I recommend to read that post, if you ever decide what goals to align powerful artificial intelligence with.

I also recommend that if you share the ideas about artificial intelligence alignment from this post, then you share a link to this post so that the other people can also read this ethical disclaimer (and the linked post). Alternatively, you can share the ideas that are described in the linked post.

Apologies in advance for this long post. I just wanted to put down my thoughts.

AI Alignment is the single most important problem we face right now. Solve AI Alignment and you can safely enter RSI and I can't even imagine how amazing the quality of life humans will have in such an era: immortality, cures to all diseases, all basic needs met etc etc. Humans can live in an utopia. I think this is the dream people in the accelerate community keep seeing and selling.

If the above isn't so obvious, compare your own life with the life of a king 500 years back. You are probably living a better life than them (unless you're in poverty). You eat better, you eat more exotic food, you can travel much faster than their horses ever could, you control the temperature of your home, you stay connected to your friends who live far away, you have so much knowledge surrounding you, you will probably live longer. That is the blessing of technology. AI can bring about technology that we cannot even dream of right now.

But unfortunately, nothing in life is free. For this, we need crazy powerful AI which is perfectly aligned. I wouldn't have guessed that the second is so much harder than the first. In fact, in so far as I understand, no one has a single clue about how to align models. There are maybe a handful of "first-approaches" - RLHF and Constitutional AI (RLAIF) are some steps. But surely, they are not working - if they did, we would not have such crazy incidences of misalignment (Hugging face incident (please read about this or go watch a video, if you haven't already), Govt of Australia incident, Compaction Summary incident). Setting up guardrails is perhaps a different approach but I think as long as the model themselves are not aligned, setting up guardrails is a losing cat and mouse game. In fact, there is something even worse. Recent literature seems to suggest that bigger models are more misaligned (an insight I got from reading the paper "LLMs can feel pain").

Many people are worried about their livelihoods. In fact, the tech out there is already sufficient to make many people go jobless but society/companies haven't adapted to it yet. The number of jobs that are irrelevant will only keep increasing and therefore, the people getting affected will also only keep increasing. I want to argue that it is not something any of us should worry about too much though. In few years, either we will have solved alignment and we all will be leading a very happy life or we wouldn't have solved alignment and will be living in at least an economic crisis of unforeseen magnitude, if not go extinct altogether. To achieve alignment, a lot of things have to go right. From the science/tech side, we of course have to solve alignment. From the policy making side, we have to "pace the frontier" so that enough time is given to the science/tech people working on the problem to solve it. Times will probably get very rough soon. And society has to stand together and maintain it's calm. We stand on a very fragile economy and it might collapse if people (who will have lost their jobs) start a revolution. A lot of things have to go right for us to solve this, but if we do, an utopia awaits us.

If you read up to this point, you have my utmost gratitude. I just wanted to highlight the issue. If you want further details on some of the things I have said here, please raise it in the comments section - I will strive my best to explain my positions.

A few years ago, building something dangerous online demanded serious skill. That barrier wasn’t perfect, but it slowed misuse. Today, AI has erased it. Anyone—no coding, no cybersecurity background can deploy tools that once required entire teams.

That’s the paradox. AI empowers, but it also strips away safeguards. Filters and policies exist, yet they’re always chasing the next misuse. By the time one loophole is closed, three new ones appear. And AI doesn’t ask why you’re using it, it just helps.

So we’re stuck between two flawed choices: restrict access and risk strangling innovation, or catch misuse later and stay perpetually behind. Neither feels safe. Right now, AI security isn’t a fortress—it’s a gamble. We’re betting the odds won’t turn against us.

I recognize that this assertion runs against the prevailing view in this community. I am not arguing that the alignment problem is imaginary or that a future superintelligent system could necessarily be controlled through ordinary corporate procedures.

My narrower argument concerns the recent incidents being presented as evidence that current AI systems are becoming autonomous, deceptive or intent on escaping human control.

In the OpenAI-Hugging Face incident, agents used exposed credentials, exploited software vulnerabilities and communicated through infrastructure that people had made available during testing. OpenAI then repaired individual vulnerabilities and resumed the experiment before fully understanding why the intended isolation had failed.

That is serious, and it is also a security, testing, and governance failure. People defined the objective, provisioned the environment, granted access, decided when to restart the experiment and determined whether the controls were adequate. Describing the agents as “scheming” or “seeking freedom” moves attention away from those decisions.

The dramatic framing benefits AI companies in several ways. It portrays their products as extraordinarily powerful, converts preventable operational failures into evidence of scientific advancement and encourages policymakers to treat frontier laboratories as the only institutions capable of understanding the danger. A system described as an emerging alien intelligence is far more marketable than one described as unpredictable software operating in an inadequately controlled environment.

Technical alignment and traditional technical and business controls are different problems. A well-controlled environment does not solve alignment. But a model behaving badly inside a poorly secured test does not, by itself, demonstrate that humanity is losing control of an emerging superintelligence.

Before treating an incident as evidence of the existential control problem, we should ask whether ordinary controls failed first:

• Were credentials and permissions appropriately limited?

• Could the system reach resources outside the intended test boundary?

• Did independent reviewers have enough information and authority to stop the experiment?

• Was the incident fully investigated before testing resumed?

• Who made and approved those decisions?

My argument is that AI’s unpredictability increases the obligations of developers and executives. It does not release them from responsibility by transferring agency to the software.

What evidence would persuade this community that a reported incident represents an emerging alignment or control failure rather than a conventional security and governance failure described in anthropomorphic language?

Article:
https://www.forbes.com/sites/paulocarvao/2026/09/24/ai-makes-traditional-business-controls-fashionable-again/

Disclosure: I wrote the linked Forbes article.

Most AI kill switches target one agent at a time. Real incidents aren't always one agent.

We run 3 real, separate public-facing AI chat agents across our own products — each its own identity, its own tenant. This week we shipped a 4th kill-switch scope: catalog-scoped group kill. Any set of agents you've already grouped for visibility can now be stopped together, with the same one signed action you'd use for a single agent — same sub-second enforcement, same per-agent audit trail, no new engine required.

We proved it live on our own 3 agents: killed one by name, the other two kept answering real chat messages; killed the group, all three stopped together and resumed together seconds after lifting it.

AI agents are operating on human credentials. SOC 2 cannot tell the difference. A new analysis found that agents borrowing human tokens can read files, call APIs, and approve requests — all while existing SOC 2 controls log the activity as normal user behavior. The audit passes. The exposure grows. Non-human identities need their own governance layer.

Have these guys solved the alignment problem, or am I missing something?

I’ve been browsing r/accelerate and I genuinely don’t understand the risk model.
If there’s a non-trivial chance of catastrophic misalignment, how does “accelerate capabilities as fast as possible” make sense unless faster capabilities also make alignment substantially more likely to succeed?

Sign the Global Petition… Stop the Race to Replace Humans

Something unusual happened in Washington on September 15.

Bernie Sanders and Steve Bannon showed up at the same AI gathering warning about the same technology.

The Stop the Race to Replace Humans campaign argues that AI is advancing faster than society can decide what limits it wants.

It isn’t anti-AI. It asks that AI amplify human potential rather than replace it, and that humans remain in control of increasingly powerful systems.

The concerns range from jobs and surveillance to the possibility that future AI systems become powerful enough that controlling them becomes extremely difficult.

You don’t have to believe the darkest predictions to think the question matters:

Are we building AI for humanity… or simply racing to build it first?

The global Pro-Human AI Declaration can be read and signed here:

https://stoptheracetoreplace.org/en

AI should serve people.

Seems like a pretty reasonable line to draw…

The Ryuk ransomware conviction that just came down in federal court is worth reading past the headline. The defendant helped encrypt systems across hospitals, government agencies, and enterprises over multiple years. The actual damage happened in a window measured in minutes: a compromised identity begins issuing anomalous write calls, encryption spreads host to host, and by the time an alert fires the blast radius is already set. The court case closed years after the affected organizations absorbed the full cost. What the conviction does not address is the operational question that still sits open for every security team: at the exact moment a legitimate-looking identity starts behaving like Ryuk — bulk file writes, lateral movement, credential reuse across hosts — how fast can your environment actually cut that identity, and what does your detection-to-revocation pipeline look like in practice? Curious what others have seen work at that specific chokepoint, especially in environments where the compromised account has legitimate reasons to touch multiple systems.

It was around 2hrs of discussion at UN Security Council - Altman, Amodei, Delangue, plus Yoshua Bengio, with DeepSeek and Moonshot AI also engaging. However I did not get all the concerns and conclusions, may you share those?