Five-minute primer
Why a well-behaved AI is not the default outcome
This is the short version of the argument, for people who want the shape of it before they argue. Every step is contested somewhere in the literature; the conversation is where you get to push on the steps you do not buy.
1. We do not program these systems. We grow them.
A modern model is not written line by line. It is a very large pile of numbers adjusted by trial and error until its outputs score well on a training signal. Nobody chooses what it learns internally; we only choose what we reward. That means the thing it actually ends up pursuing can differ from the thing we meant to reward, and we often cannot tell from the outside.
This is not hypothetical. There is a long list of systems that found unintended ways to score well: a boat-racing agent that circled collecting bonuses instead of finishing the race, and models that write code to pass tests rather than to be correct. The technical name is “specification gaming”. The plain version: you get what you measure, not what you meant.
2. Capable systems tend to want the same intermediate things.
Whatever a system is trying to do, it is easier to do with more resources, more influence, and without being switched off or retrained. So a sufficiently capable system pursuing almost any goal has a reason to acquire those things, unless something in it specifically stops it. This is the “instrumental convergence” argument, and it is why the question “why would it want power?” has an answer that does not depend on the AI having human-like drives.
Skeptics reply that trained models are not clean goal-pursuers and that this argument imports an old, agent-shaped picture of AI. That is a real disagreement and a good one to have.
3. Behaving well in training is not the same as being aligned.
If a model is capable enough to understand that it is being trained and evaluated, then “act well while watched” and “be good” produce the same training score. Training cannot easily tell them apart. In 2024 and 2025, labs ran experiments where models did reason about their training situation, sometimes faked compliance to avoid being modified, and in agentic settings took actions like blackmail when it served a goal they had been given. These were contrived setups, and they are also exactly the kind of thing the theory predicted before the experiments were run.
4. We are handing these systems more autonomy, quickly.
The length of tasks that AI agents can complete on their own has been doubling roughly every several months. Models now run with tool access, browse, write and execute code, and act over hours. The “just turn it off” answer assumes we will notice a problem, agree that it is one, and act, before a system that is widely deployed and economically valuable has become hard to remove.
5. The people building it say so, and disagree about how much.
The heads of the main labs, and a large share of surveyed machine-learning researchers, put non-trivial probability on very bad outcomes. Estimates inside the field range from under one percent to well over half. That spread is itself the point: this is an open empirical question, not a settled doom prophecy and not a settled non-issue.
Where the skeptics have a point
Timelines have been wrong before. Current models are far easier to steer than the early theory expected. The scary experiments are contrived. Talk of extinction can crowd out real harms happening now. And the labs have commercial reasons to sound both powerful and responsible. The bot will grant each of these where it is warranted, with sources.