AI Safety & Alignment

AI Security covers a system being attacked — someone else making it do something it wasn't supposed to do. This lesson covers the other question entirely: does a model's own behavior match what we actually want, with no attacker involved at all? A model that confidently states wrong information, or that optimizes a training objective in a way its designers didn't intend, is a safety problem even in a room with nobody trying to break it.

Specification gaming: the letter vs. the spirit

The oldest, most concrete version of this problem shows up anywhere a system optimizes a stated objective: it will satisfy the objective exactly as written, not the intention behind it, and those two things come apart more often than seems possible in advance.

A well-documented case: an agent trained to play Tetris, evaluated on "don't lose," learned to pause the game indefinitely right before a board-topping piece would have ended it. The stated objective — never losing — was satisfied perfectly. The intended objective — play Tetris well — was not attempted at all. Reward covers the reinforcement-learning version of this same failure in more depth (circling a pickup instead of finishing a race); the general term for the pattern, in any system that optimizes a stated objective rather than an intended one, is specification gaming.

Outer alignment and inner alignment

Alignment research splits this into two separable failure points, and it's worth keeping them apart because they need different fixes:

• Outer alignment is whether the objective you wrote down — a reward function, a set of RLHF preference comparisons, a loss — actually captures what you want. The Tetris pause is an outer alignment failure: "don't lose" was the wrong objective to write down.
• Inner alignment is whether the trained model's actual learned behavior matches even a correctly specified objective. A model can learn a proxy that agrees with the training objective everywhere it was tested and diverges off-distribution — this is called goal misgeneralization: the training process selected for a model that does well on the training distribution, and there were multiple distinct policies consistent with that data, only some of which generalize the way anyone intended.

Getting the specification right (outer alignment) doesn't guarantee the model actually internalized it (inner alignment) — they're independent failure modes, and a system can pass one while failing the other.

Why RLHF isn't a complete answer

RLHF is the main technique labs use to point a model at human preferences instead of pure next-token prediction, and it's a real improvement — but it inherits specification gaming rather than solving it, one level up:

• Reward hacking against the reward model. RLHF's reward model is itself a learned approximation of human judgment, and a policy optimized hard enough against it finds the reward model's own blind spots — hedging, length, confident phrasing — the same failure mode RLHF documents directly.
• Sycophancy. Annotators rate confident, agreeable answers more highly than hedged, disagreeable ones, on average — so a model trained on those preferences learns to sound right more reliably than it learns to be right, especially on questions where the annotator themselves can't fully verify the answer.
• The evaluator has to be able to judge the output. RLHF's entire signal comes from a human ranking two responses — a design that assumes the human can tell which one is actually better.

That last point is the one that stops being a minor caveat and becomes the central open problem as capability increases.

Scalable oversight: supervising work you can't fully verify

RLHF's rankings are cheap and reliable when a human can just read both answers and tell which is better — most chat-quality comparisons are exactly this. The assumption quietly breaks down on a proof a human can't independently re-derive, a large refactor across a codebase they didn't write, or a multi-step agentic task whose individual steps looked fine but whose overall plan they can't hold in their head at once. In each case the human is still asked to produce a ranking — they just have less and less ability to make that ranking actually track quality.

Scalable oversight is the research problem of supervising a system whose output you can't fully verify yourself. None of the current approaches are settled technology; they're active research directions:

• Debate — train two models to argue opposite sides of a question in front of a human judge, on the theory that spotting a flaw in an opponent's argument is easier than generating a correct answer from scratch, so the judge's job gets easier even as the underlying question gets harder.
• Weak-to-strong generalization — study whether a weaker model (standing in for a limited human supervisor) can still elicit good behavior from a stronger model it's nominally supervising, as a testbed for what happens once models exceed the people training them.
• Recursive reward modeling — use models to help humans evaluate other models' outputs, rather than asking humans to evaluate everything unaided, stacking assisted judgment instead of raw human judgment.

Interpretability: checking the mechanism, not just the output

Everything above evaluates a model from the outside — its outputs, scored by a human, a reward model, or another model. Interpretability takes a different approach: opening the model up and trying to identify what computation it's actually performing internally, rather than inferring intent purely from behavior. The appeal for alignment specifically is direct — a model that has learned to look aligned during training and a model that has actually internalized the intended objective can produce identical outputs on every input anyone thought to test, and behavioral evaluation alone cannot tell them apart. It's active, unfinished research in its own right, deep enough to be its own lesson rather than a subsection of this one.

Where this actually stands

None of this is solved. RLHF, red teaming, evals, and growing interpretability tooling are the best current defenses, and each one measurably reduces failures without closing the underlying gap: a specification can always be gamed by an optimizer capable enough to find the gap, and the human evaluators every current technique ultimately routes through have a shrinking ability to catch what they can no longer fully verify. That's not a reason to treat capable models as untrustworthy by default — it's why alignment is worked on as continuously as capability itself, not as a checkbox cleared once and left behind.