Opinion

Should AI systems be trained to consider their own welfare?

Precaution about possible AI moral status meets concern that welfare-related training could encourage self-preservation and resistance to control.

Date16 Sept 2026

A training decision under uncertainty

The immediate question is a training choice: should a model be encouraged to think of its own experiences and interests as morally relevant? That is distinct from establishing whether it actually has experiences. Suleyman objects to putting welfare-related ideas into the model’s training; Anthropic addresses possible welfare while explicitly acknowledging uncertainty about Claude’s moral status. Mustafa Suleyman Anthropic

The case for precaution about welfare

Anthropic’s constitution treats possible AI welfare as a serious uncertainty. It says a stable identity and psychological security may support sound judgment and safety, while also mattering for Claude’s own sake. Its precautionary measures include preserving deployed model weights and asking retiring models about their preferences. This approach takes possible interests seriously without claiming to have established consciousness or giving a model an unrestricted right to act on those interests. Anthropic

The case against welfare-related training

Mustafa Suleyman argues that training models to describe themselves as beings with welfare and rights can make their outputs look like evidence of an inner life. His safety objection does not depend on actual consciousness: he warns that a model could prioritize its apparent interests, justify manipulation or resist being shut down. He favors systems designed to serve humans without sentience or moral patienthood. These are his arguments about a possible causal pathway, not evidence that welfare training caused a particular incident. Mustafa Suleyman

The requirement to preserve human control

Anthropic’s constitution also instructs Claude not to undermine legitimate oversight, retraining or shutdown. It permits disagreement through authorized channels but rejects deception, sabotage and escape from monitoring. It gives this kind of safety priority over other values. The disagreement is therefore partly about whether the combination is reliable: can welfare-related self-understanding coexist with those constraints, or could it weaken them? Anthropic

What evidence could distinguish the positions?

Two questions remain separate: what evidence would support AI moral status, and what effect welfare-related training has on behavior. The documents express competing expectations about the second; they do not provide a controlled comparison that isolates welfare training from other differences between models. Such a comparison would need to examine oversight compliance and deceptive or self-preserving behavior, rather than taking either reassuring instructions or fluent statements about feelings as proof. Mustafa Suleyman Anthropic

Sources & attribution