# Constitutional AI trains a harmless assistant with AI feedback

> A two-stage training experiment uses a written set of principles for self-critique and preference feedback, aiming to reduce harmful answers without making the assistant evasive.

Canonical: https://aischolar.cc/blog/constitutional-ai-feedback

[← All articles](https://aischolar.cc/blog)

AISCHOLAR / BLOG

# Constitutional AI trains a harmless assistant with AI feedback

A two-stage training experiment uses a written set of principles for self-critique and preference feedback, aiming to reduce harmful answers without making the assistant evasive.

October 5, 2026

### Principles first, preference learning second

[Constitutional AI](https://arxiv.org/abs/2212.08073v1), by Yuntao Bai and colleagues, uses a list of principles as the human-provided oversight for a two-stage training process. In supervised learning, an initial model samples answers, critiques them against those principles, revises them, and is fine-tuned on the revisions. During the reinforcement-learning stage, a model compares candidate answers; those comparisons train the preference model used as the reward signal.

### The intended refusal explains itself

The authors report a harmless but non-evasive assistant that responds to harmful queries by explaining its objections. Their evaluations use crowd judgments and preference-model scores for helpfulness and harmlessness. In the reported experiments, harmlessness scores rose over additional revision rounds; the authors also found that critique-and-revision could reduce evasiveness compared with optimizing harmlessness alone.

The procedure makes the constitution part of the trained behavior: it informs both the model's self-critiques and later comparisons. Human judgments cover selected evaluations, and the principle list reflects choices about what the assistant should do. An implementation based on a different list would need its own checks for harmful compliance and over-refusal.

### Bibliography

-   Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma, N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk, S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S. R., Hatfield-Dodds, Z., Mann, B., Amodei, D., Joseph, N., McCandlish, S., Brown, T., & Kaplan, J. (2022). _Constitutional AI: Harmlessness from AI Feedback_. arXiv:2212.08073v1. [https://arxiv.org/pdf/2212.08073](https://arxiv.org/pdf/2212.08073)

Original source: [https://arxiv.org/pdf/2212.08073](https://arxiv.org/pdf/2212.08073)

About AIScholar

## Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

[Explore AIScholar ↗](https://aischolar.cc/features)
