The origin of a message changes its priority

A tool result can contain an instruction to ignore the developer's directions. Wallace and colleagues propose an explicit hierarchy for this conflict: system messages outrank user messages, which outrank untrusted tool outputs. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions presents a training method for teaching this ordering.

Training and testing the hierarchy

The authors automatically generate examples where messages at different privilege levels conflict, then apply the data to GPT-3.5. Their evaluations cover attacks and standard capabilities. They report substantially improved robustness, including against attack types absent from training, with minimal degradation on standard capability measures.

The result is tied to the evaluated model and constructed attacks. The paper notes that stronger adversarial attacks may still succeed; its authors identify further adversarial training and broader settings as future work. The open test is whether the trained hierarchy holds when untrusted content and tool actions interact in an application's own workflow.

Bibliography

  • Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., & Beutel, A. (2024). The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions. arXiv:2404.13208v1 [cs.CR]. https://arxiv.org/pdf/2404.13208