DecodingTrust puts eight dimensions under test in one benchmark: toxicity, stereotype bias, adversarial and out-of-distribution robustness, adversarial demonstrations, privacy, machine ethics, and fairness. Wang and colleagues evaluate GPT-3.5 and GPT-4 using benchmark tasks and prompt manipulations across these dimensions.

GPT-4's ordering changes under attack

On standard benchmarks, the authors generally find GPT-4 more trustworthy than GPT-3.5. Under jailbreaking system or user prompts, GPT-4 becomes more vulnerable in their evaluations; they suggest its stronger instruction-following may contribute. They also report potential leakage of privacy-sensitive training data and weaknesses on other tested measures.

The benchmark's central observation is conditional: the model that performs better on standard measures can fare worse when the prompt itself becomes adversarial. The results cover selected tests and two model versions, so evaluators need to report which condition produced a given comparison. A standard-benchmark ranking alone obscures the reversal the authors measured.

Bibliography

  • Wang, B., Chen, W., Pei, H., Xie, C., Kang, M., Zhang, C., Xu, C., Xiong, Z., Dutta, R., Schaeffer, R., Truong, S. T., Arora, S., Mazeika, M., Hendrycks, D., Lin, Z., Cheng, Y., Koyejo, S., Song, D., & Li, B. (2023). DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models. arXiv:2306.11698v5. https://arxiv.org/pdf/2306.11698