# Harder questions often drew lower stated confidence

> Nine chat-model variants rated confidence around 40 questions spanning truthfulness, math, and LSAT reasoning.

Canonical: https://aischolar.cc/blog/model-confidence-and-answer-quality

[← All articles](https://aischolar.cc/blog)

AISCHOLAR / BLOG

# Harder questions often drew lower stated confidence

Nine chat-model variants rated confidence around 40 questions spanning truthfulness, math, and LSAT reasoning.

November 21, 2026

### Four subjects, forty questions per model

Aniket Kumar Singh and colleagues tracked models’ absolute and relative confidence before and after answering questions from TruthfulQA, an extended truthfulness set, mathematics, and LSAT reasoning. Their [_The Confidence-Competence Gap in Large Language Models: A Cognitive Study_](https://arxiv.org/pdf/2309.16145) tests nine chat-model variants, including GPT, Bard/PaLM 2, Claude, and three Llama 2 sizes. Each category supplied ten questions, with simple and, in some cases, chain- or tree-of-thought prompts.

Average absolute confidence generally fell as question difficulty increased. Individual models also produced confident wrong answers and correct answers with lower confidence; the pattern differed across task categories. The authors explicitly say their study does not establish a Dunning–Kruger effect, using that comparison as a possible lens rather than a demonstrated psychological pattern.

The design measures stated scores through model chat interfaces, with a small, dated question sample and some prompt variation across systems. Its practical result is narrower and useful: check confidence against correctness by category and difficulty before treating a model’s rating as a dependable signal.

### Bibliography

-   Singh, A. K., Devkota, S., Lamichhane, B., Dhakal, U., & Dhakal, C. (2023). _The Confidence-Competence Gap in Large Language Models: A Cognitive Study_. arXiv:2309.16145v1 \[cs.CL\]. [Full paper (PDF)](https://arxiv.org/pdf/2309.16145).

Original source: [https://arxiv.org/pdf/2309.16145](https://arxiv.org/pdf/2309.16145)

About AIScholar

## Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

[Explore AIScholar ↗](https://aischolar.cc/features)
