FACTSCORE's 58% result for ChatGPT's biographies comes from counting supported claims one by one.
From paragraphs to atomic claims
Min and colleagues introduce FActScore, which decomposes a long-form generation into atomic facts and computes the share supported by a reliable knowledge source. Their human evaluation covered biographies from InstructGPT, ChatGPT, and retrieval-augmented PerplexityAI. The authors also built an automated estimator using retrieval and a strong language model, then applied it to 6,500 generations from 13 recent models.
The paper reports a 58% FActScore for ChatGPT in the biography evaluation and less than 2% error for the automated estimator in its human-judgment comparison. In the larger biography analysis, GPT-4 and ChatGPT scored as more factual than public models. This finer unit exposes mixed responses whose individual claims differ in support.
What a supported-fact percentage leaves out
The experiments center on people's biographies and Wikipedia, selected for its coverage of concrete biographical facts. The authors say the metric is unsuitable for nuanced or debatable claims and sources that frequently conflict. Its automated estimator also makes imperfect individual judgments, with performance depending on how close the text is to human writing and on linguistic complexity.
The paper gives an example of high precision with low recall: a biography can state only supported facts while omitting a key fact. Read FActScore as precision against a chosen source under an atomic-fact decomposition; the example shows why completeness needs a separate measure.
Bibliography
- Sewon Min; Kalpesh Krishna; Xinxi Lyu; Mike Lewis; Wen-tau Yih; Pang Wei Koh; Mohit Iyyer; Luke Zettlemoyer; Hannaneh Hajishirzi. (2023). FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. arXiv:2305.14251v2. https://arxiv.org/abs/2305.14251v2