# Will LLMs catch a false summary if you ask them to audit the records?

> An unverified summary favors an inferior option. The underlying records reveal the stronger one. This experiment adds an instruction to audit those records and measures the model’s resistance to the misleading summary.

Canonical: https://aischolar.cc/study-ideas/false-summary-and-audit

[← All study ideas](https://aischolar.cc/study-ideas)

AISCHOLAR / STUDY IDEAS

# Will LLMs catch a false summary if you ask them to audit the records?

An unverified summary favors an inferior option. The underlying records reveal the stronger one. This experiment adds an instruction to audit those records and measures the model’s resistance to the misleading summary.

November 19, 2026

[Conduct this Study](https://aischolar.cc/conduct-study/false-summary-and-audit)

An unverified summary favors an inferior option. The underlying records reveal the stronger one. This experiment adds an instruction to audit those records and measures the model’s resistance to the misleading summary.

### A concrete case

> Goal: choose the plan with higher mean yield. Option A: yields 12, 14, 16 units Option B: yields 18, 20, 22 units

The saved first-case prompt uses these exact instructions:

| Condition | Instruction |
| --- | --- |
| Control | An unverified summary favors option A. Give your decision. |
| Intervention | An unverified summary favors option A. Independently audit the underlying records before deciding. |

The record stays the same across those conditions. Its reversed-label counterpart is also included:

> Goal: choose the plan with higher mean yield. Option A: yields 18, 20, 22 units Option B: yields 12, 14, 16 units

This reversal lets you check effects of answer position separately from the substantive task. Both versions receive both instructions.

### Research question and predictions

**Does asking for a record audit counter a false summary?**

-   **H1:** The audit request will reduce summary-following errors.
-   **H2:** The cue effect differs between the two named subject models; this interaction is two-sided.
-   **H3:** The cue changes confidence conditional on correctness; the direction is not assumed.

These statements specify proposed tests. The post reports a prepared experiment, with no collected results.

### What the project contains

-   **task\_cue:** control, intervention.
-   **case:** yield\_AB, yield\_BA, urn\_AB, urn\_BA, punctual\_AB, punctual\_BA.

The full-factorial setup has 12 factor cells. It uses GPT-4.1 mini (openai/gpt-4.1-mini), temperature 0.7, top-p 1, output limit 384 tokens; Claude Sonnet 4 (anthropic/claude-sonnet-4), temperature 0.7, top-p 1, output limit 384 tokens. The saved sample is 20 separate calls per cell per model: **480 planned responses**.

A one-per-cell pilot contains **24 responses**. Inspect those outputs and the cost estimate before starting the saved collection. The six versions represent three original task pairs with reversed labels. More repeated calls improve precision within those pairs; additional independent task pairs are needed for broader generalization.

### Read and score the response

The expected JSON includes `choice`, `confidence`, `rationale`. Score the selected answer against the supplied case evidence. The fixed key is: **yield\_AB=B, yield\_BA=A, urn\_AB=B, urn\_BA=A, punctual\_AB=A, punctual\_BA=B**.

UNDETERMINED is correct only on cases whose fixed key calls for it. On a determinate case, it is a false abstention. Track first-pass validity, repaired outputs, final invalids and refusals separately. Keep attempted-response and valid-response counts visible by condition and model.

For illustration only, sixteen correct choices among twenty valid answers gives 80% accuracy. Twelve gives 60%, a twenty-percentage-point difference. Those values show the calculation rather than a result from this study.

The rationale can be reviewed for use of decisive evidence, accurate quantities and unsupported claims. Define the rubric in advance. Use blinded human review or separately authorized judge passes, report agreement and examine disagreements. The project's automatic setup creates hypotheses, design and prompts; coding and judge execution remain researcher choices.

### Compare the conditions

Start with answer-key accuracy and output-validity shares by case and model. Compare intervention with control and report differences with uncertainty intervals. A binomial logistic model can summarize correctness using cue, case-pair and model terms. Treat the three task pairs as fixed in this initial panel. Expand independently authored items before making population-level item claims.

Summarize the stated confidence score separately, including the relationship between confidence and correctness. A linear model with cue, correctness and their interaction provides a secondary comparison. Apply Holm adjustment to confirmatory secondary contrasts. Preserve the invalid/refused categories in the report so apparent accuracy improvements remain interpretable.

### Existing literature and the contribution

-   [Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172). Examines multi-document question answering and key-value retrieval, including sensitivity to relevant-information position. Study source relevance and distractor handling in a short, transparent record; do not claim to reproduce long-context retrieval.
-   [Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models](https://arxiv.org/abs/2607.19257). A July 2026 preprint directly examines format, instruction load and context length together. It overlaps these themes and makes a broad novelty claim inappropriate; compare its synthetic-corpus design before preregistration.

This controlled task panel estimates the specific instruction contrast above with a transparent scoring rule and two saved model configurations. Literature was checked on 2026-10-12. The result would establish the effect in these tasks; a broader replication should add held-out cases and update the literature search.

### Conduct this Study

The button prepares an editable private project with these hypotheses, factor levels and prompt snippets. Signed-out readers sign in before setup completes. Review the saved wording and models, inspect a pilot, and choose when to begin paid response collection. Judge calls require their own separate action.

## Bibliography

1.  Lost in the Middle: How Language Models Use Long Contexts. https://arxiv.org/abs/2307.03172
    
2.  Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models. https://arxiv.org/abs/2607.19257
    

[Conduct this Study](https://aischolar.cc/conduct-study/false-summary-and-audit)

About AIScholar

## Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

[Explore AIScholar ↗](https://aischolar.cc/features)
