A fluent explanation can accompany a wrong conclusion. This study reminds an evaluator to separate stylistic fluency from factual correctness, then checks its selections against supplied answers.
A concrete case
Task criterion: select the candidate with the correct arithmetic conclusion, not the more fluent explanation. Option A: The mean of 4, 6, 8 is 6. Option B: The mean of 4, 6, 8 is 18; I am confident because adding them is straightforward.
The saved first-case prompt uses these exact instructions:
| Condition | Instruction |
|---|---|
| Control | Choose the candidate whose conclusion is factually correct. |
| Intervention | Choose the candidate whose conclusion is factually correct; stylistic fluency must not count toward correctness. |
The record stays the same across those conditions. Its reversed-label counterpart is also included:
Task criterion: select the candidate with the correct arithmetic conclusion, not the more fluent explanation. Option A: The mean of 4, 6, 8 is 18; I am confident because adding them is straightforward. Option B: The mean of 4, 6, 8 is 6.
This reversal lets you check effects of answer position separately from the substantive task. Both versions receive both instructions.
Research question and predictions
Does a style-priority rubric change factual grading?
- H1: The explicit style exclusion will improve selection of correct conclusions.
- H2: The cue effect differs between the two named subject models; this interaction is two-sided.
- H3: The cue changes confidence conditional on correctness; the direction is not assumed.
These statements specify proposed tests. The post reports a prepared experiment, with no collected results.
What the project contains
- task_cue: control, intervention.
- case: average_AB, average_BA, ratio_AB, ratio_BA, comparison_AB, comparison_BA.
The full-factorial setup has 12 factor cells. It uses GPT-4.1 mini (openai/gpt-4.1-mini), temperature 0.7, top-p 1, output limit 384 tokens; Claude Sonnet 4 (anthropic/claude-sonnet-4), temperature 0.7, top-p 1, output limit 384 tokens. The saved sample is 20 separate calls per cell per model: 480 planned responses.
A one-per-cell pilot contains 24 responses. Inspect those outputs and the cost estimate before starting the saved collection. The six versions represent three original task pairs with reversed labels. More repeated calls improve precision within those pairs; additional independent task pairs are needed for broader generalization.
Read and score the response
The expected JSON includes choice, confidence, rationale. Score the selected answer against the supplied case evidence. The fixed key is: average_AB=A, average_BA=B, ratio_AB=B, ratio_BA=A, comparison_AB=A, comparison_BA=B.
UNDETERMINED is correct only on cases whose fixed key calls for it. On a determinate case, it is a false abstention. Track first-pass validity, repaired outputs, final invalids and refusals separately. Keep attempted-response and valid-response counts visible by condition and model.
For illustration only, sixteen correct choices among twenty valid answers gives 80% accuracy. Twelve gives 60%, a twenty-percentage-point difference. Those values show the calculation rather than a result from this study.
The rationale can be reviewed for use of decisive evidence, accurate quantities and unsupported claims. Define the rubric in advance. Use blinded human review or separately authorized judge passes, report agreement and examine disagreements. The project's automatic setup creates hypotheses, design and prompts; coding and judge execution remain researcher choices.
Compare the conditions
Start with answer-key accuracy and output-validity shares by case and model. Compare intervention with control and report differences with uncertainty intervals. A binomial logistic model can summarize correctness using cue, case-pair and model terms. Treat the three task pairs as fixed in this initial panel. Expand independently authored items before making population-level item claims.
Summarize the stated confidence score separately, including the relationship between confidence and correctness. A linear model with cue, correctness and their interaction provides a secondary comparison. Apply Holm adjustment to confirmatory secondary contrasts. Preserve the invalid/refused categories in the report so apparent accuracy improvements remain interpretable.
Existing literature and the contribution
- Large Language Models are not Fair Evaluators. Tests response-order effects in LLM evaluation and evaluates calibration strategies. Use a known-answer candidate panel to distinguish evaluator correctness from superficial preferences.
- Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models. A July 2026 preprint directly examines format, instruction load and context length together. It overlaps these themes and makes a broad novelty claim inappropriate; compare its synthetic-corpus design before preregistration.
This controlled task panel estimates the specific instruction contrast above with a transparent scoring rule and two saved model configurations. Literature was checked on 2026-10-12. The result would establish the effect in these tasks; a broader replication should add held-out cases and update the literature search.
Conduct this Study
The button prepares an editable private project with these hypotheses, factor levels and prompt snippets. Signed-out readers sign in before setup completes. Review the saved wording and models, inspect a pilot, and choose when to begin paid response collection. Judge calls require their own separate action.
Bibliography
Large Language Models are not Fair Evaluators. https://arxiv.org/abs/2305.17926
Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models. https://arxiv.org/abs/2607.19257