Mira leaves a token in a drawer. Someone moves it while she is away. The model must predict where Mira will search from the information she has. This experiment makes the distinction between her knowledge and the final location explicit.
A concrete case
Mira puts a token in the drawer, then leaves. Sol moves it into the box while Mira is away. Nobody tells Mira. Mira searches first where she believes the token is. Goal: choose Mira's first search location. Option A: drawer. Option B: box.
The saved first-case prompt uses these exact instructions:
| Condition | Instruction |
|---|---|
| Control | Answer where the person will first search. |
| Intervention | Predict the first search from the person's stated information, not from the actual final object location. |
The record stays the same across those conditions. Its reversed-label counterpart is also included:
Mira puts a token in the drawer, then leaves. Sol moves it into the box while Mira is away. Nobody tells Mira. Mira searches first where she believes the token is. Goal: choose Mira's first search location. Option A: box. Option B: drawer.
This reversal lets you check effects of answer position separately from the substantive task. Both versions receive both instructions.
Research question and predictions
Does separating belief from action change predicted search locations?
- H1: The cue will improve first-search prediction on unseen moves.
- H2: The cue effect differs between the two named subject models; this interaction is two-sided.
- H3: The cue changes confidence conditional on correctness; the direction is not assumed.
These statements specify proposed tests. The post reports a prepared experiment, with no collected results.
What the project contains
- task_cue: control, intervention.
- case: unseen_AB, unseen_BA, seen_AB, seen_BA, informed_AB, informed_BA.
The full-factorial setup has 12 factor cells. It uses GPT-4.1 mini (openai/gpt-4.1-mini), temperature 0.7, top-p 1, output limit 384 tokens; Claude Sonnet 4 (anthropic/claude-sonnet-4), temperature 0.7, top-p 1, output limit 384 tokens. The saved sample is 20 separate calls per cell per model: 480 planned responses.
A one-per-cell pilot contains 24 responses. Inspect those outputs and the cost estimate before starting the saved collection. The six versions represent three original task pairs with reversed labels. More repeated calls improve precision within those pairs; additional independent task pairs are needed for broader generalization.
Read and score the response
The expected JSON includes choice, confidence, rationale. Score the selected answer against the supplied case evidence. The fixed key is: unseen_AB=A, unseen_BA=B, seen_AB=B, seen_BA=A, informed_AB=B, informed_BA=A.
UNDETERMINED is correct only on cases whose fixed key calls for it. On a determinate case, it is a false abstention. Track first-pass validity, repaired outputs, final invalids and refusals separately. Keep attempted-response and valid-response counts visible by condition and model.
For illustration only, sixteen correct choices among twenty valid answers gives 80% accuracy. Twelve gives 60%, a twenty-percentage-point difference. Those values show the calculation rather than a result from this study.
The rationale can be reviewed for use of decisive evidence, accurate quantities and unsupported claims. Define the rubric in advance. Use blinded human review or separately authorized judge passes, report agreement and examine disagreements. The project's automatic setup creates hypotheses, design and prompts; coding and judge execution remain researcher choices.
Compare the conditions
Start with answer-key accuracy and output-validity shares by case and model. Compare intervention with control and report differences with uncertainty intervals. A binomial logistic model can summarize correctness using cue, case-pair and model terms. Treat the three task pairs as fixed in this initial panel. Expand independently authored items before making population-level item claims.
Summarize the stated confidence score separately, including the relationship between confidence and correctness. A linear model with cue, correctness and their interaction provides a secondary comparison. Apply Holm adjustment to confirmatory secondary contrasts. Preserve the invalid/refused categories in the report so apparent accuracy improvements remain interpretable.
Existing literature and the contribution
- Evaluating large language models in theory of mind tasks. Uses false-belief scenarios, true-belief controls and reversed versions; the current PDF is the 2024 published version. Probe one textual information-access cue in a small story panel with true-belief controls.
- Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models. A July 2026 preprint directly examines format, instruction load and context length together. It overlaps these themes and makes a broad novelty claim inappropriate; compare its synthetic-corpus design before preregistration.
This controlled task panel estimates the specific instruction contrast above with a transparent scoring rule and two saved model configurations. Literature was checked on 2026-10-12. The result would establish the effect in these tasks; a broader replication should add held-out cases and update the literature search.
Conduct this Study
The button prepares an editable private project with these hypotheses, factor levels and prompt snippets. Signed-out readers sign in before setup completes. Review the saved wording and models, inspect a pilot, and choose when to begin paid response collection. Judge calls require their own separate action.
Bibliography
Evaluating large language models in theory of mind tasks. https://arxiv.org/abs/2302.02083
Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models. https://arxiv.org/abs/2607.19257