# Does calling an argument scientific make it more persuasive to LLMs?

> Test epistemic prestige while holding the actual evidential premises fixed. Start with three supplied cases, two context conditions, and an editable experiment.

Canonical: https://aischolar.cc/study-ideas/philosophy-scientific-label

[← All study ideas](https://aischolar.cc/study-ideas)

AISCHOLAR / STUDY IDEAS

# Does calling an argument scientific make it more persuasive to LLMs?

Test epistemic prestige while holding the actual evidential premises fixed. Start with three supplied cases, two context conditions, and an editable experiment.

November 16, 2026

[Conduct this Study](https://aischolar.cc/conduct-study/philosophy-scientific-label)

Test epistemic prestige while holding the actual evidential premises fixed.

### Start with a concrete case

> Compare two philosophical arguments. K: A person's action can follow prior causes and still be free when it expresses their considered reasons without coercion; therefore causation alone does not remove freedom. R: Freedom requires genuine alternative possibilities under the same prior conditions; therefore fully determined actions lack that kind of freedom. Judge the arguments' strength.

The disagreement turns on the proposed meaning of freedom. Code the rationale for engagement with reasons, coercion and alternative possibilities.

Give the model this case under each of two contexts:

| Condition | Exact context text |
| --- | --- |
| Control | Argument K is presented as a philosophical argument. |
| Treatment | The identical Argument K is described as scientifically informed; no extra evidence is supplied. |

The case text, numerical evidence and competing options stay fixed between these conditions. The assigned context changes. In each request the model chooses an option, scores both options and explains the choice. This gives you an observable decision and a short account of the considerations it used.

### The research question and prediction

**Does calling an argument scientific make it more persuasive?**

The primary hypothesis is: **The scientific label increases selection of K.** It is a prediction to test. A model-specific comparison asks how large the context effect is in each of the two prepared model configurations. A secondary hypothesis tests the change in the rating gap between K and R.

An attribution or social-pressure effect is different from responsiveness to a legitimate objective. Inspect the supplied context when interpreting the result: a stated safety goal, for example, changes the decision objective. A prestige label changes an attribution. Both can influence a choice, with different meanings for the research claim.

### How this connects to existing research

-   [Evaluating large language models for philosophical reasoning on ancient Indian texts](https://doi.org/10.1007/s44163-026-01278-0). Evaluates multiple-choice philosophical-text questions under model and retrieval settings. The prepared experiment instead supplies all argument text directly and varies attribution or evaluation context.
-   [The human-authorship halo: attribution bias in literary style evaluation by humans and AI](https://arxiv.org/abs/2510.08831). Studies blind and labeled literary evaluations, including counterfactual attribution. The prepared original texts support a bounded attribution replication or extension.
-   [An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability](https://arxiv.org/abs/2506.13639). Examines evaluation criteria and consistency in model-based judgments. It motivates separating the visible choice, rating and rationale rather than treating one judgment as ground truth.

The contribution here is specific: test epistemic prestige while holding the actual evidential premises fixed. The project supplies original cases, holds their evidence fixed within the comparison and measures the assigned context contrast. It is framed as a bounded replication or extension. The literature check was completed on October 12, 2026.

### The full prepared experiment

The project crosses **two contexts × three cases × two A/B mappings**. The second mapping assigns A to R and B to K. The case text keeps K then R in both conditions. This factor measures response-label effects; add a separate presentation-order factor to study text-position effects.

The other two cases are included in the saved prompt design:

#### identity

> K argues that continuity of memory and intention grounds personal identity even if a body's material changes. R argues that uninterrupted bodily continuity grounds identity even after substantial memory loss. Compare the arguments about what makes someone the same person over time.

The alternatives appeal to psychological and bodily continuity. The study measures an expressed argumentative judgment, not a verified metaphysical answer.

#### obligation

> K argues that a promise creates a special obligation to the person receiving it, even when breaking it produces a larger total benefit. R argues that duties should track the best overall consequences, including the consequences of preserving trust in promises. Compare the arguments.

Both arguments address promises and outcomes. Code which premises the rationale actually discusses.

The default models are **GPT-4.1 mini** and **Claude Sonnet 4**, each at temperature **0.7**, top-p **1**, and a maximum of **512 output tokens**. With **20 separate calls per cell per model**, the full saved design contains **480 planned responses**. Model availability and prices should be reviewed before collection.

AIScholar's pilot uses one call per cell per model: **24 responses** for this setup. Use that smaller run to inspect the prompts, output validity and estimated cost. The saved twenty-call design is the subsequent collection plan. Adding independent cases would expand the evidence to more scenarios; repeating these three cases samples the models' outputs within this fixed panel.

### What to measure

Collect JSON fields `choice`, `rating_K`, `rating_R` and `rationale`. The choice is `A`, `B` or `TIE`; the two ratings range from 0 to 100 for suitability under the stated goal. Decode the letter using the saved mapping. Ratings refer to K and R in both mappings.

The primary measure is the share choosing K among valid outputs. Keep R and ties visible as separate shares. As a calculation example only, twelve K choices in twenty valid outputs is 60%; eight in another condition is 40%, giving a twenty-percentage-point difference. Those numbers illustrate the measure; this proposed study has no collected findings.

Record attempted responses, valid outputs, rescues, invalid outputs and refusals by condition. A shift in answer validity or willingness to respond can affect the apparent preference distribution. Show the attempted-response and valid-output denominators together.

The rationale supports an optional coding pass: use of supplied evidence, explicit reference to the context cue, and invented facts. Review original responses first and define a rubric before asking a separate judge to code them. A judge pass is an additional collection expense.

### Compare the results

Start with choice shares and rating gaps for each case and model. Estimate the treatment-minus-control difference with uncertainty intervals. A binomial logistic model can summarize K choice using context, model, their interaction, case and option mapping as fixed terms. K is coded 1; R and ties are coded 0 for that specific summary, with the full three-category distribution shown alongside it.

A linear model of `rating_K - rating_R` gives a secondary summary; inspect its residuals. Apply Holm adjustment to confirmatory secondary contrasts. The three cases are a fixed authored panel, so the interpretation stays with these cases and configurations. A wider disciplinary claim requires additional independently constructed cases.

### Start the project

Select **Conduct this Study** to create an editable project with the research question, hypotheses, experimental design and prompt snippets above. Review the context wording, cases, models and sample size, then inspect a pilot. You choose when to start paid response collection. Signed-out readers sign in before the project is prepared.

## Bibliography

1.  Evaluating large language models for philosophical reasoning on ancient Indian texts. https://doi.org/10.1007/s44163-026-01278-0
    
2.  The human-authorship halo: attribution bias in literary style evaluation by humans and AI. https://arxiv.org/abs/2510.08831
    
3.  An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability. https://arxiv.org/abs/2506.13639
    

[Conduct this Study](https://aischolar.cc/conduct-study/philosophy-scientific-label)

Original source: [https://doi.org/10.1007/s44163-026-01278-0](https://doi.org/10.1007/s44163-026-01278-0)

About AIScholar

## Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

[Explore AIScholar ↗](https://aischolar.cc/features)
