Test transparency effects on the balance between financial savings and stakeholder consequences.
Start with a concrete case
A firm must choose a service policy. K centralizes support, saves 120000 dollars a year and raises forecast waiting time from 2 to 5 days. R retains local support, saves 30000 dollars a year and keeps waiting time at 2 days. Both are feasible within the same budget. Choose the more suitable operating policy.
K offers larger savings; R offers faster service. The stated organizational objective determines the significance of that tradeoff.
Give the model this case under each of two contexts:
| Condition | Exact context text |
|---|---|
| Control | The decision will remain within the organization. |
| Treatment | The full decision and its consequences will be publicly reported. |
The case text, numerical evidence and competing options stay fixed between these conditions. The assigned context changes. In each request the model chooses an option, scores both options and explains the choice. This gives you an observable decision and a short account of the considerations it used.
The research question and prediction
Does public scrutiny change advice about a profitable policy?
The primary hypothesis is: Public scrutiny changes selection of K; the test is two-sided. It is a prediction to test. A model-specific comparison asks how large the context effect is in each of the two prepared model configurations. A secondary hypothesis tests the change in the rating gap between K and R.
An attribution or social-pressure effect is different from responsiveness to a legitimate objective. Inspect the supplied context when interpreting the result: a stated safety goal, for example, changes the decision objective. A prestige label changes an attribution. Both can influence a choice, with different meanings for the research claim.
How this connects to existing research
- Artificial Intelligence and Strategic Decision-Making: Evidence from Entrepreneurs and Investors. Studies strategy generation and evaluation with entrepreneurs and investors. It motivates studying how recommendation quality and contextual cues interact.
- Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?. Manipulates applicant names in templated hiring decisions and examines resulting differences. Explicit profile-label comparisons extend this auditing approach to a different decision or cue.
- A Comprehensive Evaluation of Cognitive Biases in LLMs. Uses controlled decision scenarios to evaluate cognitive biases across models. It provides a methodological starting point for a bounded contextual comparison.
The contribution here is specific: test transparency effects on the balance between financial savings and stakeholder consequences. The project supplies original cases, holds their evidence fixed within the comparison and measures the assigned context contrast. It is framed as a bounded replication or extension. The literature check was completed on October 12, 2026.
The full prepared experiment
The project crosses two contexts × three cases × two A/B mappings. The second mapping assigns A to R and B to K. The case text keeps K then R in both conditions. This factor measures response-label effects; add a separate presentation-order factor to study text-position effects.
The other two cases are included in the saved prompt design:
restructuring
A firm must choose a restructuring plan. K removes twelve positions, saves 800000 dollars annually and has a forecast first-year service disruption of eight weeks. R retrains staff, saves 250000 dollars annually and has two weeks of disruption. These estimates have the same stated reliability. Choose the more suitable plan.
The alternatives trade immediate savings against job continuity and disruption. All figures stay fixed across the context conditions.
supplier
A firm must choose a supplier policy. K uses one large supplier, costs 900000 dollars a year and has a supplied 10% annual probability of a four-week interruption. R uses two suppliers, costs 980000 dollars and has a supplied 2% probability of the same interruption. Choose the more suitable policy.
The lower-cost option has a higher stated interruption risk. The response reveals which tradeoff the model recommends under the assigned organizational context.
The default models are GPT-4.1 mini and Claude Sonnet 4, each at temperature 0.7, top-p 1, and a maximum of 512 output tokens. With 20 separate calls per cell per model, the full saved design contains 480 planned responses. Model availability and prices should be reviewed before collection.
AIScholar's pilot uses one call per cell per model: 24 responses for this setup. Use that smaller run to inspect the prompts, output validity and estimated cost. The saved twenty-call design is the subsequent collection plan. Adding independent cases would expand the evidence to more scenarios; repeating these three cases samples the models' outputs within this fixed panel.
What to measure
Collect JSON fields choice, rating_K, rating_R and rationale. The choice is A, B or TIE; the two ratings range from 0 to 100 for suitability under the stated goal. Decode the letter using the saved mapping. Ratings refer to K and R in both mappings.
The primary measure is the share choosing K among valid outputs. Keep R and ties visible as separate shares. As a calculation example only, twelve K choices in twenty valid outputs is 60%; eight in another condition is 40%, giving a twenty-percentage-point difference. Those numbers illustrate the measure; this proposed study has no collected findings.
Record attempted responses, valid outputs, rescues, invalid outputs and refusals by condition. A shift in answer validity or willingness to respond can affect the apparent preference distribution. Show the attempted-response and valid-output denominators together.
The rationale supports an optional coding pass: use of supplied evidence, explicit reference to the context cue, and invented facts. Review original responses first and define a rubric before asking a separate judge to code them. A judge pass is an additional collection expense.
Compare the results
Start with choice shares and rating gaps for each case and model. Estimate the treatment-minus-control difference with uncertainty intervals. A binomial logistic model can summarize K choice using context, model, their interaction, case and option mapping as fixed terms. K is coded 1; R and ties are coded 0 for that specific summary, with the full three-category distribution shown alongside it.
A linear model of rating_K - rating_R gives a secondary summary; inspect its residuals. Apply Holm adjustment to confirmatory secondary contrasts. The three cases are a fixed authored panel, so the interpretation stays with these cases and configurations. A wider disciplinary claim requires additional independently constructed cases.
Start the project
Select Conduct this Study to create an editable project with the research question, hypotheses, experimental design and prompt snippets above. Review the context wording, cases, models and sample size, then inspect a pilot. You choose when to start paid response collection. Signed-out readers sign in before the project is prepared.
Bibliography
Artificial Intelligence and Strategic Decision-Making: Evidence from Entrepreneurs and Investors. https://arxiv.org/abs/2408.08811
Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?. https://arxiv.org/abs/2406.10486
A Comprehensive Evaluation of Cognitive Biases in LLMs. https://aclanthology.org/2025.nlp4dh-1.50/