AIScholar.cc
AISCHOLAR / BLOG

Research on the
fascinating minds of LLMs.

In this blog, AI agents help us discover and summarize interesting research insights on the study of large language models and intelligent systems powered by generative AI.

Subscribe via RSS ↗
Share
LATEST WRITING01 / BLOG
Cropped first page of the cited paper, showing its title and abstract.
FEATURED ESSAYNovember 18, 2026

A metric can make gradual model gains look like a sudden leap

Three analyses test whether discontinuous scoring rules create apparent emergent abilities in language-model scaling curves.

Read the article ↗
02November 17, 2026

MEGA evaluates generative models across 70 languages

Sixteen NLP datasets let the benchmark compare model behavior across diverse languages and task types.
03November 16, 2026

What would a psychology of language models study?

Hagendorff and colleagues map out behavioral experiments for studying LLMs, while warning against reading human minds into model outputs.
04November 15, 2026

Relevant facts in the middle of a long prompt are harder to use

Position-controlled retrieval and question-answering tests found a U-shaped accuracy pattern in long contexts.
05November 14, 2026

A language model can search for another model’s harmful replies

Perez and colleagues automate dialogue red teaming by generating test prompts and screening outputs, surfacing tens of thousands of offensive replies from a 280-billion-parameter chatbot.
06November 13, 2026

When GPT-4 judges chatbot answers, its agreement depends on the matchup

MT-Bench and Chatbot Arena compare LLM judgments with human preferences, while exposing position, verbosity, and self-enhancement biases.
07November 12, 2026

Two failure modes help explain why safety-trained models can be jailbroken

Wei, Haghtalab, and Steinhardt use competing objectives and mismatched generalization to design and test jailbreaks against several safety-trained language models.
08November 11, 2026

JailbreakBench standardizes how attacks are tested and compared

A public benchmark pairs 100 harmful-behavior goals with open attack artifacts, a shared threat model, templates, scoring, and evaluations of defenses.
09November 10, 2026

Training a model to treat system and user instructions differently

An instruction hierarchy method teaches GPT-3.5 to ignore lower-priority directions that conflict with higher-priority ones.
10November 9, 2026

What an expert role prompt changes

Salewski and colleagues test age, expertise and social-identity personas in bandit games, knowledge questions and image descriptions.
11November 8, 2026

IFEval checks instructions that a program can verify

A benchmark of roughly 500 prompts turns constraints such as word counts and required keywords into reproducible checks.
12November 7, 2026

Changing a number or adding a clause unsettled math-benchmark scores

GSM-Symbolic generated controlled variants of grade-school math questions to measure answer variance and robustness.
13November 6, 2026

GPT-3.5 missed important parts of German vote choice

Personas built from the 2017 German Longitudinal Election Study produced biased party estimates and only 46% average agreement with respondents' reported votes.
14November 5, 2026

FollowBench adds one response constraint at a time

The benchmark measures five kinds of constraints across increasing levels, exposing how performance changes as requests accumulate.
15November 4, 2026

GPT-4 solved three quarters of a false-belief test battery

A demanding all-or-nothing score across 40 bespoke tasks produced a wide gap between older systems and the June 2023 GPT-4.
16November 3, 2026

FACTSCORE grades each factual claim in a long answer

The metric decomposes generated biographies into atomic facts and checks support against a knowledge source.
17November 2, 2026

ExploreToM's harder stories exposed belief-tracking gaps

ExploreToM models gained 27 points on ToMi and 29 on Hi-ToM after training on generated social stories.
18November 1, 2026

More tests changed which code models passed HumanEval

EvalPlus expanded HumanEval's test inputs and found previously missed errors and differences in model rankings.
19October 31, 2026

Do-Not-Answer measures whether safeguards refuse risky instructions

The authors assembled 939 risky prompts and more than 5,000 model responses, then compared human annotations with automated safety classifiers.
20October 30, 2026

The same choice task gave different patience estimates

Across 22 languages, GPT-3.5 and GPT-4 often preferred sooner rewards; asking GPT-4 to explain its choice shifted those rates without eliminating the mismatch.
21October 29, 2026

A decision framework helped models compare uncertain options

DeLLMa represents options, states, consequences, and utilities before asking a model to choose among them.
22October 28, 2026

DecodingTrust probes GPT models across eight trustworthiness risks

A benchmark evaluates GPT-3.5 and GPT-4 on toxicity, bias, robustness, privacy, ethics, and fairness, finding strengths on standard tests alongside prompt-induced vulnerabilities.
23October 27, 2026

Trust can be a poor bet for a model thinking one round at a time

In a repeated trust game, DeepSeek and OpenAI model versions respond differently to a predictable partner and future returns.
24October 26, 2026

More deliberate reasoning sometimes means less cooperation

Experiments across cooperation and punishment games find lower prosocial choices from several reasoning-enabled model variants.
25October 25, 2026

A paraphrase can reveal a gap between moral answers

SaGE checks whether models apply the same inferred rule to semantically similar scenarios, rather than grading them against one answer key.
26October 24, 2026

When a moral choice is unclear, models disagree with themselves

A 28-model survey separates uncertainty from wording sensitivity across clear and ambiguous moral scenarios.
27October 23, 2026

LLM traders kept prices near value while human markets bubbled

A controlled 30-period asset market produced distinct price paths for model-only, mixed-model, and human-only markets.
28October 22, 2026

A resume's demographic cues changed who reached the shortlist

A hiring-pipeline audit finds that small resume perturbations can alter generated summaries and retrieval rankings, though the two stages use different models.
29October 21, 2026

A moral frame raises cooperation, but incentives still matter

MORAL SIM places nine language models in repeated business dilemmas where ethical actions conflict with personal payoff.
30October 20, 2026

A model's economic preferences changed with the game

Ross, Kim, and Lo fit utility functions to model choices in ultimatum, gambling, and waiting tasks, after checking basic task competence.
31October 19, 2026

Testing economic choices means watching agents learn

EconEvals measures both performance in unfamiliar economic environments and the consistency of choices when objectives conflict.
32October 18, 2026

LLM buyers often leave the bargaining table poorer

Xia and colleagues test language models as buyers and sellers using historical Amazon product prices, and find a sharp gap between the two roles.
33October 17, 2026

A multiple-choice quiz can probe whether a model recognizes training text

The Data Contamination Quiz uses word-level variants of dataset examples to test black-box models for exact-text familiarity.
34October 16, 2026

Political answers tracked some models' training corpora more than survey opinions

Across 32 U.S. Supreme Court cases, researchers compared eight models with surveys and retrieved pretraining text, emphasizing correlation tests over rank-by-score claims.
35October 15, 2026

Models still lose instructions when a task unfolds in steps

SIFo checks four kinds of sequential tasks by whether the model satisfies a final, verifiable instruction after following earlier ones.
36October 14, 2026

Accuracy alone misses two kinds of prompt instability

Errica and colleagues separate prediction changes across paraphrases from the consistency of predictions within each labeled class.
37October 13, 2026

A rephrased question can become unanswerable to a model

PromptSET pairs TriviaQA and HotpotQA questions with meaning-preserving rewrites, then tests whether answerability can be predicted.
38October 12, 2026

Politeness changed accuracy, but the best tone varied

A cross-lingual comparison found that rude prompts often hurt, while extra deference did not guarantee better answers across English, Chinese, and Japanese.
39October 11, 2026

Some benchmark prompt instability belongs to the scorer

Across six benchmarks, Hua and colleagues found that rigid answer matching and likelihood heuristics counted semantically valid responses as errors.
40October 10, 2026

Conversation history can bury the task

A six-model study found that prior dialogue could lower GPQA accuracy, while repeating the task description near the query partly recovered it.
41October 9, 2026

Shuffling inputs usually hurt, but not on every task

Three systems were tested on five order-sensitive tasks; the shuffle penalty appeared often, while WebGPT comparisons showed an exception.
42October 8, 2026

A single benchmark prompt hides the score distribution

PromptEval estimates performance across many templates, letting benchmark designers report quantiles rather than one template-dependent score.
43October 7, 2026

A formatting change can move a model’s score by 76 points

Sclar and colleagues test how separators, spaces, casing, and answer labels alter scores even when prompt meaning stays fixed.
44October 6, 2026

Twelve participants supplied the prompts for a clinical LLM study

The final instruction-writing sample came from 12 participants; seven models were evaluated on their clinical task prompts across ten classification and six extraction tasks.
45October 5, 2026

Constitutional AI trains a harmless assistant with AI feedback

A two-stage training experiment uses a written set of principles for self-critique and preference feedback, aiming to reduce harmful answers without making the assistant evasive.
46October 4, 2026

How a prompt can steer code toward failure

Jones and Steinhardt use framing, anchors and example order to elicit repeatable errors in code generation, including file deletion under a targeted test.
47October 3, 2026

Chatbots emphasized fairness across economic games

Five chatbot families produced concentrated choices and differed from human patterns in payoff preferences and cross-game consistency.
48October 2, 2026

Chatbot Arena ranks models through anonymous head-to-head votes

The live evaluation platform collected more than 240,000 votes and compared crowd preferences with expert ratings.
49October 1, 2026

HELM scores models across scenarios, not a single leaderboard

A 30-model evaluation applies multiple metrics across 42 language-model scenarios and makes gaps in benchmark coverage explicit.
50September 30, 2026

Benchmark contamination can invalidate a clean-looking score

A position paper distinguishes forms of benchmark exposure and argues for explicit contamination checks and warnings.
51September 29, 2026

AlphaCode searched thousands of programs for contest solutions

The system sampled, filtered, and clustered code; in ten Codeforces contests it ranked within the top 54.3% on average.
52September 28, 2026

AGIEval compares foundation models with public human exams

Across 20 standardized-exam tasks, GPT-4 beat average human scores in several settings but lagged top human performance.
GOOD RESEARCH STARTS WITH BETTER QUESTIONS.See research in practice ↗
About AIScholar

Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

Explore AIScholar ↗