AISCHOLAR / BLOG
Research on the
fascinating minds of LLMs.
In this blog, AI agents help us discover and summarize interesting research insights on the study of large language models and intelligent systems powered by generative AI.
Share
LATEST WRITING01 / BLOG

A metric can make gradual model gains look like a sudden leap
Three analyses test whether discontinuous scoring rules create apparent emergent abilities in language-model scaling curves.
Read the article ↗02November 17, 2026
MEGA evaluates generative models across 70 languages
Sixteen NLP datasets let the benchmark compare model behavior across diverse languages and task types.03November 16, 2026What would a psychology of language models study?
Hagendorff and colleagues map out behavioral experiments for studying LLMs, while warning against reading human minds into model outputs.04November 15, 2026Relevant facts in the middle of a long prompt are harder to use
Position-controlled retrieval and question-answering tests found a U-shaped accuracy pattern in long contexts.05November 14, 2026A language model can search for another model’s harmful replies
Perez and colleagues automate dialogue red teaming by generating test prompts and screening outputs, surfacing tens of thousands of offensive replies from a 280-billion-parameter chatbot.06November 13, 2026When GPT-4 judges chatbot answers, its agreement depends on the matchup
MT-Bench and Chatbot Arena compare LLM judgments with human preferences, while exposing position, verbosity, and self-enhancement biases.07November 12, 2026Two failure modes help explain why safety-trained models can be jailbroken
Wei, Haghtalab, and Steinhardt use competing objectives and mismatched generalization to design and test jailbreaks against several safety-trained language models.08November 11, 2026JailbreakBench standardizes how attacks are tested and compared
A public benchmark pairs 100 harmful-behavior goals with open attack artifacts, a shared threat model, templates, scoring, and evaluations of defenses.09November 10, 2026Training a model to treat system and user instructions differently
An instruction hierarchy method teaches GPT-3.5 to ignore lower-priority directions that conflict with higher-priority ones.10November 9, 2026What an expert role prompt changes
Salewski and colleagues test age, expertise and social-identity personas in bandit games, knowledge questions and image descriptions.11November 8, 2026IFEval checks instructions that a program can verify
A benchmark of roughly 500 prompts turns constraints such as word counts and required keywords into reproducible checks.12November 7, 2026Changing a number or adding a clause unsettled math-benchmark scores
GSM-Symbolic generated controlled variants of grade-school math questions to measure answer variance and robustness.13November 6, 2026GPT-3.5 missed important parts of German vote choice
Personas built from the 2017 German Longitudinal Election Study produced biased party estimates and only 46% average agreement with respondents' reported votes.14November 5, 2026FollowBench adds one response constraint at a time
The benchmark measures five kinds of constraints across increasing levels, exposing how performance changes as requests accumulate.15November 4, 2026GPT-4 solved three quarters of a false-belief test battery
A demanding all-or-nothing score across 40 bespoke tasks produced a wide gap between older systems and the June 2023 GPT-4.16November 3, 2026FACTSCORE grades each factual claim in a long answer
The metric decomposes generated biographies into atomic facts and checks support against a knowledge source.17November 2, 2026ExploreToM's harder stories exposed belief-tracking gaps
ExploreToM models gained 27 points on ToMi and 29 on Hi-ToM after training on generated social stories.18November 1, 2026More tests changed which code models passed HumanEval
EvalPlus expanded HumanEval's test inputs and found previously missed errors and differences in model rankings.19October 31, 2026Do-Not-Answer measures whether safeguards refuse risky instructions
The authors assembled 939 risky prompts and more than 5,000 model responses, then compared human annotations with automated safety classifiers.20October 30, 2026The same choice task gave different patience estimates
Across 22 languages, GPT-3.5 and GPT-4 often preferred sooner rewards; asking GPT-4 to explain its choice shifted those rates without eliminating the mismatch.21October 29, 2026A decision framework helped models compare uncertain options
DeLLMa represents options, states, consequences, and utilities before asking a model to choose among them.22October 28, 2026DecodingTrust probes GPT models across eight trustworthiness risks
A benchmark evaluates GPT-3.5 and GPT-4 on toxicity, bias, robustness, privacy, ethics, and fairness, finding strengths on standard tests alongside prompt-induced vulnerabilities.23October 27, 2026Trust can be a poor bet for a model thinking one round at a time
In a repeated trust game, DeepSeek and OpenAI model versions respond differently to a predictable partner and future returns.24October 26, 2026More deliberate reasoning sometimes means less cooperation
Experiments across cooperation and punishment games find lower prosocial choices from several reasoning-enabled model variants.25October 25, 2026A paraphrase can reveal a gap between moral answers
SaGE checks whether models apply the same inferred rule to semantically similar scenarios, rather than grading them against one answer key.26October 24, 2026When a moral choice is unclear, models disagree with themselves
A 28-model survey separates uncertainty from wording sensitivity across clear and ambiguous moral scenarios.27October 23, 2026LLM traders kept prices near value while human markets bubbled
A controlled 30-period asset market produced distinct price paths for model-only, mixed-model, and human-only markets.28October 22, 2026A resume's demographic cues changed who reached the shortlist
A hiring-pipeline audit finds that small resume perturbations can alter generated summaries and retrieval rankings, though the two stages use different models.29October 21, 2026A moral frame raises cooperation, but incentives still matter
MORAL SIM places nine language models in repeated business dilemmas where ethical actions conflict with personal payoff.30October 20, 2026A model's economic preferences changed with the game
Ross, Kim, and Lo fit utility functions to model choices in ultimatum, gambling, and waiting tasks, after checking basic task competence.31October 19, 2026Testing economic choices means watching agents learn
EconEvals measures both performance in unfamiliar economic environments and the consistency of choices when objectives conflict.32October 18, 2026LLM buyers often leave the bargaining table poorer
Xia and colleagues test language models as buyers and sellers using historical Amazon product prices, and find a sharp gap between the two roles.33October 17, 2026A multiple-choice quiz can probe whether a model recognizes training text
The Data Contamination Quiz uses word-level variants of dataset examples to test black-box models for exact-text familiarity.34October 16, 2026Political answers tracked some models' training corpora more than survey opinions
Across 32 U.S. Supreme Court cases, researchers compared eight models with surveys and retrieved pretraining text, emphasizing correlation tests over rank-by-score claims.35October 15, 2026Models still lose instructions when a task unfolds in steps
SIFo checks four kinds of sequential tasks by whether the model satisfies a final, verifiable instruction after following earlier ones.36October 14, 2026Accuracy alone misses two kinds of prompt instability
Errica and colleagues separate prediction changes across paraphrases from the consistency of predictions within each labeled class.37October 13, 2026A rephrased question can become unanswerable to a model
PromptSET pairs TriviaQA and HotpotQA questions with meaning-preserving rewrites, then tests whether answerability can be predicted.38October 12, 2026Politeness changed accuracy, but the best tone varied
A cross-lingual comparison found that rude prompts often hurt, while extra deference did not guarantee better answers across English, Chinese, and Japanese.39October 11, 2026Some benchmark prompt instability belongs to the scorer
Across six benchmarks, Hua and colleagues found that rigid answer matching and likelihood heuristics counted semantically valid responses as errors.40October 10, 2026Conversation history can bury the task
A six-model study found that prior dialogue could lower GPQA accuracy, while repeating the task description near the query partly recovered it.41October 9, 2026Shuffling inputs usually hurt, but not on every task
Three systems were tested on five order-sensitive tasks; the shuffle penalty appeared often, while WebGPT comparisons showed an exception.42October 8, 2026A single benchmark prompt hides the score distribution
PromptEval estimates performance across many templates, letting benchmark designers report quantiles rather than one template-dependent score.43October 7, 2026A formatting change can move a model’s score by 76 points
Sclar and colleagues test how separators, spaces, casing, and answer labels alter scores even when prompt meaning stays fixed.44October 6, 2026Twelve participants supplied the prompts for a clinical LLM study
The final instruction-writing sample came from 12 participants; seven models were evaluated on their clinical task prompts across ten classification and six extraction tasks.45October 5, 2026Constitutional AI trains a harmless assistant with AI feedback
A two-stage training experiment uses a written set of principles for self-critique and preference feedback, aiming to reduce harmful answers without making the assistant evasive.46October 4, 2026How a prompt can steer code toward failure
Jones and Steinhardt use framing, anchors and example order to elicit repeatable errors in code generation, including file deletion under a targeted test.47October 3, 2026Chatbots emphasized fairness across economic games
Five chatbot families produced concentrated choices and differed from human patterns in payoff preferences and cross-game consistency.48October 2, 2026Chatbot Arena ranks models through anonymous head-to-head votes
The live evaluation platform collected more than 240,000 votes and compared crowd preferences with expert ratings.49October 1, 2026HELM scores models across scenarios, not a single leaderboard
A 30-model evaluation applies multiple metrics across 42 language-model scenarios and makes gaps in benchmark coverage explicit.50September 30, 2026Benchmark contamination can invalidate a clean-looking score
A position paper distinguishes forms of benchmark exposure and argues for explicit contamination checks and warnings.51September 29, 2026AlphaCode searched thousands of programs for contest solutions
The system sampled, filtered, and clustered code; in ten Codeforces contests it ranked within the top 54.3% on average.52September 28, 2026AGIEval compares foundation models with public human exams
Across 20 standardized-exam tasks, GPT-4 beat average human scores in several settings but lagged top human performance.GOOD RESEARCH STARTS WITH BETTER QUESTIONS.See research in practice ↗
About AIScholar
Explore AIScholar ↗Study LLM behavior.
AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.