Hiring is a sequence of decisions, not one score

A candidate can be filtered out before a recruiter reads a summary. Preethi Seshadri, Hongyu Chen, Sameer Singh, and Seraphina Goldfarb-Tarrant examine both stages in Small Changes, Large Consequences: Analyzing the Allocational Fairness of LLMs in Hiring Contexts. They test resume summarization with language models and resume retrieval with dense embedding models, making the distinction between generation and ranking central to their audit.

The authors constructed synthetic resumes and paired them with job postings, then perturbed names as proxies for gender and race. They also tested non-demographic changes such as spacing and typos. Summary comparisons used automated measures of tone and readability, checked against an HR-staff preference task; retrieval tests tracked whether altered resumes stayed among the highest-ranked candidates.

Demographic and formatting edits both mattered

Generated summaries showed more frequent significant changes after race-name perturbations than gender-name perturbations in the tested setup. In retrieval, demographic name changes altered which resumes made the shortlist. But demographic changes were not the only source of instability: spacing and other non-demographic edits also moved candidates in the rankings. The study therefore identifies both allocation disparities and a broader brittleness problem.

The retrieval result comes from embedding models, not from the generative LLMs used for summaries. Keeping those pipeline components separate matters when deciding where a system needs an intervention.

Synthetic resumes have a boundary

The generated resume sample is tech-focused, and names are proxies rather than direct measures of identity. The authors also use automated summary metrics and a limited set of job postings; performance in this constructed audit does not establish actual hiring outcomes. Their clearest finding is narrower: under these controlled changes, ostensibly equivalent candidate materials did not always receive equivalent treatment.

Bibliography

  • Preethi Seshadri, Hongyu Chen, Sameer Singh, and Seraphina Goldfarb-Tarrant (2025). Small Changes, Large Consequences: Analyzing the Allocational Fairness of LLMs in Hiring Contexts. arXiv:2501.04316v2 [cs.CL]. Full paper (PDF).