Represent the choice before recommending an action
In an uncertain decision, a recommendation depends on which outcomes are possible and how the decision-maker values them. DeLLMa: Decision Making Under Uncertainty with Large Language Models gives those elements an explicit structure. Ollie Liu and colleagues combine decision- and utility-theoretic calculations with a multi-step language-model procedure, so a reader can inspect how the choice was represented.
An accuracy gain on selected tasks
The researchers compare DeLLMa with direct prompting and other approaches in several realistic, structured decision environments. They test leading models and use human evaluations to examine components of the procedure. Across these tasks, they report improved accuracy, reaching up to a 40% increase over competing methods; more inference-time computation also improved scores.
What the representation leaves outside
DeLLMa currently works with small, discrete sets of states and actions, a constraint the authors connect to context-window limits. Sequential decisions or a larger option space therefore require a different or expanded representation. The paper recommends field testing and human judgment before deployment, noting that a suboptimal recommendation could have financial consequences. A useful next experiment is to enlarge the represented choice while keeping its assumptions visible to the person deciding.
Bibliography
- Liu, O., Fu, D., Yogatama, D., & Neiswanger, W. (2024). DeLLMa: Decision Making Under Uncertainty with Large Language Models. arXiv:2402.02392v3 [cs.AI]. Full paper (PDF).