Sixteen datasets across a broad language sample

A result on English benchmarks alone says little about a model's measured performance in other languages. Ahuja and colleagues introduce MEGA in MEGA: Multilingual Evaluation of Generative AI, evaluating generative models on 16 NLP datasets across 70 typologically diverse languages. The tasks cover language understanding, reasoning, and generation, and the comparison includes systems such as ChatGPT and GPT-4.

Results vary across languages and tasks

The paper reports uneven outcomes; some low-resource languages in Latin script show promising results. The authors also examine possible test-data contamination, which could inflate measured capabilities. MEGA reports outcomes across the language-task pairs it includes, while explanations that connect performance to pretraining-language representation are not causally isolated by the study.

Dataset availability sets the boundary

The benchmark relies on available datasets and adaptation choices, so it does not establish comparable quality for every language or real-world use. Its contamination analysis is particularly relevant when reading a high score for an individual test set: possible overlap with training data affects what that score can establish.

Bibliography

  • Ahuja, K., Diddee, H., Hada, R., Ochieng, M., Ramesh, K., Jain, P., Nambi, A., Ganu, T., Segal, S., Axmed, M., Bali, K., & Sitaram, S. (2023). MEGA: Multilingual Evaluation of Generative AI. arXiv:2303.12528v4 [cs.CL]. https://arxiv.org/pdf/2303.12528