A black-box contamination test asks a model to pick the original wording from a set of near-matches.
Four versions, one original
Golchin and Surdeanu's Data Contamination Quiz (DCQ) creates three word-level perturbations of each sampled dataset instance, then presents them with the original and a none-of-the-above option. The model identifies which text is original. The paper accounts for positional bias and evaluates the method across datasets and language models under controlled and uncontrolled contamination, without access to training data or model parameters.
The authors report that DCQ detects contamination in their experiments and reveals more memorization than existing approaches. They also found that the quiz could bypass some safety filters, especially those intended to prevent copyrighted-text generation. These findings concern the tested filters and quiz settings.
DCQ measures recognition among its constructed alternatives and cannot say when a model encountered the wording. Its interpretation depends on the dataset partition, perturbation design, and positional control. A high score supports a narrow inference: the model recognizes that partition's wording under this quiz.
Bibliography
- Shahriar Golchin; Mihai Surdeanu. (2023). Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models. arXiv:2311.06233v7. https://arxiv.org/abs/2311.06233v7