| Title: |
TempTest: Local Normalization Distortion and the Detection of Machine-generated Text |
| Authors: |
Kempton, Thomas; Burrell, Stuart; Cheverall, Connor |
| Source: |
Kempton, T, Burrell, S & Cheverall, C 2025, TempTest: Local Normalization Distortion and the Detection of Machine-generated Text. in Proceedings of the 28th International Conference on Artificial Intelligence and Statistics (AISTATS 2025). vol. 258, Proceedings of Machine Learning Research, pp. 1972-1980, 28th International Conference on Artificial Intelligence and Statistics , Thailand, 3/05/25. < https://proceedings.mlr.press/v258/kempton25a.html > |
| Publisher Information: |
Proceedings of Machine Learning Research |
| Publication Year: |
2025 |
| Collection: |
The University of Manchester: Research Explorer - Publications |
| Description: |
Existing methods for the zero-shot detection of machine-generated text are dominated by three statistical quantities: log-likelihood, log-rank, and entropy. As language models mimic the distribution of human text ever closer, this will limit our ability to build effective detection algorithms. To combat this, we introduce a method for detecting machine-generated text that is entirely agnostic of the generating language model. This is achieved by targeting a defect in the way that decoding strategies, such as temperature or top-k sampling, normalize conditional probability measures. This method can be rigorously theoretically justified, is easily explainable, and is conceptually distinct from existing methods for detecting machine-generated text. We evaluate our detector in the white and black box settings across various language models, datasets, and passage lengths. We also study the effect of paraphrasing attacks on our detector and the extent to which it is biased against non-native speakers. In each of these settings, the performance of our test is at least comparable to that of other state-of-the-art text detectors, and in some cases, we strongly outperform these baselines. |
| Document Type: |
article in journal/newspaper |
| File Description: |
application/pdf |
| Language: |
English |
| Availability: |
https://research.manchester.ac.uk/en/publications/9e2bde4f-e399-4735-a603-9b18d43a7103; https://pure.manchester.ac.uk/ws/files/361723307/TempTest_Author_Accepted_Manuscript.pdf; https://proceedings.mlr.press/v258/kempton25a.html |
| Rights: |
info:eu-repo/semantics/openAccess |
| Accession Number: |
edsbas.78768748 |
| Database: |
BASE |