| Title: |
Data leakage in deep learning studies of translational EEG |
| Authors: |
Geoffrey Brookshire; Jake Kasper; Nicholas M. Blauch; Yunan Charles Wu; Ryan Glatt; David A. Merrill; Spencer Gerrol; Keith J. Yoder; Colin Quirk; Ché Lucero |
| Source: |
Frontiers in Neuroscience, Vol 18 (2024) |
| Publisher Information: |
Frontiers Media S.A. |
| Publication Year: |
2024 |
| Collection: |
Directory of Open Access Journals: DOAJ Articles |
| Subject Terms: |
electroencephalography; deep neural networks; data leakage; cross-validation; Alzheimer's disease; epilepsy; Neurosciences. Biological psychiatry. Neuropsychiatry; RC321-571 |
| Description: |
A growing number of studies apply deep neural networks (DNNs) to recordings of human electroencephalography (EEG) to identify a range of disorders. In many studies, EEG recordings are split into segments, and each segment is randomly assigned to the training or test set. As a consequence, data from individual subjects appears in both the training and the test set. Could high test-set accuracy reflect data leakage from subject-specific patterns in the data, rather than patterns that identify a disease? We address this question by testing the performance of DNN classifiers using segment-based holdout (in which segments from one subject can appear in both the training and test set), and comparing this to their performance using subject-based holdout (where all segments from one subject appear exclusively in either the training set or the test set). In two datasets (one classifying Alzheimer's disease, and the other classifying epileptic seizures), we find that performance on previously-unseen subjects is strongly overestimated when models are trained using segment-based holdout. Finally, we survey the literature and find that the majority of translational DNN-EEG studies use segment-based holdout. Most published DNN-EEG studies may dramatically overestimate their classification performance on new subjects. |
| Document Type: |
article in journal/newspaper |
| Language: |
English |
| Relation: |
https://www.frontiersin.org/articles/10.3389/fnins.2024.1373515/full; https://doaj.org/toc/1662-453X; https://doaj.org/article/8746e7d329c94692b2cf5f78afbf8ff4 |
| DOI: |
10.3389/fnins.2024.1373515 |
| Availability: |
https://doi.org/10.3389/fnins.2024.1373515; https://doaj.org/article/8746e7d329c94692b2cf5f78afbf8ff4 |
| Accession Number: |
edsbas.2F43BDB0 |
| Database: |
BASE |