| Title: |
Same data, different analysts : variation in effect sizes due to analytical decisions in ecology and evolutionary biology |
| Authors: |
Gould, Elliot; Fraser, Hannah S.; Parker, Timothy H.; Nakagawa, Shinichi; Griffith, Simon C.; Vesk, Peter A.; Fidler, Fiona; Hamilton, Daniel G.; Abbey-Lee, Robin N.; Abbott, Jessica K.; Aguirre, Luis A.; Alcaraz, Carles; Aloni, Irith; Altschul, Drew; Arekar, Kunal; Atkins, Jeff W.; Atkinson, Joe; Baker, Christopher M.; Barrett, Meghan; Bell, Kristian; Bello, Suleiman Kehinde; Beltrán, Iván; Berauer, Bernd J.; Bertram, Michael Grant; Billman, Peter D.; Blake, Charlie K.; Blake, Shannon; Bliard, Louis; Bonisoli-Alquati, Andrea; Bonnet, Timothée; Bordes, Camille Nina Marion; Bose, Aneesh P.H.; Botterill-James, Thomas; Boyd, Melissa Anna; Boyle, Sarah A.; Bradfer-Lawrence, Tom; Bradham, Jennifer; Brand, Jack A.; Brengdahl, Martin I.; Bulla, Martin; Bussière, Luc; Camerlenghi, Ettore; Campbell, Sara E.; Campos, Leonardo L.F.; Caravaggi, Anthony; Cardoso, Pedro; Carroll, Charles J.W.; Catanach, Therese A.; Chen, Xuan; Chik, Heung Ying Janet; Choy, Emily Sarah; Christie, Alec Philip; Chuang, Angela; Chunco, Amanda J.; Clark, Bethany L.; Contina, Andrea; Covernton, Garth A.; Cox, Murray P.; Cressman, Kimberly A.; Crotti, Marco; Crouch, Connor Davidson; D’Amelio, Pietro B.; de Sousa, Alexandra Allison; Döbert, Timm Fabian; Dobler, Ralph; Dobson, Adam J.; Doherty, Tim S.; Drobniak, Szymon Marian; Duffy, Alexandra Grace; Duncan, Alison B.; Dunn, Robert P.; Dunning, Jamie; Dutta, Trishna; Eberhart-Hertel, Luke; Elmore, Jared Alan; Elsherif, Mahmoud Medhat; English, Holly M.; Ensminger, David C.; Ernst, Ulrich Rainer; Ferguson, Stephen M.; Fernandez-Juricic, Esteban; Ferreira-Arruda, Thalita; Fieberg, John; Finch, Elizabeth A.; Fiorenza, Evan A.; Fisher, David N.; Fontaine, Amélie; Forstmeier, Wolfgang; Fourcade, Yoan; Frank, Graham S.; Freund, Cathryn A.; Fuentes-Lillo, Eduardo; Gandy, Sara L.; Gannon, Dustin G.; García-Cervigón, Ana I.; Garretson, Alexis C.; Ge, Xuezhen; Geary, William L.; Géron, Charly; Gilles, Marc; Girndt, Antje; Gliksman, Daniel; Goldspiel, Harrison B.; Gomes, Dylan G.E.; Good, Megan Kate; Goslee, Sarah C.; Gosnell, J. Stephen; Grames, Eliza M.; Gratton, Paolo; Grebe, Nicholas M.; Greenler, Skye M.; Griffioen, Maaike; Griffith, Daniel M.; Griffith, Frances J.; Grossman, Jake J.; Güncan, Ali; Haesen, Stef; Hagan, James G.; Hager, Heather A.; Harris, Jonathan Philo; Harrison, Natasha Dean; Hasnain, Sarah Syedia; Havird, Justin Chase; Heaton, Andrew J.; Herrera-Chaustre, María Laura; Howard, Tanner J.; Hsu, Bin Yan; Iannarilli, Fabiola; Iranzo, Esperanza C.; Iverson, Erik N.K.; Jimoh, Saheed Olaide; Johnson, Douglas H.; Johnsson, Martin; Jorna, Jesse; Jucker, Tommaso; Jung, Martin; Kačergytė, Ineta; Kaltz, Oliver; Ke, Alison; Kelly, Clint D.; Keogan, Katharine; Keppeler, Friedrich Wolfgang; Killion, Alexander K.; Kim, Dongmin; Kochan, David P.; Korsten, Peter; Kothari, Shan; Kuppler, Jonas; Kusch, Jillian M.; Lagisz, Malgorzata; Lalla, Kristen Marianne; Larkin, Daniel J.; Larson, Courtney L.; Lauck, Katherine S.; Lauterbur, M. Elise; Law, Alan; Léandri-Breton, Don Jean; Lembrechts, Jonas J.; L’Herpiniere, Kiara; Lievens, Eva J.P.; de Lima, Daniela Oliveira; Lindsay, Shane; Luquet, Martin; MacLeod, Ross; Macphie, Kirsty H.; Magellan, Kit; Mair, Magdalena M.; Malm, Lisa E.; Mammola, Stefano; Mandeville, Caitlin P.; Manhart, Michael; Manrique-Garzon, Laura Milena; Mäntylä, Elina; Marchand, Philippe; Marshall, Benjamin Michael; Martin, Charles A.; Martin, Dominic Andreas; Martin, Jake Mitchell; Martinig, April Robin; McCalum, Erin S.; McCauley, Mark; McNew, Sabrina M.; Meiners, Scott J.; Merkling, Thomas; Michelangeli, Marcus; Moiron, Maria; Moreira, Bruno; Mortensen, Jennifer; Mos, Benjamin; Muraina, Taofeek Olatunbosun; Murphy, Penelope Wrenn; Nelli, Luca; Niemelä, Petri; Nightingale, Josh; Nilsonne, Gustav; Nolazco, Sergio; Nooten, Sabine S.; Novotny, Jessie Lanterman; Olin, Agnes Birgitta; Organ, Chris L.; Ostevik, Kate L.; Palacio, Facundo Xavier; Paquet, Matthieu; Parker, Darren James; Pascall, David J.; Pasquarella, Valerie J.; Paterson, John Harold; Payo-Payo, Ana; Pedersen, Karen Marie; Perez, Grégoire; Perry, Kayla I.; Pottier, Patrice; Proulx, Michael J.; Proulx, Raphaël; Pruett, Jessica L.; Ramananjato, Veronarindra; Randimbiarison, Finaritra Tolotra; Razafindratsima, Onja H.; Rennison, Diana J.; Riva, Federico; Riyahi, Sepand; Roast, Michael James; Rocha, Felipe Pereira; Roche, Dominique G.; Román-Palacios, Cristian; Rosenberg, Michael S.; Ross, Jessica; Rowland, Freya E.; Rugemalila, Deusdedith; Russell, Avery L.; Ruuskanen, Suvi; Saccone, Patrick; Sadeh, Asaf; Salazar, Stephen M.; Sales, Kris; Salmón, Pablo; Sánchez-Tójar, Alfredo; Santos, Leticia Pereira; Santostefano, Francesca; Schilling, Hayden T.; Schmidt, Marcus; Schmoll, Tim; Schneider, Adam C.; Schrock, Allie E.; Schroeder, Julia; Schtickzelle, Nicolas; Schultz, Nick L.; Scott, Drew A.; Scroggie, Michael Peter; Shapiro, Julie Teresa; Sharma, Nitika; Shearer, Caroline L.; Simón, Diego; Sitvarin, Michael I.; Skupien, Fabrício Luiz; Slinn, Heather Lea; Smith, Grania Polly; Smith, Jeremy A.; Sollmann, Rahel; Whitney, Kaitlin Stack; Still, Shannon Michael; Stuber, Erica F.; Sutton, Guy F.; Swallow, Ben; Taff, Conor Claverie; Takola, Elina; Tanentzap, Andrew J.; Tarjuelo, Rocío; Telford, Richard J.; Thawley, Christopher J.; Thierry, Hugo; Thomson, Jacqueline; Tidau, Svenja; Tompkins, Emily M.; Tortorelli, Claire Marie; Trlica, Andrew; Turnell, Biz R.; Urban, Lara; Van de Vondel, Stijn; van der Wal, Jessica Eva Megan; Van Eeckhoven, Jens; van Oordt, Francis; Vanderwel, K. Michelle; Vanderwel, Mark C.; Vanderwolf, Karen J.; Vélez, Juliana; Vergara-Florez, Diana Carolina; Verrelli, Brian C.; Vieira, Marcus Vinícius; Villamil, Nora; Vitali, Valerio; Vollering, Julien; Walker, Jeffrey; Walker, Xanthe J.; Walter, Jonathan A.; Waryszak, Pawel; Weaver, Ryan J.; Wedegärtner, Ronja E.M.; Weller, Daniel L.; Whelan, Shannon; White, Rachel Louise; Wolfson, David William; Wood, Andrew; Yanco, Scott W.; Yen, Jian D.L.; Youngflesh, Casey; Zilio, Giacomo; Zimmer, Cédric; Zimmerman, Gregory Mark; Zitomer, Rachel A. |
| Contributors: |
University of St Andrews.Statistics; University of St Andrews.Centre for Research into Ecological & Environmental Modelling |
| Publication Year: |
2025 |
| Collection: |
University of St Andrews: Digital Research Repository |
| Subject Terms: |
Analytical heterogeneity; Metascience; Many-analyst; Replication crisis; Reproducibility; QH301 Biology; Biotechnology; Structural Biology; Ecology; Evolution; Behavior and Systematics; Physiology; General Biochemistry,Genetics and Molecular Biology; General Agricultural and Biological Sciences; Plant Science; Developmental Biology; Cell Biology; DAS; QH301 |
| Description: |
Funding: EG’s contributions were supported by an Australian Government Research Training Program Scholarship, AIMOS top-up scholarship (2022) and Melbourne Centre of Data Science Doctoral Academy Fellowship (2021). FF’s contributions were supported by ARC Future Fellowship FT150100297. ; Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small “many analyst” study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more ... |
| Document Type: |
article in journal/newspaper |
| File Description: |
application/pdf |
| Language: |
English |
| Relation: |
BMC Biology; 294203712; 85218290951; RIS: urn:FC01B0D94C823F254688757DBFE4A623; RIS: Gould2025; https://hdl.handle.net/10023/31736; https://www.scopus.com/pages/publications/85218290951; https://ecoevorxiv.org/repository/view/6000/ |
| DOI: |
10.32942/X2GG62 |
| Availability: |
https://hdl.handle.net/10023/31736; https://doi.org/10.32942/X2GG62; https://www.scopus.com/pages/publications/85218290951; https://ecoevorxiv.org/repository/view/6000/ |
| Rights: |
Copyright © The Author(s) 2025. Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modiied the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. |
| Accession Number: |
edsbas.54FC1B07 |
| Database: |
BASE |