| Title: |
Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology |
| Authors: |
Gould, Elliot; Fraser, Hannah S.; Parker, Timothy H.; Nakagawa, Shinichi; Griffith, Simon C.; Vesk, Peter A.; Fidler, Fiona; Hamilton, Daniel G.; Abbey-Lee, Robin N.; Abbott, Jessica K.; Aguirre, Luis A.; Alcaraz, Carles; Aloni, Irith; Altschul, Drew; Arekar, Kunal; Atkins, Jeff W.; Atkinson, Joe; Baker, Christopher M.; Barrett, Meghan; Bell, Kristian; Bello, Suleiman Kehinde; Beltrán, Iván; Berauer, Bernd J.; Bertram, Michael Grant; Billman, Peter D.; Blake, Charlie K.; Blake, Shannon; Bliard, Louis; Bonisoli-Alquati, Andrea; Bonnet, Timothée; Bordes, Camille Nina Marion; Bose, Aneesh P. H.; Botterill-James, Thomas; Boyd, Melissa Anna; Boyle, Sarah A.; Bradfer-Lawrence, Tom; Bradham, Jennifer; Brand, Jack A.; Brengdahl, Martin I.; Bulla, Martin; Bussière, Luc; Camerlenghi, Ettore; Campbell, Sara E.; Campos, Leonardo L. F.; Caravaggi, Anthony; Cardoso, Pedro; Carroll, Charles J.W.; Catanach, Therese A.; Chen, Xuan; Chik, Heung Ying Janet; Choy, Emily Sarah; Christie, Alec Philip; Chuang, Angela; Chunco, Amanda J.; Clark, Bethany L.; Contina, Andrea; Covernton, Garth A.; Cox, Murray P.; Cressman, Kimberly A.; Crotti, Marco; Davidson Crouch, Connor; D'Amelio, Pietro B.; de Sousa, Alexandra Allison; Döbert, Timm Fabian; Dobler, Ralph; Dobson, Adam J.; Doherty, Tim S.; Drobniak, Szymon Marian; Duffy, Alexandra Grace; Duncan, Alison B.; Dunn, Robert P.; Dunning, Jamie; Dutta, Trishna; Eberhart-Hertel, Luke; Elmore, Jared Alan; Elsherif, Mahmoud Medhat; English, Holly M.; Ensminger, David C.; Ernst, Ulrich Rainer; Ferguson, Stephen M.; Fernandez-Juricic, Esteban; Ferreira-Arruda, Thalita; Fieberg, John; Finch, Elizabeth A.; Fiorenza, Evan A.; Fisher, David N.; Fontaine, Amélie; Forstmeier, Wolfgang; Fourcade, Yoan; Frank, Graham S.; Freund, Cathryn A.; Fuentes-Lillo, Eduardo; Gandy, Sara L.; Gannon, Dustin G.; García-Cervigón, Ana I.; Garretson, Alexis C.; Ge, Xuezhen; Geary, William L.; Géron, Charly; Gilles, Marc; Girndt, Antje; Gliksman, Daniel; Goldspiel, Harrison B.; Gomes, Dylan G. E.; Good, Megan Kate; Goslee, Sarah C.; Gosnell, J. Stephen; Grames, Eliza M.; Gratton, Paolo; Grebe, Nicholas M.; Greenler, Skye M.; Griffioen, Maaike; Griffith, Daniel M.; Griffith, Frances J.; Grossman, Jake J.; Güncan, Ali; Haesen, Stef; Hagan, James G.; Hager, Heather A.; Harris, Jonathan Philo; Harrison, Natasha Dean; Hasnain, Sarah Syedia; Havird, Justin Chase; Heaton, Andrew J.; Herrera-Chaustre, María Laura; Howard, Tanner J.; Hsu, Bin-Yan; Iannarilli, Fabiola; Iranzo, Esperanza C.; Iverson, Erik N. K.; Jimoh, Saheed Olaide; Johnson, Douglas H.; Johnsson, Martin; Jorna, Jesse; Jucker, Tommaso; Jung, Martin; Kačergytė, Ineta; Kaltz, Oliver; Ke, Alison; Kelly, Clint D.; Keogan, Katharine; Keppeler, Friedrich Wolfgang; Killion, Alexander K.; Kim, Dongmin; Kochan, David P.; Korsten, Peter; Kothari, Shan; Kuppler, Jonas; Kusch, Jillian M.; Lagisz, Malgorzata; Lalla, Kristen Marianne; Larkin, Daniel J.; Larson, Courtney L.; Lauck, Katherine S.; Lauterbur, M. Elise; Law, Alan; Léandri-Breton, Don-Jean; Lembrechts, Jonas J.; L'Herpiniere, Kiara; Lievens, Eva J.P.; Oliveira de Lima,, Daniela; Lindsay, Shane; Luquet, Martin; MacLeod, Ross; Macphie, Kirsty H.; Magellan, Kit; Mair, Magdalena M.; Malm, Lisa E.; Mammola, Stefano; Mandeville, Caitlin P.; Manhart, Michael; Manrique-Garzon, Laura Milena; Mäntylä, Elina; Marchand, Philippe; Marshall, Benjamin Michael; Martin, Charles A.; Martin, Dominic Andreas; Martin, Jake Mitchell; Martinig, April Robin; McCallum, Erin S.; McCauley, Mark; McNew, Sabrina M.; Meiners, Scott J.; Merkling, Thomas; Michelangeli, Marcus; Moiron, Maria; Moreira, Bruno; Mortensen, Jennifer; Mos, Benjamin; Muraina, Taofeek Olatunbosun; Murphy, Penelope Wrenn; Nelli, Luca; Niemelä, Petri; Nightingale, Josh; Nilsonne, Gustav; Nolazco, Sergio; Nooten, Sabine S.; Novotny, Jessie Lanterman; Olin, Agnes Birgitta; Organ, Chris L.; Ostevik, Kate L.; Palacio, Facundo Xavier; Paquet, Matthieu; Parker, Darren James; Pascall, David J.; Pasquarella, Valerie J.; Paterson, John Harold; Payo-Payo, Ana; Pedersen, Karen Marie; Perez, Grégoire; Perry, Kayla I.; Pottier, Patrice; Proulx, Michael J.; Proulx, Raphaël; Pruett, Jessica L.; Ramananjato, Veronarindra; Randimbiarison, Finaritra Tolotra; Razafindratsima, Onja H.; Rennison, Diana J.; Riva, Federico; Riyahi, Sepand; Roast, Michael James; Rocha, Felipe Pereira; Roche, Dominique G.; Román-Palacios, Cristian; Rosenberg, Michael S.; Ross, Jessica; Rowland, Freya E.; Rugemalila, Deusdedith; Russell, Avery L.; Ruuskanen, Suvi; Saccone, Patrick; Sadeh, Asaf; Salazar, Stephen M.; Sales, Kris; Salmón, Pablo; Sánchez-Tójar, Alfredo; Santos, Leticia Pereira; Santostefano, Francesca; Schilling, Hayden T.; Schmidt, Marcus; Schmoll, Tim; Schneider, Adam C.; Schrock, Allie E.; Schroeder, Julia; Schtickzelle, Nicolas; Schultz, Nick L.; Scott, Drew A.; Scroggie, Michael Peter; Shapiro, Julie Teresa; Sharma, Nitika; Shearer, Caroline L.; Simón, Diego; Sitvarin, Michael I.; Skupien, Fabrício Luiz; Slinn, Heather Lea; Smith, Grania Polly; Smith, Jeremy A.; Sollmann, Rahel; Stack Whitney, Kaitlin; Still, Shannon Michael; Stuber, Erica F.; Sutton, Guy F.; Swallow, Ben; Taff, Conor Claverie; Takola, Elina; Tanentzap, Andrew J.; Tarjuelo, Rocío; Telford, Richard J.; Thawley, Christopher J.; Thierry, Hugo; Thomson, Jacqueline; Tidau, Svenja; Tompkins, Emily M.; Tortorelli, Claire Marie; Trlica, Andrew; Turnell, Biz R.; Urban, Lara; Van de Vondel, Stijn; van der Wal, Jessica Eva Megan; Van Eeckhoven, Jens; van Oordt, Francis; Vanderwel, K. Michelle; Vanderwel, Mark C.; Vélez, Juliana; Vergara-Florez, Diana Carolina; Verrelli, Brian C.; Vieira, Marcus Vinícius; Villamil, Nora; Vitali, Valerio; Vollering, Julien; Walker, Jeffrey; Walker, Xanthe J.; Walter, Jonathan A.; Waryszak, Pawel; Weaver, Ryan J.; Wedegärtner, Ronja E. M.; Weller, Daniel L.; Whelan, Shannon; White, Rachel Louise; Wolfson, David William; Wood, Andrew; Yanco, Scott W.; Yen, Jian D. L.; Youngflesh, Casey; Zilio, Giacomo; Zimmer, Cédric; Zimmerman, Gregory Mark; Zitomer, Rachel A. |
| Publication Year: |
2023 |
| Collection: |
University of Glasgow: Enlighten - Publications |
| Description: |
Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different (mostly social science) fields, and has found substantial variability among results, despite analysts having the same data and research question. We implemented an analogous study in ecology and evolutionary biology, fields in which there have been no empirical exploration of the variation in effect sizes or model predictions generated by the analytical decisions of different researchers. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment), and the project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects ... |
| Document Type: |
article in journal/newspaper |
| File Description: |
text |
| Language: |
English |
| Relation: |
https://eprints.gla.ac.uk/307802/4/307802.pdf; Gould, E. et al. (2023) Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology. EcoEvoRxiv , (doi:10.32942/X2GG62 ) |
| DOI: |
10.32942/X2GG62 |
| Availability: |
https://eprints.gla.ac.uk/307802/; https://eprints.gla.ac.uk/307802/4/307802.pdf; https://doi.org/10.32942/X2GG62 |
| Rights: |
cc_by_4 |
| Accession Number: |
edsbas.3462E159 |
| Database: |
BASE |