| Title: |
Investigating the analytical robustness of the social and behavioural sciences. |
| Authors: |
Aczel, Balazs; Szaszi, Barnabas; Clelland, Harry T; Kovacs, Marton; Holzmeister, Felix; van Ravenzwaaij, Don; Schulz-Kümpel, Hannah; Hoffmann, Sabine; Nilsonne, Gustav; Kosa, Livia; Torma, Zoltan A; Abdelfatah, Yousuf; Aberson, Christopher L; Acar, Oguz A; Acem, Ensar; Adamkovic, Matus; Adamovich, Timofey; Adiasto, Krisna; Ahnström, Love; Akil, Atakan M; Al-Busaidi, Adil S; Al-Hoorie, Ali H; Albers, Casper J; Allen, Peter J; Alsalti, Taym; Altman, Micah; Alzahawi, Shilaan; Ambrosini, Ettore; Anafinova, Saule; Anand, Rahul; Angerer, Martin; Angulo-Brunet, Ariadna; Antonietti, Alberto; Arato, Jozsef; Arenas, Andreu; Aviña, Marco M; Azevedo, Flavio; Bachl, Marko; Bago, Bence; Bahník, Štěpán; Baker, Bradley J; Balayan, Elza; Baldwin, Cassandra L; Banai, Benjamin; Banas, Kasia; Bartoš, František; Baskin, Ernest; Bastiaansen, Jojanneke A; Bault, Nadège; Bauman, Christopher W; Beazer, Quintin H; Behnke, Maciej; Bendixen, Theiss; Berger, Sebastian; Bernard, Anna; Bernardic, Ursa; Bloom, Paul A; Boldt, Annika; Bosch-Rosa, Ciril; Botvinik-Nezer, Rotem; Bouyamourn, Adam; Bozkurt, Ozge; Brehm, Laurel; Breuer, Johannes; Briggs, Ryan; Brohmer, Hilmar; Buchanan, Erin; Buckenmaier, Johannes; Buckley, Jeffrey; Buczny, Jacek; Burghart, Matthias; Butt, Bilal H; Byrd, Nick; Cafarelli, Valentina; Callahan, Patrick; Capitán, Tabaré; Carriere, Kevin; Cataldo, Andrea M; Cepaluni, Gabriel; Chan, Eugene; Chandler, Jesse J; Chang, Chia-Chen; Chen, Xi; Chen, Shirley Shuo; Chen, Fadong; Chen, Hao; Chirkov, Valerii; Cialfi, Daniela; Clarke, Beth; Coelho, Sophie G; Cohen, Clara; Collins, Jason; Cook, Susan W; Corlazzoli, Gaia; Cummins, Jamie; Czymara, Christian; D'hondt, Jonathan; Rosa, Anna Dalla; Davis, Abi MB; Davis, Charles P; Day, Martin V; De Keyzer, Freya; de Leeuw, Joshua R; de Vries, Tjeerd Rudmer; Debnath, Ramit; Dechterenko, Filip; Demiral, Elif E; Desgroseilliers, Marc; Dianovics, Dominik; Diveica, Veronica; Dochow-Sondershaus, Stephan; Dohle, Simone; Dong, LiChen; Dora, Jonas; Dorrough, Angela R; Dreber, Anna; Du, Hongfei; Edlund, John E; Eerland, Anita; Efendić, Emir; Elder, Jacob; Elsherif, Mahmoud M; Ernst, Mareike; Estrada, Eduardo; Eudave, Luis; Evans, Thomas R; Farrera, Arodi; Ferrouhi, El Mehdi; Fiala, Lenka; Fialho, Fabrício M; Fiechter, Joshua L; Fišar, Miloš; Flores-Kanter, Pablo Ezequiel; Folwarczny, Michał; Fossum, Jessica L; Franco, Vithor R; Freichel, René; Freire, Danilo; Frese, Joris; Furnas, Alexander C; Gaebler, Johann D; Gajary, Lisa C; Galang, Carl Michael; Ganschow, Benjamin; Garrison, S Mason; Gasiorowska, Agata; Ponne, Bruno Gasparotto; Gauriot, Romain; Geminiani, Alice; Geraldes, Diogo; Gernsbacher, Morton Ann; Giani, Cinzia; Glerean, Enrico; Gligorić, Vukašin; Gnambs, Timo; Godefroidt, Amélie; González-Bustamante, Bastián; Goreis, Andreas; Graf-Vlachy, Lorenz; Grieder, Manuel; Grigoryev, Dmitry; Grinschgl, Sandra; Grüning, David J; Guassi Moreira, João F; Guichet, Clément; Gurgand, Lilas; Habibnia, Hooman; Hafenbrack, Andrew C; Hafenbrädl, Sebastian; Häffner, Carolin; Hagemeister, Felix; Haigh, Matthew; Hajdu, Nandor; Hajimoladarvish, Narges; Hall, Jonathan D; Hamjediers, Maik; Hardwick, Robert M; Harma, Mehmet; Harp, Nicholas R; Hartvig, Áron D; Heiberger, Raphael H; Heim, Arthur; Hernæs, Øystein; Hernaus, Dennis; Heyman, Tom; Hicks, Joshua; Hogeveen, Jeremy; Höpler, Julia; Houlihan, Sean Dae; Huber, Christoph; Hughes, Conor; Hummler, Teresa; Huth, Karoline; Ingendahl, Moritz; Ishii, Tatsunori; Isler, Ozan; Izydorczak, Kamil; Jackson, Iain R; Jahn, Andrew; Jain, Maitri; Jakubow, Alexander; Jang, Daisung; Jang, JunHyeok; Jekel, Marc; Jia, Fanli; Jiménez-Leal, William; Johnson, Rebecca; Jones, Alex; Jungkunz, Sebastian; Kačmár, Pavol; Kaiser, Caspar; Kalaycı, Yağmur; Kantorowicz, Jaroslaw; Karabulut, Anıl; Karch, Julian D; Karimi-Rouzbahani, Hamid; Karl, Johannes A; Kažemekaitytė, Austėja; Kazlou, Aliaksandr; Kekecs, Zoltan; Kim, Jin; Kirchler, Michael H; Kiss-Dobronyi, Bence; Klasmeier, Kai N; Klein, Jack W; Koba, Cemal; Kołczyńska, Marta; Kolias, Pavlos; Kolouch Grabovský, Matěj; Korbmacher, Max; Korda, Živa; Kowal, Marta; Kretzschmar, André; Krivoshchekov, Vladislav; Krypotos, Angelos-Miltiadis; Kubsch, Marcus; Kunisato, Yoshihiko; Lacko, David; Landwehr, Jan R; Lange, Martin; Lee, Hongmi; Lee, Daniel; Lee, Sangil; Lemay, Edward P; Lempert, Daniel; Leo, Andrea; Lesage, Elise; Levin, Joel M; Li, Peng; Lin, Jing; Lindsay, Luke; Lisovoj, Daria; Liu, Meng; Liu, Sihong; Liu, Tingshu; Iacono, Sergio Lo; Lodder, Paul; López-Bueno, Rubén; Lopez-Nicolas, Ruben; Loter, Katharina; Lou, Nigel Mantou; Lovakov, Andrey; Lu, Jackson G; Ludwig, Jonas; Luebber, Finn; Lukavský, Jiří; Luo, Charles Q; Lyu, Xuanyu; Maassen, Esther; Máčel, Martin; Mack, Michael L; Madan, Christopher R; Mädebach, Andreas; Maffly-Kipp, Joseph; Mallinson, Daniel J; Marchetti, Igor; Marghetis, Tyler; Marini, Matteo M; Fages, Diego Marino; Martínez, Mayte; Martinoli, Mario; Masiliunas, Aidas; Massoni, Sébastien; Mathieu, Kaleb C; Mayer, Stefan; Mayer, Duncan J; Mayer, Maren; McCormick, Ethan M; McDonough, Ian M; McGowan, Amanda L; McIntyre, Miranda M; McKee, Paul; Meier, Armando N; Meier, Pascal F; Melero, Helena; Merkle, Christoph; Merz, Raphael; Michaelides, Michalis P; Michaelsen, Patrik; Mikolajczak, Gosia; Mill, Wladislaw; Millroth, Philip; Miroshnik, Kirill G; Misiak, Michal; Mora, Youri L; Moreau, David; Moreh, Chris; Morvinski, Coby; Mushtaq, Faisal; Nagy, Tamás; Nater, Christa; Naumann, Elias; Navarrete, Gorka; Nebe, Stephan; Nedderhoff, Andre; Nennstiel, Richard; Neugebauer, Martin; Nicolaisen-Sobesky, Eliana; Nielsen, Yngwie A; Niso, Guiomar; Nowak, Benjamin; Okan, Mehmet; Ong, Kenneth; Onicas, Adrian I; Oswald, Christian; Otten, Kasper; Pandey, Shubham; Pantazi, Myrto; Papale, Paolo; Pärnamets, Philip; Pauer, Shiva; Pavlov, Yuri G; Pawel, Samuel; Peelle, Jonathan E; Peetz, Hannah K; Peez, Anton; Pesciarelli, Francesca; Peterson, Brenton D; Petruželka, Benjamin; Petter, Jonas; Pfänder, Jan; Pfuhl, Gerit; Phillips, Joseph; Pietryka, Matthew T; Pirrone, Angelo; Pit, Ilse L; Plachti, Anna; Plank, Irene Sophia; Ploner, Matteo; Poldrack, Russell A; Pollmann, Monique MH; Porcher, Simon; Präg, Patrick; Pua, Andrew Adrian Y; Pugel, Jessica; Puri, Rohan; Püski, Marcell; Radkani, Setayesh; Raes, Louis; Rafaï, Ismaël; Raiber, Klara; Rathje, Steve; Rehms, Raphael; Reshetnikov, Mikhail; Reynolds, Caleb J; Reynolds, James P; Rigaud, Kévin; Rioux, Charlie; Rivera, Sebastian; Robertson, Olly; Román-Caballero, Rafael; Ropovik, Ivan; Röseler, Lukas; Ross, Robert M; Rotella, Amanda; Rüffer, Franziska F; Rusche, Felix; Rusconi, Massimo; Russo, Irene; Sahm, Alexander HJ; Salamon, Janos; Samahita, Margaret; Sanaei, Ali; Sangchooli, Arshiya; Sarafoglou, Alexandra; Scandola, Michele; Schaak, Henning; Schaerer, Michael; Schares, Eric; Schilling, Hayden T; Schmalz, Xenia; Schmidt, Kathleen; Schonberg, Tom; Schreiner, Marcel R; Schröder, Joris M; Schubert, Anna-Lena; Schuetze, Brendan; Schultz, Douglas H; Schulze, Lars; Schwartz, Shawn T; Schwitter, Nicole; Scoggins, Bermond; Seetahul, Yashvin; Seri, Raffaello; Shanks, David R; Shaw, Stacy T; Shaw, Joseph; Shen, Qiang; Siemroth, Christoph; Sladekova, Martina; Somo, Angela; Sondhi, Arjun; Sonmez, Burak; Spantig, Lisa; Speekenbrink, Maarten; Stamos, Angelos; Stasielowicz, Lukasz; Steckermeier, Leonie C; Steinkamp, Simon R; Stoevenbelt, Andrea H; Street, Chris NH; Suchow, Jordan W; Sunde, Hans Fredrik; Sundquist, James; Suschevskiy, Vsevolod; Swain, Scott D; Szecsi, Peter; Szekely-Copîndean, Raluca D; Szumowska, Ewa; Tacconelli, Alessandro; Talbert, Eli; Tang, John P; Tendeiro, Jorge N; Testori, Martina; Toffalini, Enrico; Tomašević, Aleksandar; Topel, Selin; Torkkeli, Lasse; Tozzi, Leonardo; Traczyk, Jakub; Trinidad, Alexander; Trübutschek, Darinka; Turek, Konrad; Uhlich, Maximiliane; Uhlmann, Eric L; Urbanska, Karolina; Van Assche, Jasper; van Assen, Marcel ALM; van Dongen, Noah NN; van Lieshout, Kenny; van Veldhuizen, Roel; Varga, Marton A; Vaughn, Leigh Ann; Venczel, Fruzsina; Vezzoli, Michela; Vierus, Paul; Visalli, Antonino; Voldal, Emily; Votta, Fabio; Wagenmakers, Eric-Jan; Waldendorf, Anica; Walker, Matthew J; Wall, Matthew B; Wallen, Henri; Wang, Ke; Wang, Iris; Wang, Y Andre; Weinmann, Markus; Weiß, Martin; Westheide, Christian; Wichman, Aaron; Wilcke, Juliane C; Williams, Benedict J; Wisniewski, David; Woiczyk, Thomas KA; Woźniak, Mateusz; Wright, Joshua D; Youyou, Wu; Wulff, Jesper N; Yang, Tao; Yeung, Siu Kit; Yuen, Kenneth SL; Zawistowski, Michał; Zein, Rizqy A; Zhao, Xian; Zheng, Zefan; Zhou, Steven; Ziller, Conrad; Zimmerman, David; Zogmaister, Cristina; Zultan, Ro'i; Fox, Nicholas; Errington, Timothy M; Nosek, Brian A |
| Publisher Information: |
Springer Science and Business Media LLC; Department of Architecture; //doi.org/10.1038/s41586-025-09844-9 |
| Publication Year: |
2026 |
| Collection: |
Apollo - University of Cambridge Repository |
| Subject Terms: |
Social Sciences; Behavioral Sciences; Reproducibility of Results; Humans; Research Design; Behavioral Research |
| Description: |
The same dataset can be analysed in different justifiable ways to answer the same research question, potentially challenging the robustness of empirical science1-3. In this crowd initiative, we investigated the degree to which research findings in the social and behavioural sciences are contingent on analysts' choices. We examined a stratified random sample of 100 studies published between 2009 and 2018, in which, for one claim per study, at least five reanalysts independently reanalysed the original data. The statistical appropriateness of the reanalyses was assessed in peer evaluations, and the robustness indicators were inspected along a range of research characteristics and study designs. We found that 34% of the independent reanalyses yielded the same result (within a tolerance region of ±0.05 Cohen's d) as the original report; with a four times broader tolerance region, this indicator increased to 57%. Of the reanalyses conducted, 74% reached the same conclusion as the original investigation, 24% yielded no effects or inconclusive results and 2% reported the opposite effect. This exploratory study indicates that the common single-path analyses in social and behavioural research should not be simply assumed to be robust to alternative analyses4. Therefore, we recommend the development and use of practices to explore and communicate this neglected source of uncertainty. |
| Document Type: |
article in journal/newspaper |
| File Description: |
application/pdf |
| Language: |
English |
| Relation: |
https://www.repository.cam.ac.uk/handle/1810/392107; https://doi.org/10.17863/CAM.122939 |
| DOI: |
10.17863/CAM.122939 |
| Availability: |
https://www.repository.cam.ac.uk/handle/1810/392107; https://doi.org/10.17863/CAM.122939 |
| Rights: |
Attribution 4.0 International ; https://creativecommons.org/licenses/by/4.0/ |
| Accession Number: |
edsbas.E386EC07 |
| Database: |
BASE |