Download
Share
NOT PEER-REVIEWED
"PeerJ Preprints" is a venue for early communication or feedback before peer review. Data may be preliminary.

Manipulating the alpha level cannot cure significance testing

, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
1 Department of Psychology, New Mexico State University, Las Cruces, U.S.A.
2 Zoological Institute, University of Basel, Basel, Switzerland
3 Centre for Neuroscience Studies, Queens University, Ontario, Canada
4 Faculty of Applied and Exact Sciences, Metropolitan Technological Institute, Medellín, Colombia
5 School of Mathematical & Physical Sciences, University of Newcastle, Callaghan, Australia
6 Department of Mathematics, State University of New York at Geneseo, Geneseo, U.S.A.
7 Faculty of Psychology, University of Barcelona, Barcelona, Spain
8 Department of Psychology, University of New Brunswick, Saint John, Canada
9 Independent Researcher, New York, U.S.A.
10 School of Psychology, Benemérita Universidad Autónoma de Puebla, Puebla, México
11 School of Psychology, Universidad Adolfo Ibáñez, Santiago, Chile
12 Oncology Laboratory, CCT CONICET Mendoza, Mendoza, Argentina
13 School of Statistics, National University of Colombia, Medellín, Colombia
14 School of Psychology, University of Ottawa, Ottawa, Canada
15 Department of Health Sciences, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands
16 Department of Mathematics and Statistics, IIT Kanpur, Kanpur, India
17 Jessenius Faculty of Medicine, Comenius University, Bratislava, Slovakia
18 Department of Psychology, Oklahoma State University, Stillwater, U.S.A.
19 Faculty of Statistics, Saint Thomas University, Bogotá, Colombia
20 Department of Allied Health Sciences, University of Connecticut, Storrs, U.S.A.
21 Simón Bolívar University, Caracas, Venezuela
22 Department of Cognitive Science and Psychology, New Bulgarian University, Sofia, Bulgaria
23 Department of Biochemistry, Microbiology, and Immunology, University of Ottawa, Ottawa, Canada
24 Oikostat GmbH, Ettiswil, Switzerland
25 Department of Education, Yamaguchi University, Yamaguchi, Japan
26 Multimodal Interaction Lab, Leibniz Knowledge Media Research Center, Tübingen, Germany
27 Facultad de Psicología, Universidad Nacional de Mar del Plata, Buenos Aires, Argentina
28 School of Pedagogy, Pontifical Catholic University of Valparaíso, Valparaíso, Chile
29 Department of Medical and Surgical Sciences, “Magna Graecia” University of Catanzaro, Catanzaro, Italy
30 Economics Department, University of Campania “Luigi Vanvitelli”, Capua, Italy
31 Department of Psychosomatic Medicine, University of Basel, Basel, Switzerland
32 Laboratoire de Psychologie et NeuroCognition, Univ. Grenoble Alpes, Grenoble, France
33 Faculty of Economics, Chiang Mai University, Chiang Mai, Thailand
34 Department of Statistics, Universidade Federal de Pernambuco, Recife, Brazil
35 Business School, Massey University, Massey, New Zealand
36 Department of Psychology III, University of Würzburg, Würzburg, Germany
37 School of Psychology, National Autonomous University of Mexico, Mexico City, Mexico
38 Faculty of Engineering, University of Porto, Porto, Portugal
39 Department of Psychology, Universidad del Norte, Barranquilla, Colombia
40 Department of Psychology, University of Basel, Basel, Switzerland
41 Facultad de Ciencias Naturales y Exactas, Universidad de Playa Ancha, Valparaíso, Chile
42 Department of Methods and Statistics, Utrecht University, Utrecht, The Netherlands
43 Centre for Mathematical Morphology, MINES Paristech, Paris, France
44 Department of Mathematical Sciences, New Mexico State University, Las Cruces, U.S.A.
45 Faculty of Arts and Science, Kyushu University, Hakozaki, Japan
46 School of Psychology, The University of Adelaide, Adelaide, Australia
DOI
10.7287/peerj.preprints.3411v3
Published
Accepted
Subject Areas
Science Policy, Statistics
Keywords
P-value, Significance, NHST, Alpha level, Threshold, Publication bias, Winner’s curse, Replicability, Bayes factor, Statistics
Copyright
© 2018 Trafimow et al.
Licence
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ Preprints) and either DOI or URL of the article must be cited.
Cite this article
Trafimow D, Amrhein V, Areshenkoff CN, Barrera-Causil C, Beh EJ, Bilgiç Y, Bono R, Bradley MT, Briggs WM, Cepeda-Freyre HA, Chaigneau SE, Ciocca DR, Carlos Correa J, Cousineau D, de Boer MR, Dhar SS, Dolgov I, Gómez-Benito J, Grendar M, Grice J, Guerrero-Gimenez ME, Gutiérrez A, Huedo-Medina TB, Jaffe K, Janyan A, Karimnezhad A, Korner-Nievergelt F, Kosugi K, Lachmair M, Ledesma R, Limongi R, Liuzza MT, Lombardo R, Marks M, Meinlschmidt G, Nalborczyk L, Nguyen HT, Ospina R, Perezgonzalez JD, Pfister R, Rahona JJ, Rodríguez-Medina DA, Romão X, Ruiz-Fernández S, Suarez I, Tegethoff M, Tejo M, van de Schoot R, Vankov I, Velasco-Forero S, Wang T, Yamada Y, Zoppino FC, Marmolejo-Ramos F. 2018. Manipulating the alpha level cannot cure significance testing. PeerJ Preprints 6:e3411v3

Abstract

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p= .05 to .005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of .05, .01, .005, or anything else, is not acceptable.

Author Comment

Published on 15 May 2018 in Frontiers in Psychology 9, 699, https://doi.org/10.3389/fpsyg.2018.00699

Additional Information

Competing Interests

FK-N is an employee of Oikostat GmbH. GM has been acting as consultant for Janssen Research & Development, LLC.

Author Contributions

David Trafimow conceived and designed the experiments, authored or reviewed drafts of the paper, approved the final draft.

Valentin Amrhein conceived and designed the experiments, authored or reviewed drafts of the paper, approved the final draft.

Corson N. Areshenkoff authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Carlos Barrera-Causil authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Eric J. Beh authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Yusuf Bilgiç authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Roser Bono authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Michael T. Bradley authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

William M. Briggs authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Héctor A. Cepeda-Freyre authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Sergio E. Chaigneau authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Daniel R. Ciocca authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Juan Carlos Correa authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Denis Cousineau authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Michiel R. de Boer authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Subhra Sankar Dhar authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Igor Dolgov authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Juana Gómez-Benito authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Marian Grendar authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

James Grice authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Martin E. Guerrero-Gimenez authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Andrés Gutiérrez authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Tania B. Huedo-Medina authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Klaus Jaffe authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Armina Janyan authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Ali Karimnezhad authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Fränzi Korner-Nievergelt authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Koji Kosugi authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Martin Lachmair authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Rubén Ledesma authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Roberto Limongi authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Marco Tullio Liuzza authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Rosaria Lombardo authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Michael Marks authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Gunther Meinlschmidt authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Ladislas Nalborczyk authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Hung T. Nguyen authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Raydonal Ospina authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Jose D. Perezgonzalez authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Roland Pfister authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Juan José Rahona authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

David A. Rodríguez-Medina authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Xavier Romão authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Susana Ruiz-Fernández authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Isabel Suarez authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Marion Tegethoff authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Mauricio Tejo authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Rens van de Schoot authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Ivan Vankov authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Santiago Velasco-Forero authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Tonghui Wang authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Yuki Yamada authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Felipe C. Zoppino authored or reviewed drafts of the paper, approved the final draft, contributed ideas.

Fernando Marmolejo-Ramos conceived and designed the experiments, authored or reviewed drafts of the paper, approved the final draft.

Data Deposition

The following information was supplied regarding data availability:

This is a comment without empirical data.

Funding

MG received support from VEGA 2/0047/15 grant. RvdS was supported by a grant from the Netherlands organization for scientific research: NWO-VIDI-45-14-006. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Add your feedback

Before adding feedback, consider if it can be asked as a question instead, and if so then use the Question tab. Pointing out typos is fine, but authors are encouraged to accept only substantially helpful feedback.

Some Markdown syntax is allowed: _italic_ **bold** ^superscript^ ~subscript~ %%blockquote%% [link text](link URL)
 
By posting this you agree to PeerJ's commenting policies
Download
Content
Alert
Just enter your email

Download article

Manipulating the alpha level cannot cure significance testing
PDF Save to Mendeley RIS XML BibTeX
4 Citations   Views   Downloads