Best Approaches To Peer Review Experimental Results

On why you should be measured in the analysis that you ask authors to run (or moving towards a more reasonable peer review system). I recently had an editor complain, "The reviewers ask for so much analysis! It's crazy!" They commented on this after my team had just resubmitted a paper with hundreds of additional analyses requested by a reviewer. That experience lent itself to a lot of contemplation on my morning walk. I've concluded that we need to do better. BUT. We will only do better if we train reviewers how to approach requesting analyses. So, here are my suggestions for asking authors to include additional analysis in a paper. You should begin by asking yourself three questions: (1) Would additional analysis improve the strength, clarity, or validity of the manuscript? (2) Would the suggested analysis be reasonable given the available data and methodological rigor? (3) Will the analysis meaningfully contribute to readers’ confidence in the results? If yes to both, consider mapping your requests to one of the following. Ask for additional analysis when: (1) When the analysis directly addresses unclear results. If an outcome feels ambiguous or unsupported, suggest specific analyses (e.g., sensitivity analyses, robustness checks) that could clarify or strengthen the findings. (3) you feel it is important to test plausible alternative explanations. Consider analyses that rule out plausible competing hypotheses or alternative explanations for the findings. Asking for analyses controlling for key variables or interactions can help solidify conclusions - increase a paper's impact! (4) you need more information on critical assumptions. If uncertain assumptions underlying the paper's method are not met, then clearly request analyses that explicitly test these assumptions, like diagnostic checks, validation samples, or robustness tests. Be specific and clear, so the authors don't get lost in responding. (5) you need to know more about generalizability. Suggest analyses that examine whether findings generalize across sub-samples, contexts, or measurement approaches. For instance, subgroup analyses or replication using alternative operationalizations can strengthen conclusions. And if the authors offer a reasonable explanation why they can't, based on their data, then have them acknowledge the limitations of their work. (6) you want the authors to address a specific, relevant methodological weakness or concern. If you identify a methodological gap or limitation, propose analyses (e.g., different statistical models, corrections for bias, or additional robustness tests) to mitigate the concern. Again, be specific in your request. If we all start asking for reasonable additional analysis, that clearly adds value, increases confidence in results, and directly tackles concerns that influence the paper’s validity or impact, then we will contribute to developing a more reasonable peer review process. Thank you. #academicjourney

Peer review is the cornerstone of scholarly publishing. Some reviewers offer gentle, yet unhelpful feedback. Others may be harsh but give insightful comment. Striking the right balance is key. Let me share my approach on being the 'just right' peer reviewer The are 2 parts Part 1: What to pay attention to (per section) Part 2: Scripts on how to critique politely ----------- Part 1: 📝1️⃣ Abstract: • Is it a short, clear summary of the aims, key methods, important findings, and conclusions? • Can it stand alone? • Does it contain unnecessary information? 🚪2️⃣ Introduction: Study Premise: Is it talking about something new on something old? • Does it summarize the current state of the topic? • Does it address the limitations of current state in this field? • Does it explain why this study was necessary? • Are the aims clear? 🧩3️⃣ Methods: • Study design: right to answer the question? • Population: unbiased? • Data source and collection: clearly defined? • Outcome: accurate, clinically meaningful? • Variables: well justified? • Statistical analysis: right method, sufficient power? • Study robustness: sensitivity analysis, data management. • Ethical concerns addressed? 🎯4️⃣ Results: • Are results presented clearly, accurately, and in order? • Easy to understand? • Tables make sense? • Measures of uncertainty (standard errors/P values) included? 9/16: 📈6️⃣ Figures: • Easy to understand? • Figure legends make sense? • Titles, axis clear? 🌐7️⃣ Discussion: The interpretation. • Did they compare the findings with current literature? • Is there a research argument? (claim + evidence) • Limitations/strengths addressed? • Future direction? 📚8️⃣ References: • Key references missing? • Do the authors cite secondary sources (narrative review papers) instead of the original paper? ------------ Part 2: 🗣️ How do you give your critique politely? Use these scripts. Interesting/useful research question, BUT weak method: - The study premise is strong, but the approach is underdeveloped." Robust research method, BUT the research question is not interesting/useful: -"The research method is robust and well thought out, but the study premise is weak." Bad writing: -"While the study/ research appears to be strong, the writing is difficult to follow. I recommend the authors work with a copyeditor to improve the flow/clarity and readability of the text" Results section do not make sense: -"The data reported in {page x/table y} should be expanded and clarified." Wrong interpretation/ wrong conclusion: -"The authors stated that {***}, but the data does not fully support this conclusion. We can only conclude that {***}. Poor Discussion section -"The authors {did not/fails to} address how their findings relate to the literature in this field." Copy this post into a word document and save it as a template. Use this every time you have to review a paper. If you are the receiver of peer review - you can also use this to decode what the reviewer is saying.😉

Instead of complaining that peer review is dead, take a positive step to improve it today. The reviewers are not aliens, they are us! - Revise your review and make it clear. Identify the crucial points that impacted your score negatively and positively. - If the paper is lacking information about its claims, communicate your asks and the reasoning concretely. Don't just ask for 2 more experiments because you feel the authors didn't work hard enough. Don't ask for experiments unless they verify a hypothesis (which you clearly explained). - Look for the missing information that you identified and make sure they are not in the paper. - Recommend acceptance if the paper's claim is adding a new nugget of information to the literature (no matter the size of the nugget), and if the paper has substantiated the claim via theoretical / empirical evidence. Believe me, this doesn't take much time, and will improve the state of peer review significantly!