Critical Analysis in Physical Activity, Health, Sport and Rehab
Critical analysis of information and more specifically peer-reviewed literature is an essential academic skill for any student. Critical analysis involves thinking with a critical perspective, evaluating strengths and weaknesses, and understanding a paper's contributions to the field.
A key point when learning how to do this is that a "critical analysis" is not a negative concept that requires the dismantling and debunking of a paper. It's more likely that you will need to be able to describe the critical aspects of the study and give context to its findings. What does it add to the wider body of knowledge and what does it not add? In some ways, "contextual analysis" is a better way to think about this task.
This page contains resources designed to help improve the reader's ability to critically analyse a study or an argument. Enhancing your knowledge, skills and abilities to review and critique research articles will help your academic grades improve and make you a better practitioner in the future.
Tod, D., Booth, A. and Smith, B. (2021) Critical Appraisal. International Review of Sport and Exercise Psychology, 15(1), 52-72. https://doi.org/10.1080/1750984X.2021.1952471
Page, P. (2021). The Need for Critical Thinking in Rehabilitation Research. International Journal of Sports Physical Therapy, 16(4). https://doi.org/10.26603/001c.27146
How to critique a research paper #1: The first sweep
When reading a paper, some reflections could be described as Descriptive (e.g. how many people as participants, what type of study is it, what type of analysis was used, what were the conclusions, what limitations were highlighted) and some reflections could be described as Analytical (e.g. was it the best method of analysis, what limitations were not mentioned but would add context)
When starting a "critique" of a study/paper, it helps to do an initial skim read/analysis of the paper. Questions like this can help:
What type of study/paper/book is it?
When was it published?
What is it trying to find out?
What method has it used?
Do the findings provide enough insight to justify the conclusion?
How might the study/findings relate to other work in the field?
What are the strengths and weaknesses of the study?
Task: Identify a paper/study and conduct a preliminary, surface level review of it using the tips above
How to critique a research paper #2: The Deep Dive
After an initial review of the paper, it is sometimes necessary to conduct a deep-dive analysis of a study. The following section offers suggestions for doing this.
TASK: review the tips below, then pick the most suitable checklist from the CASP resource at the bottom of this section and review that. Then revisit your chosen paper and conduct a second review using this insight.
The following tips and recommendations are designed to work with both qualitative and quantitative research
Key Point: Critical analysis usually means offering context and perspective on a paper rather than undermining its conclusions. "Critical analysis", or "critique" does not often result in a full and extensive "takedown" of a paper as most published papers are good in many ways - that's why they were published. They're usually produced by academics and practitioners with experience and training and they have usually been well-planned, and the write-up has had multiple edits before going for peer-review. The peer-review process will not spot every limitation with a study but it will act as a form of filter, rejecting most poor work and helping to improve the final version of any accepted work. With that in mind, we can proceed to review papers with a "critical", balanced, contextual approach.
As you read, it is important to take notes and highlight key passages. This will help you to identify the main points of the paper and to keep track of your thoughts and reactions.
After you have finished reading the paper, it is helpful to write a summary of the main points and to reflect on your own understanding of the paper.
Examine the methods used in the study. Carefully consider the methods that the author used to collect and analyze their data. Are the methods appropriate for the research question? Are the data reliable and valid?
Be aware of the limitations of the study. No study is perfect, and all studies have limitations. The author should discuss the limitations of their study in the paper. Be critical of the author's discussion of the limitations and consider how they might affect the results of the study.
Compare and contrast the paper with other work in the field. Consider how the paper's findings relate to other work in the field. Are the findings consistent with other research? Are there any contradictions?
Consider any biases that might be present in the study. This could be in the design, or in the interpretation. It could be acknowledged by the author(s) or it might not be. NB: Identifying a bias does not invalidate a research paper's findings, it just gives it more context and perspective. [For more on types of bias, see this link https://catalogofbias.org/]
Be aware of your own biases. We all have our own biases, and it is important to be aware of them when we are reading papers. Try to be as objective as possible and to avoid letting your own biases influence your interpretation of the paper.
Critical Appraisal Checklists - Once you have identified what type of study you are reviewing you can pick an appropriate checklist https://casp-uk.net/casp-tools-checklists/
Also See Critical Appraisal Checklist below
Critical Appraisal Checklists
Critical analysis checklists are tools that can be used to evaluate the quality of peer-reviewed papers. They typically consist of a list of questions or criteria that should be considered when reading a paper. Critical analysis checklists can be used to evaluate the following aspects of a paper:
A brief example of a critical analysis checklist for a randomized controlled trial (RCT) could be:
Introduction: Is the PICO question clearly stated? Is the background information relevant and accurate?
Methodology: Are the study participants randomly allocated to the intervention and control groups? Are the groups similar in terms of baseline characteristics? Is the intervention clearly described?
Results: Are the results presented clearly and accurately? Are the results statistically significant?
Discussion: Are the results interpreted in a reasonable and balanced way? Are the limitations of the study discussed?
Conclusion: Are the conclusions supported by the evidence? Are the implications of the study discussed?
The strengths of using critical analysis checklists are that they:
Can help to ensure that all aspects of a paper are evaluated.
Can help to identify potential biases or flaws in a paper.
Can help to improve the quality of research.
Can help to educate readers about the research process.
The limitations of using critical analysis checklists are that they:
Can be time-consuming to use.
Can be subjective.
May not be appropriate for all types of research.
Tips for using a checklist:
Choose a checklist that is appropriate for the type of research study you are evaluating.
Use the checklist as a guide, but do not rely on it solely.
Discuss the paper with others.
Examples of Checklists
University of Glasgow EBP Checklists - https://www.gla.ac.uk/schools/healthwellbeing/research/generalpractice/ebp/checklists/#d.en.19536
GRADE-CERQual - For reviewing qualitative research - https://www.cerqual.org/
University of Calgary: Critique/Review of Research Article https://www.ucalgary.ca/live-uc-ucalgary-site/sites/default/files/teams/9/critique-or-reviews-of-research-articles-academic-genre.pdf
JBI Checklist for systematic reviews and research syntheses - https://jbi.global/sites/default/files/2019-05/JBI_Critical_Appraisal-Checklist_for_Systematic_Reviews2017_0.pdf
Case Study # 1: Ketogenesis and resistance training
This sample exercise could be followed in a similar way to sample exercise #1 if you have a particular interest in physiology, nutrition, and resistance training.
Journal article for this workshop
Wilson, J. M., Lowery, R. P., Roberts, M. D., Sharp, M. H., Joy, J. M., Shields, K. A., ... & D'Agostino, D. P. (2020). Effects of ketogenic dieting on body composition, strength, power, and hormonal profiles in resistance training men. The Journal of Strength & Conditioning Research, 34(12), 3463-3474. https://doi.org/10.1519/jsc.0000000000001935
Read the paper above and produce some bullet point notes that summarise the essential information we need to know about this paper
What makes this a good paper/study?
What potential critiques of the paper/study can we find? What are we looking for even if we're not sure about the actual detail of this paper?
Only once you have read the article above, and made notes as outlined in the task above should you open the following resource.
An online article that has conducted a critique of the article https://sci-fit.net/wilson-keto-analysis/
Case Study #2: Ergogenics in sport | Nanobubbles and sprint cycling
Case Study: Read the King et al paper below and conduct a brief critical analysis. Then read the comment paper and tweet/replies listed below.
The original paper in question: King, D. G., Stride, E., Mendis, J., Gurton, W. H., Macrae, H., Jones, L., & Hunt, J. (2023). A Double-Blind, Randomized, Placebo-Controlled Pilot Study examining an Oxygen Nanobubble Beverage for 16.1-km Time Trial and Repeated Sprint Cycling Performance. Journal of Dietary Supplements, 1-15. https://doi.org/10.1080/19390211.2023.2203738
The comment paper by Tiller et al: Tiller, N. B., & Jeukendrup, A. E. (2023). Comment On:“A Double-Blind, Randomized, Placebo-Controlled Pilot Study Examining an Oxygen Nanobubble Beverage for 16.1-km Time Trial and Repeated Sprint Cycling Performance.”. Journal of Dietary Supplements, 1-3. https://doi.org/10.1080/19390211.2023.2263563
A Tweet with replies/comments: Tiller, N. [@NBTiller]. (2023, October 4). A new study claims that oxygenated water containing "nanobubbles" significantly improves power output during cycling by 4-7%. @Jeukendrop and I had something to say. https://twitter.com/NBTiller/status/1709574746252206255
Case Study #3: The Psychology of Injury
Pick one of the papers below (quant or qual) and spend five minutes doing an initial sweep of the paper. Aim to identify as many key features of the study that might help form the foundation of a "critical analysis". What are the strengths and limitations of the studies approach?
Key Learning Objective: It is possible to say something meaningful about a study from a few minutes of concentrated reading.
Sample Papers for Analysis
A quantitative approach: Niering, M., & Muehlbauer, T. (2023). Changes After a Conventional vs. an Alternative Therapy Program on Physical, Psychological, and Injury-Related Parameters in Male Youth Soccer Players With Patellar Tendinopathy During Return to Competition. The Journal of Strength & Conditioning Research, 37(9), 1834-1843. https://doi.org/10.1519/jsc.0000000000004467
A qualitative approach: Williams, T., Evans, L., Robertson, A., Hardy, L., Roy, S., & Lewis, D. (2021). Distinguishing characteristics between high and low adherence patients following anterior cruciate ligament reconstruction: A qualitative examination. The Sport Psychologist, 36(1), 61-72. https://doi.org/10.1123/tsp.2020-0035
Further optional reading on the topic of psychology of injury
Goddard, K., Roberts, C. M., Byron-Daniel, J., & Woodford, L. (2021). Psychological factors involved in adherence to sport injury rehabilitation: A systematic review. International Review of Sport and Exercise Psychology, 14(1), 51-73.
Gledhill, A., & Forsdyke, D. (Eds.). (2021). The psychology of sports injury: from risk to retirement. Routledge.
Critical Writing as an aid to Critical Thinking
Critical writing is essential for critical thinking. It is a way to process and synthesise information, develop and evaluate arguments, and communicate your thoughts and ideas in a clear and concise way.
When you write critically, you are not simply summarizing or regurgitating information. You are actively engaging with the material, analyzing it, and evaluating it. You are also developing your own ideas and perspectives, and supporting them with evidence.
This process of critical thinking can help you to:
Understand complex topics more deeply. When you write about something, you are forced to think about it in more detail. You have to consider the different perspectives, weigh the evidence, and come to your own conclusions.
Develop your own ideas and perspectives. Critical writing helps you to develop your own voice and to think for yourself. You are not simply parroting back what others have said. You are using your own knowledge and understanding to develop your own arguments.
Communicate your thoughts and ideas more effectively. Critical writing teaches you to be clear, concise, and persuasive in your writing. You learn to communicate your ideas in a way that others can understand and appreciate.
Critical Writing Resources
University of York - Criticality in Academic writing - https://subjectguides.york.ac.uk/academic-writing/criticality
More on this website - Writing
Critical Analysis of Quantitative/Statistical Studies
Makin, T. R., & Orban de Xivry, J. J. (2019). Ten common statistical mistakes to watch out for when writing or reviewing a manuscript. Elife, 8, e48175. https://doi.org/10.7554/eLife.48175
Reinhart, A. (2015). Statistics done wrong: The woefully complete guide. No starch press. [full e-book at https://www.statisticsdonewrong.com/]
Kass, R. E., Caffo, B. S., Davidian, M., Meng, X. L., Yu, B., & Reid, N. (2016). Ten simple rules for effective statistical practice. PLoS computational biology, 12(6), e1004961. https://doi.org/10.1371/journal.pcbi.1004961
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108
More on this website - Statistical Analysis in Sport & Exercise Science
Critical Analysis of Systematic Reviews/Meta-Analyses
Ekkekakis, P. (2015). Honey, I shrunk the pooled SMD! Guide to critical appraisal of systematic reviews and meta-analyses using the Cochrane review on exercise for depression as example. Mental health and physical activity, 8, 21-36. https://doi.org/10.1016/j.mhpa.2014.12.001
Ekkekakis, P. [@Ekkekakis]. (2023, Nov 8). In November 2020, my students and I discovered a completely fake meta-analysis, now cited more than 100 times. I notified...[Tweet] https://x.com/Ekkekakis/status/1722055398801154105?s=20
but, as shown below, they have no intention to act. Each year, on the anniversary of the discovery, I re-post this thread...
STORK Winter Series #2 February 24, 2021 Dr. Ekkekakis - A 20-year retrospective of meta-analyses concluding that exercise is ineffective as antidepressant treatment. A thorough analysis of the literature. https://youtu.be/4A6fsvtfEkY?si=yaOE0FlEunwkmoNx
JBI Checklist for systematic reviews and research syntheses - https://jbi.global/sites/default/files/2019-05/JBI_Critical_Appraisal-Checklist_for_Systematic_Reviews2017_0.pdf
Travers et al. (2019) Should this systematic review and meta-analysis change my practice? Part 1: exploring treatment effect and trustworthiness BJSM http://dx.doi.org/10.1136/bjsports-2018-099958
Travers et al. (2019) Should this systematic review and meta-analysis change my practice? Part 2: exploring the role of the comparator, diversity, risk of bias and confidence BJSM http://dx.doi.org/10.1136/bjsports-2018-099959
Recommended Book: Calling Bullshit by Bergstrom & West
In a world where information is more abundant than ever before, it is more important than ever to be able to think critically and spot bad science. This book will teach you how to do just that.
Calling Bullshit is a comprehensive guide to critical thinking and spotting bad science. It covers a wide range of topics, including:
The different types of bad science
The common tactics used by bad scientists
How to evaluate scientific claims
How to identify and avoid biases
How to communicate your findings to others
The Book - Bergstrom, C. T., & West, J. D. (2021). Calling bullshit: The art of skepticism in a data-driven world. Random House Trade Paperbacks. https://www.penguinrandomhouse.com/books/563882/calling-bullshit-by-carl-t-bergstrom-and-jevin-d-west/
The University syllabus with links - https://callingbullshit.org/syllabus.html#Spotting
Lecture Videos - https://callingbullshit.org/videos.html
Sample Content - Visualization: Misleading axes on graphs - https://callingbullshit.org/tools/tools_misleading_axes.html
Sample Content - How do you know a paper is legit? - https://callingbullshit.org/tools/tools_legit.html
Case Studies in Bullshit - https://callingbullshit.org/case_studies.html
Sample Case Study - World records as measures of senescence - https://callingbullshit.org/case_studies/case_study_track_records.html
Sample Case Study - The gender gap in 100-meter dash times - https://callingbullshit.org/case_studies/case_study_gender_gap_running.html
Recommended Book: The Skeptic's Guide to Sports Science (2020) by Nick Tiller
Skepticism in Health & Wellbeing
Tiller, N. (2020). The Skeptic's Guide to Sports Science: Confronting Myths of the Health and Fitness Industry. Routledge. [Publisher site]
Tiller, N. B., & Phillips, S. M. (2023). How Skepticism (not Cynicism) Can Raise Scientific Standards and Reform the Health and Wellness Industry. International Journal of Sport Nutrition and Exercise Metabolism, 1(aop), 1-5. https://doi.org/10.1123/ijsnem.2023-0037
Recommended Books: Ben Goldacre
Ben Goldacre is a British doctor, science writer, and academic. He has been a leading figure in the movement to promote science literacy and to combat the spread of misinformation. He is well known for his two seminal books on bad science, Bad Science (2008) and Bad Pharma (2012).
In Bad Science, Goldacre exposes the widespread presence of bad science in popular culture and the media. He debunks common myths and misconceptions about health, nutrition, and medicine, and he criticizes the way that science is often misrepresented or distorted in the news.
In Bad Pharma, Goldacre turns his attention to the pharmaceutical industry. He argues that the industry is corrupt and that it puts profits ahead of patients' best interests. He also criticizes the way that clinical trials are often designed and conducted, and he highlights the dangers of over-prescription of drugs.
Goldacre's work has also had a significant impact on public policy. In the UK, he helped to establish the AllTrials register, which requires all clinical trials to be registered and their results published. He has also been a vocal advocate for open access to scientific data and research.
Goldacre, B. (2009). Bad science. Fourth Estate.
Goldacre, B. (2013). Bad pharma: how drug companies mislead doctors and harm patients. Fourth Estate.
Recommended Resource: CEBM Catalogue of Bias
Stemming from original work by David Sackett, and to better understand the persistent presence, diversity, and impact of biases, the Centre of Evidence Based Medicine at Oxford have compiled and maintained the Catalogue of Bias. The entries describe a wide range of biases – outlining their potential impact in research studies.
Further Reading and Resources
Lack, C. W., & Rousseau, J. (2016). Critical thinking, science, and pseudoscience: Why we can't trust our brains. Springer Publishing Company.
Tapper, K. (2021). Health Psychology and Behaviour Change: From Science to Practice. Bloomsbury Publishing. [Ch 11: Bias and Barriers]
Cottrell, S. (2011). Critical thinking skills: Developing effective analysis and argument. Palgrave Macmillan. [contents]
Lewis, K., & Burton, K. (2020). Time to turn the hierarchy of evidence on its head?. Sport and Exercise Scientist, (64), 10-11. https://www.bases.org.uk/imgs/0000_bas_tses___summer_2020_online___editors_choice_pg10_1176015.pdf
Subramanyam, R. V. (2013). Art of reading a journal article: Methodically and effectively. Journal of oral and maxillofacial pathology: JOMFP, 17(1), 65. https://doi.org/10.4103%2F0973-029X.110733
UCL Critical Appraisal of a journal article - https://www.ucl.ac.uk/child-health/sites/child-health/files/library_critical_appraisal_handout.pdf
Research Bias in Exercise Science
The article below is a great read on quantitative research that should be of interest to anyone interested in the human sciences. It lays out common errors and biases and gives some good examples. the video below is part of that article.
Nunan, D. (2019) Salt reduction in heart failure: where has the uncertainty come from? https://www.cebm.ox.ac.uk/news/views/salt-reduction-in-heart-failure-where-has-the-uncertainty-come-from
Follow on Twitter
Jacques Rousseau - https://twitter.com/JacquesR
Carl Bergstrom (not using) - https://twitter.com/CT_Bergstrom
David Nunan - @dnunan79