Critical Analysis in Physical Activity, Health, Sport and Rehab

When reading a paper, some reflections could be described as Descriptive (e.g. how many people as participants, what type of study is it, what type of analysis was used, what were the conclusions, what limitations were highlighted) and some reflections could be described as Analytical (e.g. was it the best method of analysis, what limitations were not mentioned but would add context)

• When starting a "critique" of a study/paper, it helps to do an initial skim read/analysis of the paper. Questions like this can help:
  
  

  • What type of study/paper/book is it?

  • When was it published?

  • What is it trying to find out?

  • What method has it used?

  • Do the findings provide enough insight to justify the conclusion?

  • How might the study/findings relate to other work in the field?

  • What are the strengths and weaknesses of the study?

Task: Identify a paper/study and conduct a preliminary, surface level review of it using the tips above

After an initial review of the paper, it is sometimes necessary to conduct a deep-dive analysis of a study. The following section offers suggestions for doing this. 

TASK: review the tips below, then pick the most suitable checklist from the CASP resource at the bottom of this section and review that. Then revisit your chosen paper and conduct a second review using this insight. 

The following tips and recommendations are designed to work with both qualitative and quantitative research 

Key Point: Critical analysis usually means offering context and perspective on a paper rather than undermining its conclusions. "Critical analysis", or "critique" does not often result in a full and extensive "takedown" of a paper as most published papers are good in many ways - that's why they were published. They're usually produced by academics and practitioners with experience and training and they have usually been well-planned, and the write-up has had multiple edits before going for peer-review. The peer-review process will not spot every limitation with a study but it will act as a form of filter, rejecting most poor work and helping to improve the final version of any accepted work. With that in mind, we can proceed to review papers with a "critical", balanced, contextual approach. 

• As you read, it is important to take notes and highlight key passages. This will help you to identify the main points of the paper and to keep track of your thoughts and reactions.

• After you have finished reading the paper, it is helpful to write a summary of the main points and to reflect on your own understanding of the paper.

• Examine the methods used in the study. Carefully consider the methods that the author used to collect and analyze their data. Are the methods appropriate for the research question? Are the data reliable and valid?

• Be aware of the limitations of the study. No study is perfect, and all studies have limitations. The author should discuss the limitations of their study in the paper. Be critical of the author's discussion of the limitations and consider how they might affect the results of the study.

• Compare and contrast the paper with other work in the field. Consider how the paper's findings relate to other work in the field. Are the findings consistent with other research? Are there any contradictions?

• Consider any biases that might be present in the study. This could be in the design, or in the interpretation. It could be acknowledged by the author(s) or it might not be. NB: Identifying a bias does not invalidate a research paper's findings, it just gives it more context and perspective. [For more on types of bias, see this link https://catalogofbias.org/]

• Be aware of your own biases. We all have our own biases, and it is important to be aware of them when we are reading papers. Try to be as objective as possible and to avoid letting your own biases influence your interpretation of the paper.

Recommended Resource: 

• Critical Appraisal Checklists - Once you have identified what type of study you are reviewing you can pick an appropriate checklist https://casp-uk.net/casp-tools-checklists/   

Also See Critical Appraisal Checklist below

Critical analysis checklists are tools that can be used to evaluate the quality of peer-reviewed papers. They typically consist of a list of questions or criteria that should be considered when reading a paper. Critical analysis checklists can be used to evaluate the following aspects of a paper:

A brief example of a critical analysis checklist for a randomized controlled trial (RCT) could be: 

• Introduction: Is the PICO question clearly stated? Is the background information relevant and accurate?

• Methodology: Are the study participants randomly allocated to the intervention and control groups? Are the groups similar in terms of baseline characteristics? Is the intervention clearly described?

• Results: Are the results presented clearly and accurately? Are the results statistically significant?

• Discussion: Are the results interpreted in a reasonable and balanced way? Are the limitations of the study discussed?

• Conclusion: Are the conclusions supported by the evidence? Are the implications of the study discussed?

The strengths of using critical analysis checklists are that they:

• Can help to ensure that all aspects of a paper are evaluated.

• Can help to identify potential biases or flaws in a paper.

• Can help to improve the quality of research.

• Can help to educate readers about the research process.

The limitations of using critical analysis checklists are that they: 

• Can be time-consuming to use.

• Can be subjective.

• May not be appropriate for all types of research.

Tips for using a checklist: 

• Choose a checklist that is appropriate for the type of research study you are evaluating.

• Use the checklist as a guide, but do not rely on it solely.

• Discuss the paper with others.

Examples of Checklists

• CASP - https://casp-uk.net/casp-tools-checklists/ 

• SIGN - https://www.sign.ac.uk/what-we-do/methodology/checklists/ 

• CEBM - https://www.cebm.ox.ac.uk/resources/ebm-tools/critical-appraisal-tools 

• University of Glasgow EBP Checklists - https://www.gla.ac.uk/schools/healthwellbeing/research/generalpractice/ebp/checklists/#d.en.19536 

• GRADE-CERQual - For reviewing qualitative research - https://www.cerqual.org/ 

• University of Calgary: Critique/Review of Research Article https://www.ucalgary.ca/live-uc-ucalgary-site/sites/default/files/teams/9/critique-or-reviews-of-research-articles-academic-genre.pdf 

• JBI Checklist for systematic reviews and research syntheses - https://jbi.global/sites/default/files/2019-05/JBI_Critical_Appraisal-Checklist_for_Systematic_Reviews2017_0.pdf 

This sample exercise could be followed in a similar way to sample exercise #1 if you have a particular interest in physiology, nutrition, and resistance training. 

Journal article for this workshop

• Wilson, J. M., Lowery, R. P., Roberts, M. D., Sharp, M. H., Joy, J. M., Shields, K. A., ... & D'Agostino, D. P. (2020). Effects of ketogenic dieting on body composition, strength, power, and hormonal profiles in resistance training men. The Journal of Strength & Conditioning Research, 34(12), 3463-3474. https://doi.org/10.1519/jsc.0000000000001935

Task

• Read the paper above and produce some bullet point notes that summarise the essential information we need to know about this paper

• What makes this a good paper/study?

• What potential critiques of the paper/study can we find? What are we looking for even if we're not sure about the actual detail of this paper?

Teaching Input

• Only once you have read the article above, and made notes as outlined in the task above should you open the following resource.

• An online article that has conducted a critique of the article https://sci-fit.net/wilson-keto-analysis/

Case Study: Read the King et al paper below and conduct a brief critical analysis. Then read the comment paper and tweet/replies listed below. 

• The original paper in question: King, D. G., Stride, E., Mendis, J., Gurton, W. H., Macrae, H., Jones, L., & Hunt, J. (2023). A Double-Blind, Randomized, Placebo-Controlled Pilot Study examining an Oxygen Nanobubble Beverage for 16.1-km Time Trial and Repeated Sprint Cycling Performance. Journal of Dietary Supplements, 1-15. https://doi.org/10.1080/19390211.2023.2203738 

• The comment paper by Tiller et al: Tiller, N. B., & Jeukendrup, A. E. (2023). Comment On:“A Double-Blind, Randomized, Placebo-Controlled Pilot Study Examining an Oxygen Nanobubble Beverage for 16.1-km Time Trial and Repeated Sprint Cycling Performance.”. Journal of Dietary Supplements, 1-3. https://doi.org/10.1080/19390211.2023.2263563 

• A Tweet with replies/comments: Tiller, N. [@NBTiller]. (2023, October 4). A new study claims that oxygenated water containing "nanobubbles" significantly improves power output during cycling by 4-7%. @Jeukendrop and I had something to say. https://twitter.com/NBTiller/status/1709574746252206255 

Task

• Pick one of the papers below (quant or qual) and spend five minutes doing an initial sweep of the paper. Aim to identify as many key features of the study that might help form the foundation of a "critical analysis". What are the strengths and limitations of the studies approach?

• Key Learning Objective: It is possible to say something meaningful about a study from a few minutes of concentrated reading. 

Sample Papers for Analysis

A quantitative approach: Niering, M., & Muehlbauer, T. (2023). Changes After a Conventional vs. an Alternative Therapy Program on Physical, Psychological, and Injury-Related Parameters in Male Youth Soccer Players With Patellar Tendinopathy During Return to Competition. The Journal of Strength & Conditioning Research, 37(9), 1834-1843. https://doi.org/10.1519/jsc.0000000000004467 

A qualitative approach: Williams, T., Evans, L., Robertson, A., Hardy, L., Roy, S., & Lewis, D. (2021). Distinguishing characteristics between high and low adherence patients following anterior cruciate ligament reconstruction: A qualitative examination. The Sport Psychologist, 36(1), 61-72. https://doi.org/10.1123/tsp.2020-0035 

Contextual Reading 

Further optional reading on the topic of psychology of injury

• Alvarez, G., Núñez-Cortés, R., Solà, I., Sitjà-Rabert, M., Fort-Vanmeerhaeghe, A., Fernández, C., ... & Urrútia, G. (2021). Sample size, study length, and inadequate controls were the most common self-acknowledged limitations in manual therapy trials: A methodological review. Journal of Clinical Epidemiology, 130, 96-106. https://doi.org/10.1016/j.jclinepi.2020.10.018 

• Goddard, K., Roberts, C. M., Byron-Daniel, J., & Woodford, L. (2021). Psychological factors involved in adherence to sport injury rehabilitation: A systematic review. International Review of Sport and Exercise Psychology, 14(1), 51-73.

• Gledhill, A., & Forsdyke, D. (Eds.). (2021). The psychology of sports injury: from risk to retirement. Routledge. 

Critical writing is essential for critical thinking. It is a way to process and synthesise information, develop and evaluate arguments, and communicate your thoughts and ideas in a clear and concise way.

When you write critically, you are not simply summarizing or regurgitating information. You are actively engaging with the material, analyzing it, and evaluating it. You are also developing your own ideas and perspectives, and supporting them with evidence.

This process of critical thinking can help you to:

• Understand complex topics more deeply. When you write about something, you are forced to think about it in more detail. You have to consider the different perspectives, weigh the evidence, and come to your own conclusions.

• Develop your own ideas and perspectives. Critical writing helps you to develop your own voice and to think for yourself. You are not simply parroting back what others have said. You are using your own knowledge and understanding to develop your own arguments.

• Communicate your thoughts and ideas more effectively. Critical writing teaches you to be clear, concise, and persuasive in your writing. You learn to communicate your ideas in a way that others can understand and appreciate.

Critical Writing Resources

• University of York - Criticality in Academic writing - https://subjectguides.york.ac.uk/academic-writing/criticality 

• More on this website -  Writing 

• Makin, T. R., & Orban de Xivry, J. J. (2019). Ten common statistical mistakes to watch out for when writing or reviewing a manuscript. Elife, 8, e48175. https://doi.org/10.7554/eLife.48175

• Reinhart, A. (2015). Statistics done wrong: The woefully complete guide. No starch press. [full e-book at https://www.statisticsdonewrong.com/]

• Kass, R. E., Caffo, B. S., Davidian, M., Meng, X. L., Yu, B., & Reid, N. (2016). Ten simple rules for effective statistical practice. PLoS computational biology, 12(6), e1004961. https://doi.org/10.1371/journal.pcbi.1004961  

• Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129-133. https://doi.org/10.1080/00031305.2016.1154108  

• More on this website - Statistical Analysis in Sport & Exercise Science 

A high-quality systematic review should clearly describe its comprehensive and unbiased search methods, use appropriate tools to assess the quality and risk of bias in the included studies, employ suitable data extraction and synthesis techniques, and draw conclusions that are well-supported by the evidence while acknowledging any limitations or uncertainties. The review should be transparent about its methods and allow readers to critically evaluate the validity and reliability of the findings. Key aspects to assess include the thoroughness of the literature search, the rigor of the quality appraisal, the appropriateness of the analysis methods, and the alignment between the conclusions and the synthesized evidence. 

• Ekkekakis, P. (2015). Honey, I shrunk the pooled SMD! Guide to critical appraisal of systematic reviews and meta-analyses using the Cochrane review on exercise for depression as example. Mental health and physical activity, 8, 21-36. https://doi.org/10.1016/j.mhpa.2014.12.001 

• Ekkekakis, P. [@Ekkekakis]. (2023, Nov 8). In November 2020, my students and I discovered a completely fake meta-analysis, now cited more than 100 times. I notified...[Tweet] https://x.com/Ekkekakis/status/1722055398801154105?s=20 

•  but, as shown below, they have no intention to act. Each year, on the anniversary of the discovery, I re-post this thread...

• STORK Winter Series #2 February 24, 2021 Dr. Ekkekakis - A 20-year retrospective of meta-analyses concluding that exercise is ineffective as antidepressant treatment. A thorough analysis of the literature. https://youtu.be/4A6fsvtfEkY?si=yaOE0FlEunwkmoNx 

• JBI Checklist for systematic reviews and research syntheses - https://jbi.global/sites/default/files/2019-05/JBI_Critical_Appraisal-Checklist_for_Systematic_Reviews2017_0.pdf 

• Shaheen, N., Shaheen, A., Ramadan, A., Hefnawy, M. T., Ramadan, A., Ibrahim, I. A., ... & Flouty, O. (2023). Appraising systematic reviews: a comprehensive guide to ensuring validity and reliability. Frontiers in Research Metrics and Analytics, 8, 1268045. https://doi.org/10.3389/frma.2023.1268045 

• Travers et al. (2019) Should this systematic review and meta-analysis change my practice? Part 1: exploring treatment effect and trustworthiness BJSM http://dx.doi.org/10.1136/bjsports-2018-099958 

• Travers et al. (2019) Should this systematic review and meta-analysis change my practice? Part 2: exploring the role of the comparator, diversity, risk of bias and confidence BJSM http://dx.doi.org/10.1136/bjsports-2018-099959 

• UCL (n.d.) Describing and appraising studies - https://library-guides.ucl.ac.uk/systematic-reviews/describing-appraising 

In a world where information is more abundant than ever before, it is more important than ever to be able to think critically and spot bad science. This book will teach you how to do just that.

Calling Bullshit is a comprehensive guide to critical thinking and spotting bad science. It covers a wide range of topics, including:

• The different types of bad science

• The common tactics used by bad scientists

• How to evaluate scientific claims

• How to identify and avoid biases

• How to communicate your findings to others

Links 

• The Book - Bergstrom, C. T., & West, J. D. (2021). Calling bullshit: The art of skepticism in a data-driven world. Random House Trade Paperbacks. https://www.penguinrandomhouse.com/books/563882/calling-bullshit-by-carl-t-bergstrom-and-jevin-d-west/ 

• The University syllabus with links - https://callingbullshit.org/syllabus.html#Spotting  

  • Lecture Videos - https://callingbullshit.org/videos.html 

  • Sample Content - Visualization: Misleading axes on graphs - https://callingbullshit.org/tools/tools_misleading_axes.html 

  • Sample Content - How do you know a paper is legit? - https://callingbullshit.org/tools/tools_legit.html 

• Case Studies in Bullshit - https://callingbullshit.org/case_studies.html 

  • Sample Case Study - World records as measures of senescence - https://callingbullshit.org/case_studies/case_study_track_records.html 

  • Sample Case Study - The gender gap in 100-meter dash times - https://callingbullshit.org/case_studies/case_study_gender_gap_running.html 

• Listen to a sample - https://www.penguinrandomhouse.com/books/563882/calling-bullshit-by-carl-t-bergstrom-and-jevin-d-west/9780593147443/?ref=PRHA41A0D52358A 

Skepticism in Health & Wellbeing

• Tiller, N. (2020). The Skeptic's Guide to Sports Science: Confronting Myths of the Health and Fitness Industry. Routledge. [Publisher site]

• Tiller, N. B., & Phillips, S. M. (2023). How Skepticism (not Cynicism) Can Raise Scientific Standards and Reform the Health and Wellness Industry. International Journal of Sport Nutrition and Exercise Metabolism, 1(aop), 1-5. https://doi.org/10.1123/ijsnem.2023-0037 

Ben Goldacre is a British doctor, science writer, and academic. He has been a leading figure in the movement to promote science literacy and to combat the spread of misinformation. He is well known for his two seminal books on bad science, Bad Science (2008) and Bad Pharma (2012).

In Bad Science, Goldacre exposes the widespread presence of bad science in popular culture and the media. He debunks common myths and misconceptions about health, nutrition, and medicine, and he criticizes the way that science is often misrepresented or distorted in the news.

In Bad Pharma, Goldacre turns his attention to the pharmaceutical industry. He argues that the industry is corrupt and that it puts profits ahead of patients' best interests. He also criticizes the way that clinical trials are often designed and conducted, and he highlights the dangers of over-prescription of drugs.

Goldacre's work has also had a significant impact on public policy. In the UK, he helped to establish the AllTrials register, which requires all clinical trials to be registered and their results published. He has also been a vocal advocate for open access to scientific data and research.

• Goldacre, B. (2009). Bad science. Fourth Estate. 

• Goldacre, B. (2013). Bad pharma: how drug companies mislead doctors and harm patients. Fourth Estate. 

Stemming from original work by David Sackett, and to better understand the persistent presence, diversity, and impact of biases, the Centre of Evidence Based Medicine at Oxford have compiled and maintained the Catalogue of Bias. The entries describe a wide range of biases –  outlining their potential impact in research studies.

• The website - https://catalogofbias.org/ 

• The list of biases - https://catalogofbias.org/biases/ 

• Example #1: Confounding bias https://catalogofbias.org/biases/confounding/ 

Books

• Lack, C. W., & Rousseau, J. (2016). Critical thinking, science, and pseudoscience: Why we can't trust our brains. Springer Publishing Company. 

• Tapper, K. (2021). Health Psychology and Behaviour Change: From Science to Practice. Bloomsbury Publishing. [Ch 11: Bias and Barriers]

• Cottrell, S. (2011). Critical thinking skills: Developing effective analysis and argument. Palgrave Macmillan. [contents] 

• Katz, L. (2018) Critical Thinking and Persuasive Writing for Postgraduates. Palgrave Macmillan. [google books]

• Wallace, M. and Wray, A. (2011) Critical Reading and Writing for Postgraduates (2nd ed) London: Sage [amazon] 

• Morrow, D. R., & Weston, A. (2011). A workbook for arguments: A complete course in critical thinking. Hackett Publishing. [chap 1] [companion website] 

Other sources

• Lewis, K., & Burton, K. (2020). Time to turn the hierarchy of evidence on its head?. Sport and Exercise Scientist, (64), 10-11. https://www.bases.org.uk/imgs/0000_bas_tses___summer_2020_online___editors_choice_pg10_1176015.pdf  

• Subramanyam, R. V. (2013). Art of reading a journal article: Methodically and effectively. Journal of oral and maxillofacial pathology: JOMFP, 17(1), 65. https://doi.org/10.4103%2F0973-029X.110733 

• UCL Critical Appraisal of a journal article -  https://www.ucl.ac.uk/child-health/sites/child-health/files/library_critical_appraisal_handout.pdf 

• CEBM: The Catalogue of Bias - https://catalogofbias.org/biases/ 

Research Bias in Exercise Science

• The article below is a great read on quantitative research that should be of interest to anyone interested in the human sciences. It lays out common errors and biases and gives some good examples. the video below is part of that article. 

  • Tzur, A. (2022) Research Limitations – Why Study Results Don’t Always Apply to You - https://sci-fit.net/research-limitations/

  • Video: Research Bias in Exercise Science - https://youtu.be/FK7bATcil8M?si=_zKsISKyzpyLza-P 

Case Study

Nunan, D. (2019) Salt reduction in heart failure: where has the uncertainty come from? https://www.cebm.ox.ac.uk/news/views/salt-reduction-in-heart-failure-where-has-the-uncertainty-come-from 

Follow on Twitter

Jacques Rousseau - https://twitter.com/JacquesR   or https://bsky.app/profile/jacquesr.bsky.social 

Carl Bergstrom - https://twitter.com/CT_Bergstrom  or https://bsky.app/profile/carlbergstrom.com 

David Nunan - @dnunan79