Systems Evaluation | Systems Mapping | Systems Leadership
This page is designed to support professional development work around the engagement with systems approaches to intervention development AND the evaluation of interventions that have adopted systems approaches to change.
What is a systems approach to intervention design?
What is a system?
What is system mapping and what is it's purpose?
Discuss the benefits of systems mapping in terms of both process and outcome.
Discuss the difference between evaluating a system and evaluating a systems driven intervention. (Mowles, 2014)
Once a system has been "mapped", how do we know where to intervene? (e.g. Nobles et al, 2021; Meadows 12 places; Intervention Level Framework; Iceberg Model)
Also on this website: Logic Models in Physical Activity Health Promotion
Latest Recommended Resource
Barbrook-Johnson, P., & Penn, A. S. (2022). Systems Mapping: How to build and use causal models of systems. [FREE Open Access Book]
Systems Thinking & Public Health
Bolton, K. A., Whelan, J., Fraser, P., Bell, C., Allender, S., & Brown, A. D. (2022). The Public Health 12 framework: interpreting the ‘Meadows 12 places to act in a system’ for use in public health. Archives of Public Health, 80(1), 1-8. https://doi.org/10.1186/s13690-022-00835-0
Diez Roux, A. V. (2011). Complex systems thinking and current impasses in health disparities research. American journal of public health, 101(9), 1627-1634.
Midgley, G. (2006). Systemic intervention for public health. American journal of public health, 96(3), 466-472.
Luna Pinzon, A., Stronks, K., Dijkstra, C. et al. The ENCOMPASS framework: a practical guide for the evaluation of public health programmes in complex adaptive systems. Int J Behav Nutr Phys Act 19, 33 (2022). https://doi.org/10.1186/s12966-022-01267-3
Published Papers: Systems Thinking & Mapping
Barbrook-Johnson, P., & Penn, A. (2021). Participatory systems mapping for complex energy policy evaluation. Evaluation, 27(1), 57-79. https://doi.org/10.1177%2F1356389020976153
Elsawah, S., Pierce, S. A., Hamilton, S. H., Van Delden, H., Haase, D., Elmahdi, A., & Jakeman, A. J. (2017). An overview of the system dynamics process for integrated modelling of socio-ecological systems: Lessons on good modelling practice from five case studies. Environmental Modelling & Software, 93, 127-145. https://doi.org/10.1016/j.envsoft.2017.03.001
Foster-Fishman, P. G., Nowell, B., & Yang, H. (2007). Putting the system back into systems change: A framework for understanding and changing organizational and community systems. American journal of community psychology, 39(3-4), 197-215. https://doi.org/10.1007/s10464-007-9109-0
Gates, E. F. (2016). Making sense of the emerging conversation in evaluation about systems thinking and complexity science. Evaluation and Program Planning, 59, 62-73. https://doi.org/10.1016/j.evalprogplan.2016.08.004
Green, L.W., Sim, L., Breiner, H. (2013) (Eds.) Ch 9: Systems and Evaluation: Placing a Systems Approach in Context. In Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress. Committee on Evaluating Progress of Obesity Prevention Effort; Food and Nutrition Board; Institute of Medicine. https://www.ncbi.nlm.nih.gov/books/NBK202498/
Grove, J. T. (2015). Aiming for Utility in ‘Systems‐based Evaluation’: A Research‐based Framework for Practitioners. IDS Bulletin, 46(1), 58-70. https://doi.org/10.1111/1759-5436.12121
Hamilton, S. H., Fu, B., Guillaume, J. H., Badham, J., Elsawah, S., Gober, P., ... & Zare, F. (2019). A framework for characterising and evaluating the effectiveness of environmental modelling. Environmental modelling & software, 118, 83-98. https://doi.org/10.1016/j.envsoft.2019.04.008
Hawe, P., Shiell, A., & Riley, T. (2009). Theorising interventions as events in systems. American journal of community psychology, 43(3-4), 267-276. https://doi.org/10.1007/s10464-009-9229-9
Kelly, R. A., Jakeman, A. J., Barreteau, O., Borsuk, M. E., ElSawah, S., Hamilton, S. H., ... & Voinov, A. A. (2013). Selecting among five common modelling approaches for integrated environmental assessment and management. Environmental modelling & software, 47, 159-181. https://doi.org/10.1016/j.envsoft.2013.05.005
McGill, E., Er, V., Penney, T., Egan, M., White, M., Meier, P., ... & Petticrew, M. (2021). Evaluation of public health interventions from a complex systems perspective: a research methods review. Social Science & Medicine, 113697. https://doi.org/10.1016/j.socscimed.2021.113697
Nobles, J. D., Radley, D., Mytton, O. T., & Whole Systems Obesity programme team. (2021). The Action Scales Model: A conceptual tool to identify key points for action within complex adaptive systems. Perspectives in Public Health, 17579139211006747. https://doi.org/10.1177%2F17579139211006747
Nobles J, Wheeler J, Dunleavy-Harris K, et al. (2021) Ripple Effects Mapping: Capturing the Wider Impacts of Systems Change Efforts in Public Health. Research Square. https://doi.org/10.21203/rs.3.rs-1008063/v1
Alford, C. (2017) How systems mapping can help you build a better theory of change. https://blog.kumu.io/how-systems-mapping-can-help-you-build-a-better-theory-of-change-4c85ae4301a8
Madden, H. and Ohlson, I. (2 June 2020) Systems Mapping - a brief overview of what, why noand how (Part 1) - https://mojdigital.blog.gov.uk/2020/06/02/systems-mapping-a-brief-overview-of-what-why-and-how-part-1/
Acaroglu, L. (2017) Tools for Systems Thinkers: The 6 Fundamental Concepts of Systems Thinking - https://medium.com/disruptive-design/tools-for-systems-thinkers-the-6-fundamental-concepts-of-systems-thinking-379cdac3dc6a
Todres, G./NPC (2021) What we’ve learnt from systems mapping https://www.thinknpc.org/blog/what-weve-learnt-from-systems-mapping/
Mowles, C. (2014). Complex, but not quite complex enough: The turn to the complexity sciences in evaluation scholarship. Evaluation, 20(2), 160-175.
Peters, D. H. (2014). The application of systems thinking in health: why use systems thinking?. Health research policy and systems, 12(1), 1-6. https://doi.org/10.1186/1478-4505-12-51
Pluchinotta, I., Esposito, D., & Camarda, D. (2019). Fuzzy cognitive mapping to support multi-agent decisions in development of urban policymaking. Sustainable Cities and Society, 46, 101402. https://doi.org/10.1016/j.scs.2018.12.030
Renger, R., Atkinson, L., Renger, J., Renger, J., & Hart, G. (2019). The connection between logic models and systems thinking concepts. Evaluation Journal of Australasia, 19(2), 79-87. https://doi.org/10.1177%2F1035719X19853660
Rosas, S., & Knight, E. (2019). Evaluating a complex health promotion intervention: case application of three systems methods. Critical Public Health, 29(3), 337-352. https://doi.org/10.1080/09581596.2018.1455966
Rutter, H., Savona, N., Glonti, K., Bibby, J., Cummins, S., Finegood, D. T., ... & White, M. (2017). The need for a complex systems model of evidence for public health. The lancet, 390(10112), 2602-2604. https://doi.org/10.1016/S0140-6736(17)31267-9
Stave, K., & Hopper, M. (2007, July). What constitutes systems thinking? A proposed taxonomy. In 25th international conference of the system dynamics Society. https://proceedings.systemdynamics.org/2007/proceed/papers/STAVE210.pdf
Stave, K. (2010). Participatory system dynamics modeling for sustainable environmental management: Observations from four cases. Sustainability, 2(9), 2762-2784. https://doi.org/10.3390/su2092762
Stansfield, J., Cavill, N., Marshall, L., Robson, C., & Rutter, H. (2021). Using complex systems mapping to build a strategic public health response to mental health in England. Journal of Public Mental Health. Vol. 20 No. 4, pp. 286-297. https://doi.org/10.1108/JPMH-10-2020-0140
Tourais, P., & Videira, N. (2021). A participatory systems mapping approach for sustainability transitions: Insights from an experience in the tourism sector in Portugal. Environmental Innovation and Societal Transitions, 38, 153-168. https://doi.org/10.1016/j.eist.2021.01.002
Voinov, A., Jenni, K., Gray, S., Kolagani, N., Glynn, P. D., Bommel, P., ... & Smajgl, A. (2018). Tools and methods in participatory modeling: Selecting the right tool for the job. Environmental Modelling & Software, 109, 232-255. https://doi.org/10.1016/j.envsoft.2018.08.028
Wilkinson, H., Hills, D., Penn, A., & Barbrook-Johnson, P. (2021). Building a system-based theory of change using participatory systems mapping. Evaluation, 27(1), 80-101. https://doi.org/10.1177%2F1356389020980493
Systems Thinking: Physical Activity & Public Health
Barnes, A., Black, M., Baxter, S., Furber, A., Strong, M., Beynon, C., ... & Clowes, M. (2019). Understanding public health systems: a participatory systematic review and systems infographic. The Lancet, 394, S22. https://doi.org/10.1016/S0140-6736(19)32819-3
Bird, E.L., Evans, D., Gray, S. et al. (2022) Opportunities to engage health system leaders in whole systems approaches to physical activity in England. BMC Public Health 22, 254. https://doi.org/10.1186/s12889-022-12602-5
Bellew, W., Smith, B. J., Nau, T., Lee, K., Reece, L., & Bauman, A. (2020). Whole of Systems approaches to physical activity policy and practice in Australia: the ASAPa Project overview and initial systems map. Journal of Physical Activity and Health, 17(1), 68-73. https://doi.org/10.1123/jpah.2019-0121
CLOA/SportEngland (2021) Navigating Local Systems Test and Learn approach to system change https://cloa.org.uk/wp-content/uploads/2021/03/Final-Navigating-Local-Systems.pdf
Hall, J., Bingham, D. D., Seims, A., Dogra, S. A., Burkhardt, J., Nobles, J., ... & Daly-Smith, A. (2021). A whole system approach to increasing children’s physical activity in a multi-ethnic UK city: a process evaluation protocol. BMC public health, 21(1), 1-16. https://doi.org/10.1186/s12889-021-12255-w
Rutter, H., Cavill, N., Bauman, A., & Bull, F. (2019). Systems approaches to global and national physical activity plans. Bulletin of the World Health Organization, 97(2), 162. https://dx.doi.org/10.2471%2FBLT.18.220533
Pogrmilovic, B. K., O’Sullivan, G., Milton, K., Biddle, S. J., Bauman, A., Bellew, W., ... & Pedisic, Z. (2019). The development of the Comprehensive Analysis of Policy on Physical Activity (CAPPA) framework. International Journal of Behavioral Nutrition and Physical Activity, 16(1), 1-15. https://doi.org/10.1186/s12966-019-0822-5
Rigby, B.P., Dodd-Reynolds, C.J. & Oliver, E.J. (2022). The understanding, application and influence of complexity in national physical activity policy-making. Health Res Policy Sys 20, 59 https://doi.org/10.1186/s12961-022-00864-9
Systems Thinking: Food and Obesity
Allender, S., Owen, B., Kuhlberg, J., Lowe, J., Nagorcka-Smith, P., Whelan, J., & Bell, C. (2015). A community based systems diagram of obesity causes. PloS one, 10(7), e0129683. https://doi.org/10.1371/journal.pone.0129683
Maitland, N., Wardle, K., Whelan, J., Jalaludin, B., Creighton, D., Johnstone, M., ... & Allender, S. (2021). Tracking implementation within a community-led whole of system approach to address childhood overweight and obesity in south west Sydney, Australia. BMC public health, 21(1), 1-11. https://doi.org/10.1186/s12889-021-11288-5
Swinburn, B. A., Kraak, V. I., Allender, S., Atkins, V. J., Baker, P. I., Bogard, J. R., ... & Dietz, W. H. (2019). The global syndemic of obesity, undernutrition, and climate change: the Lancet Commission report. The lancet, 393(10173), 791-846. https://doi.org/10.1016/S0140-6736(18)32822-8
Waterlander, W. E., Luna Pinzon, A., Verhoeff, A., Den Hertog, K., Altenburg, T., Dijkstra, C., ... & Stronks, K. (2020). A system dynamics and participatory action research approach to promote healthy living and a healthy weight among 10–14-year-old adolescents in Amsterdam: The LIKE programme. International journal of environmental research and public health, 17(14), 4928. https://doi.org/10.3390/ijerph17144928
The Butterfly Effect: Everything You Need to Know About This Powerful Mental Model - https://fs.blog/the-butterfly-effect/
OMIDYAR Group (2017) Systems Practice Workbook https://oecd-opsi.org/toolkits/systems-practice-workbook/
Penn, AS, Barbrook-Johnson, P (2019) Participatory Systems Mapping: A practical guide. CECAN toolkit. Available at: https://www.cecan.ac.uk/resources/toolkits/
CECAN (2021) The Complexity Evaluation Toolkit. July 2021. Version 1.0. Online at www.cecan.ac.uk
Egan M, McGill E, Penney T, Anderson de Cuevas R, Er V, Orton L, White M, Lock K, Cummins S, Savona N, Whitehead M, Popay J, Smith R, Meier P, De Vocht F, Marks D, Andreeva M, Rutter H, Petticrew M. (2019) NIHR SPHR Guidance on Systems Approaches to Local Public Health Evaluation Part 1: Introducing systems thinking https://sphr.nihr.ac.uk/wp-content/uploads/2018/08/NIHR-SPHR-SYSTEM-GUIDANCE-PART-1-FINAL_SBnavy.pdf
Egan M, McGill E, Penney T, Anderson de Cuevas R, Er V, Orton L, White M, Lock K, Cummins S, Savona N, Whitehead M, Popay J, Smith R, Meier P, De Vocht F, Marks D, Andreeva M, Rutter H, Petticrew M. (2019) NIHR SPHR Guidance on Systems Approaches to Local Public Health Evaluation. Part 2: What to consider when planning a systems evaluation. London: National Institute for Health Research School for Public Health Research https://sphr.nihr.ac.uk/wp-content/uploads/2018/08/NIHR-SPHR-SYSTEM-GUIDANCE-PART-2-v2-FINALSBnavy.pdf
Kania, J., Kramer, M. and Senge, P (2018) The Water of Systems Change https://www.fsg.org/resource/water_of_systems_change/
NPC (2018) Thinking big: How to use theory of change for systems change https://www.thinknpc.org/resource-hub/thinking-big-how-to-use-theory-of-change-for-systems-change/
Systems Thinking Toolkit: Putting Systems Thinking into Practice in Your Organization https://www.fsg.org/resource/systems-thinking-toolkit-0/
Preskill, H., Gopal, S. Mack, K. and Cook, J. (nd) Evaluating Complexity: Propositions for Improving Practice https://www.fsg.org/resource/evaluating-complexity/
Tamarack Institute. TOOL: The Systems Change Evaluation Canvas https://www.tamarackcommunity.ca/library/systems-change-evaluation-canvas
TASO Run your own theory of change workshop https://taso.org.uk/evidence/resources-hub/toc/
WHO (Sept, 2022). Systems thinking for noncommunicable disease prevention policy: guidance to bring systems approaches into practice. https://www.who.int/europe/publications/i/item/WHO-EURO-2022-4195-43954-61946
Software & Mapping Tools
PRSM - https://cress.soc.surrey.ac.uk/prsm/ [see Barbrook-Johnson, P., & Penn, A. (2021)]
https://www.diagrams.net/ - diagrams.net: easy to use and edit but doesn't allow automated analysis.
Kumu - https://kumu.io/
Miro - https://miro.com/
Vensim - https://vensim.com/
STICKE from Deakin Uni - https://sticke2.deakin.edu.au/#about
SimpleMind - https://simplemind.eu/
NetDraw (Borgatti, 2002)
Gephi (Bastian et al., 2009)
Free Online Learning Resources
Open University. Systems thinking. https://www.open.edu/openlearn/science-maths-technology/engineering-technology/systems-thinking-free-courses
John Hopkins University (Coursera). Systems thinking in Public Health. https://www.coursera.org/learn/systems-thinking
Systems Thinking: Books & Chapters
Anderson, V. and Johnson, L. (2015) Systems Thinking Basics: From Concepts to Causal Loops (Pegasus Workbook Series). Pegasus Communications.
Boulton, J. G., Allen, P. M., & Bowman, C. (2015). Embracing complexity: Strategic perspectives for an age of turbulence. OUP Oxford.
Byrne, D., & Callaghan, G. (2013). Complexity theory and the social sciences: The state of the art. Routledge.
Castellani, B., Rajaram, R., Buckwalter, J. G., Ball, M., & Hafferty, F. (2015). Place and health as complex systems: A case study and empirical test. Springer.
Finegood, D., Johnston, L., Steinberg, M., Matteson, C. L., & Deck, P. (2014). Complexity, systems thinking, and health behavior change. In Health Behavior Change in Populations; Kahan, S., Gielen, AC, Fagan, PJ, Lawrence, WG, Eds, 208-236.
Holland, J. H. (2000). Emergence: From chaos to order. OUP Oxford.
Ison, R. (2010). Systems practice: how to act in a climate change world. Springer. [sample chapter]
Meadows, D. H. (2008). Thinking in systems: A primer. Chelsea Green Publishing.
Barbrook-Johnson, P., & Penn, A. S. (2022). Systems Mapping: How to build and use causal models of systems. [FREE Open Access Book]
Stroh, D. P. (2015). Systems thinking for social change: A practical guide to solving complex problems, avoiding unintended consequences, and achieving lasting results. Chelsea Green Publishing.
West, G. (2017). Scale: The Universal Laws of Life, Growth, and Death in Organisms. Cities, and Companies. Penguin Publishing Group.