Mind the Gap, the Evaluation Metro Map
ARTD’s Lunchtime Learning is a weekly time for our staff to come together to share and learn. A recent addition to our Lunchtime Learning sessions is the ARTD Journal Club, where our team reviews academic articles together and discusses what they mean for our practice as evaluators. We’re excited to share these lessons with you through our new blog series, ARTD Journal Club, where we highlight what we’ve learnt and the questions we’re still exploring.
The first session was led by ARTD Partner and Chief Evaluator, Andrew Hawkins, who walked us through The Evaluation Metro Map by Sebastian Lemire, published in 2024. Lemire is an evaluator who, like many of us, thinks visually. Inspired by his young son’s love of train maps, he created a metro map to help make sense of the wide range of evaluation approaches and designs that exist in the field. He builds his map around three main lines:
- The design line shows the broad range of study designs available, from case study to randomised control trial.
- The theory line shows the epistemological underpinning of evaluation choices.
- The use line shows the diversity of evaluation approaches focusing on the utilisation of evaluation, particularly those centred on equity and social justice.
Additionally, Lemire’s work shows us that you don’t have to sit on one track, with hybrid and emerging lines building connections for approaches that blend across the three. It’s a useful tool for understanding how different approaches relate to each other and where your evaluation sits within the landscape.
The Evaluation Metro Map (Lemire, 2023)
Three things we found useful
- The map helped us explore complexity and makes explicit the choices we make as evaluators. Some of our team reflected that it was useful to see that we do not have to be limited by a single approach or design; rather, you can pick and blend across lines. As we know from our work, real evaluations often draw on multiple approaches and design choices, and the map could support us to navigate the choices we make.
- It sparked a useful conversation amongst the team about causal claims. If you want to say that a specific program caused a specific outcome, you need to be using designs far along the design line, in quasi-experimental or experimental territory. As Andrew always reminds our team, overclaiming causation without the evidence to back it up is one of the biggest mistakes in evaluation.
- Our team also highlighted the importance of the use line. Over time, it has grown to include culturally responsive approaches, reflecting a shift from measuring whether a bureaucratic goal was met to embedding questions like ‘whose values count?’ and ‘who gets a voice in this?’, which are now central in many evaluations.
Three questions our team are still pondering
- With our team working across a range of different designs, we were quick to note that the map’s design line leans heavily toward quantitative and experimental methods, which is unsurprising given Lemire’s specific area of work but a major gap all the same. If, your practice is qualitative, participatory, or community-led, you may find a lack of stops along the design line that suit your evaluation.
- We also reflected that the map is not new. However, unlike earlier evaluation theory trees that organised approaches around the academics who created them, which is a pretty narrow demographic, through the use line, the map leans into tying the field’s knowledge to specific ideas or communities whom those approaches are meant to serve.
- The map is a great conversation starter, but its limitation lies in answering the most important question: What’s the right approach for this evaluation, right now, given what the client needs and what’s available? That judgement, finding the fit-for-purpose sweet spot, still belongs to the evaluator. (You can read our other blogs on fit-for-purpose evaluation to help you frame the answer to that question – Fit for Purpose Evaluation #1 The Big Questions, Fit for Purpose Evaluation #2 The Lens Determines what you can see, Fit for Purpose #3 Propositional Evaluation as a Structure for a Rigorous Developmental Approach).
Overall, our team valued exploring the Metro Map is a useful heuristic tool. Its value lies in the conversations it starts, and if our session was anything to go by, those conversations don’t stop when you reach the end of the line.
Read the full article here: The Evaluation Metro Map – Sebastian Lemire (2024)
