Strengthening Reasoning in Evaluation: Logic, Fallacies, and the Data-to-Claim Process
Effective evaluation relies on more than systematic collection and analysis of data and reporting: it requires robust reasoning that connects evidence to claims. And the type of reasoning that is used should be informed by both the logic of evaluation, and the logic of the approach, as well as awareness of our personal logic: our mental models and worldview as evaluators.
For my 2025 AES exploratory workshop, Logical Leap or Logical Lapse, I tried to identify a process evaluators can follow to:
- be more explicit about how we go from having data to having a finding, and from there to a recommendation.
- how we can more explicitly step out our reasoning for ourselves as evaluators, to help to strengthen our evaluative practice and accuracy of our findings, both solo and as a team.
This piece is based on that presentation. It first presents some basics in defining logic, logical arguments and the criteria for a good argument, then discusses the layers of logic operating within evaluations, identifies where logical errors can be made, and finally provides a 3 step process for strengthening the data to claim-making process.
Foundations: logic and good arguments
A quick 101 on a few items, before digging into the layered logics of evaluation.
Logic is the study of correct reasoning: specifically, how inferences are drawn from premises to reach conclusions. Or to put it another way… it’s a set of rules and techniques to help us understand whether reasoning produces a valid argument.
Michael Cole (2023) offers a succinct definition:
“Logic and logical inference are foundational to critical thinking. Logical reasoning involves working step by step from a premise(s) to arrive at a valid solution or conclusion, and each step in the logical argument must be correct to reach a valid solution or conclusion. With logical reasoning, two different people would arrive at the same conclusion if given exactly the same information.”[1]
A logical argument is constructed from three main components:
- Premises: The reasons or evidence supporting a claim.
- Conclusion/Claim: The statement being supported.
- Warrant: The principle, assumption, or rule (either explicit or implicit) that connects the premises to the conclusion justifying the inferential leap to get to the conclusion.[2]
In evaluation, reasoning is often inductive or abductive. Abductive reasoning seeks the most plausible explanation based on the available data (“inference to the best explanation”). Inductive reasoning generalises from past observations to future expectations, supporting generalisations and forecasts.[3]
These concepts, but warrants in particular can feel a little slippery to get ahold of without concrete examples, so here are a few:
Example 1
The program should be discontinued [claim] because it has not met its target number of participants [premise], it exceeded its budget [premise], and there is insufficient evidence that the program achieved its intended behavioural changes for participants [premise].
Unstated warrant: We should not invest in programs that can’t achieve what they set out to.
In this example, we’ve got 3 premises (or the 3 reasons) supporting the conclusion. The unspoken part of our argument is the warrant, which supports our inferential leap from those 3 things that are not great about our program to the conclusion that our program should not continue.
The warrant is that we should not invest in programs that can’t achieve what they set out to. That’s something that’s so self-evident in the way we’ve set up our societies that it has become an unspoken assumption.
Example 2
Grant applicants held concerns about how assessments of applications are being made, who is involved in assessing them, and whether they have the appropriate expertise [premises]. The grant assessment process should be documented and made available to applicants in order to improve probity and transparency [claim], in accordance with the principles of the Grants Administration Guide [warrant].
Explicit warrant: accepted principles of how good grant programs should be administered; and based in the authority of the NSW Government.
There’s one more piece of the argumentation puzzle, which is what, beyond containing the key components of an argument, makes for a good argument?
There are 3 characteristics of a good argument:
- It offers reasons for the conclusions.
- The premises are acceptable and the warrant for each conclusion holds.
- It contains enough relevant information required for the audience to find the conclusion acceptable.[4]
Layers of Logic in Evaluation
There’s a lot of logic at work in evaluation, operating at several different layers. Fournier’s work distinguishes between general logic and working logic in evaluation, to which I would also add ‘practitioner logic’.
General Logic: guides practice, specifying “the basic conditions under which rationally motivated argumentation can take place….it specifies to practitioners how someone would reason to justify claims.”[5]. Its steps are well described by Scriven (1980) and more recently Gullickson (2020) (see below).
The expanded logic of evaluation (Gullickson)
- Clarify evaluation purpose and assess evaluability
- Define the evaluand
- Define the group to which the evaluand belongs
- Identify criteria/delineate evaluation questions
- Identify performance standards
- Justify the criteria and standards
- Measure; observe evaluand’s performance
- Justify the measures
- Synthesise an evaluative judgement
- Justify the synthesis method
- Report judgement[6]
Working Logic: Below the general logic, is the ‘working logic’ specific to the evaluation approach. Each approach has its own warrants to support inferential leaps made to get to evaluation findings and judgements. (see below)
Practitioner Logic: The individual evaluator’s reasoning approach informed by their experiences, worldview, values, unexamined assumptions and potential biases (including how they interpret the working logic of the approach they have chosen). We bring our own mental models to our evaluation practice – these are the frameworks, or mental shortcuts that we use to interpret information, informed by and often intertwined with our values, our socio-cultural influences, and even the language we speak. Mental models can be very helpful in many circumstances…but they can also stop us from ‘seeing’ our assumptions, or from integrating evidence that contradicts them into our reality.
These operate together to produce the reasoning by which an evaluation is constructed, conducted and presented. The strength of the first two layers of logic – the general logic of evaluation and the working logic of the approach – are to some extent reliant on the practitioner logic layer. It is largely at this level that errors in logic can occur, in program design, evaluation design, analysis and synthesis.
The purpose of this piece is for us as evaluators to become more aware of this, and to present one method by which we can consciously probe for weaknesses in logic – especially through the process of moving from analysing data to making claims.
Logical Fallacies: Risks in the Data-to-Claim Process
Logical fallacies are systematic mistakes made within arguments – issues with the premises being untrue or not relevant to the conclusion, the inferences being erroneous or having a basis in ambiguity, or with premises, inferences or conclusions resting on unexamined and erroneous assumptions.
Logical fallacies are sometimes used intentionally and persuasively, but frequently they are made unintentionally.
You’ll find a list of those more commonly found in evaluation, and examples in the Evaluator’s Logical Reasoning Buddy. There are more at YourLogicalFallacyIs.com
Strengthening the Data-to-Claim Process
Drawing on these key concepts in logic and argumentation, I’ve outlined a 3 step process evaluators can use to strengthen the process of moving from having data to making claims, including by testing for weaknesses.
Step 1 – Check for the 3 characteristics of good reasoning in the findings and recommendations
Step 2 – Make explicit the connections between the data/program theory/literature, the claim and the warrant
Step 3 – Check for logical fallacies, and if one is identified, check:
- if you have identified a fallacy or are making the fallacy fallacy? (the assumption that because the argument contains a fallacy or is lacking some of the elements of a good argument, the argument itself is not just poorly made, but wrong)
- Identify what errors have been made to lead to the fallacy (in the premises, or conclusion) and correct them, or, strengthen the argument through steps 1 and 2.
The Evaluator’s Logical Reasoning Buddy is a tool to use as a solo evaluator, or to facilitate discussions with teams, for just this purpose.
One more practical tool…
Identifying logical fallacies isn’t easy at first, but it gets much easier with practice.
Take a look at this scenario, to see how many logical fallacies you can identify, and then check them against the answers sheet.
Further reading & listening
The foundations of my thinking on this topic are informed by
- Tyson Yunkaporta, who applied logical fallacy coding in his 2024 article “The Ancient Wisdom Grift: Mad Studies and Indigenous Methods applied to the problem of spiritual disinformation narratives” (International Mad Studies Journal) and tested his thinking with a collective of his community members.
- Andrew Hawkins – whose conversations and writings have informed my understanding of what evaluation is and what it can and can’t do. Especially: Hawkins, A., Bayley, S. 2024. Managing the risk of program failure: Propositional Evaluation as a tool for risk management
- Deborah Fournier. Establishing Evaluative Conclusions: A Distinction between General and Working Logic. New Directions for Evaluation, 68
- Amy Gullickson. 2020. The Whole Elephant – Defining Evaluation. Evaluation and Program Planning (79)
- Stephen Toulmin. 2003. The Uses of Argument. Cambridge University Press
- David McRaney’s You Are Not So Smart Podcast
- Jesse Richardson’s Your Logical Fallacy Is website.
[1] Cole, M (2023) ‘Evaluative Thinking’, Evaluation Journal of Australia, 23 (2), 70-90
[2] Toulmin, S (2003) The Uses of Arguments, Cambridge University Press, 87-100; Smith, N (2020) Evaluating Arguments – Introduction to Philosophy: Logic’, in Martin, B and Hendricks C (eds) Introduction to Philosophy: Logic, Rebus Press
[3] Smith, N (2020) ‘Evaluating Arguments – Introduction to Philosophy: Logic’, in Martin, B and Hendricks C (eds) Introduction to Philosophy: Logic, Rebus Press
[4]Rodrigues, C. 2020. Informal Fallacies – Introduction to Philosophy: Logic. Rebus Press, in Martin, B and Hendricks C (eds) Introduction to Philosophy: Logic, Rebus Press
[5] Fournier, D. 1995. ‘Establishing Evaluative Conclusions: A distinction between general and working logic’, New Directions in Evaluation, 68
[6] Gullickson, A (2020) ‘The Whole Elephant: Defining Evaluation’, Evaluation and Program Planning, 79
