What is measured is changed.
Some attribute this statement to Einstein, others to Peter Drucker or Karl Pearson, but whoever said it, it makes intuitive sense. Just by observing something, your presence affects that thing. Say a parent monitors their child’s grades. Or a dieter checks their weight on the scale as they start a diet. Or a scientist measures the size of bacteria growing in a petri dish.
Or an evaluator drops into a project with questions about outcomes.
Because the number on the scale, combined with the desire to lose weight, means action taken to lose weight. The act of measuring one’s weight brings about different eating habits or more exercise (or rebelling against these strictures). In the same way, the attention of a scientist or an evaluator measuring outcomes has its own impact. We are all subject to biases and observer effects.
Which means we are not objective – and when we try to be, we fool ourselves and others.
Wait a minute, what do you mean we’re not objective?
There is no scenario in which judgmental outsiders are neutral to an activity. Each step of an evaluation is fully dependent on human interactions, in which we’re all subjects with viewpoints and belief systems: there’s simply no way to be objective. It is possible, however, to pretend. If I go into an office full of people working on a project, and I’m carrying my clipboard and judging the activity in front of me, I can say I am neutral. I might fool myself and others into thinking I am measuring concrete data on the project’s status and progress. I might convince myself and others that I am measuring everything that matters. But does that mean, if I’m not measuring something, it doesn’t matter?
But who sets the standards by which I am measuring? Whose values set the standards? Most likely, those of the dominant culture. In the case of international cooperation projects, that means the standards and ratings are those of the donors. “The piper plays the tune,” right? This assumes the donors are “right”: more small businesses means success, or greater average income means success, or more trained teachers means success: whatever the measurements are for that particular project.
Most of us who’ve worked in cooperation projects can point to times when this assumption was flat-out wrong. Girls receiving individual scholarships in Africa who were then negatively targeted by their peers and community members. Small businesses still touted in a donor’s online “success stories” years after they’ve gone out of business. Numbers of people trained as a proxy for transforming institutions and social practices. If we evaluate projects solely by imperfect donor measures, we are doing participants and even the donors a disservice.
I recently read a typology of evaluations that makes a lot of sense to me. I’m paraphrasing closely from a 2025 article in Evaluative Practice, by Nicoletta Stame, Sapienza University of Rome[1]:
- Bureaucratic evaluations – responsible to policymakers and mainly concerned with efficiency.
- Autocratic evaluations – responsible to academic committees and concerned with “methodological rigor.”
- Democratic evaluations – responsible to informed citizens and concerned with pluralism and accessibility.
Those bureaucratic evaluations – box-ticking exercises – are based on the donor’s values. They don’t tend to result in much change, but they do provide evidence that an agency is at least playing by its own rules. They give the illusion of objectivity, since the donor’s values are laid out from the beginning of a project, in the form of indicators and theories of change and binding contracts. Evaluators in this situation have little latitude to explore – their work is tightly planned around the donors’ evaluation questions.
I’ve also found myself enmeshed in autocratic evaluations – guided by academic institutions and their method-of-the-month design choices – where the benefits to people and communities in crisis may be worse than minimal. Academics are great at explaining their work with a blend of lay and insider language, and they sound unassailable. (I still laugh about a university researcher who bragged about his “platinum standard” design – apparently the “gold standard” wasn’t a great enough name, he had to go that one step further.) But the findings, conclusions and recommendations that emerge are often disappointing, not least because they are fundamentally disconnected from participating individuals and communities.
Democratic evaluations are fewer and further between, in my experience. And I think the issue is this fetish for objectivity: listening to and respecting the opinions and questions of affected populations is not considered rigorous or neutral. And that is unfortunately commonplace, even though the interventions, and the evaluations that follow, must be relevant precisely to those people to be worth the expense.
Obliterating objectivity
Let’s place objectivity onto a flat surface and have a look at it from all sides. It’s one of those words, like patriotism or motherhood, that we cannot question: we assume a more objective design is a better design, no matter the context, conditions or trade-offs. We do this passively – not recognizing that objectivity, as we use it, is just a way to lock in our own worldview as the smart one, the useful one, the best one. It’s really quite Trumpian and for that reason alone, we need to discard it.
When we say objectivity, rather than being impartial, it actually means the stance of the powerful or dominant group – like the donor or the commissioner of the evaluation. In other words, we may say we are neutral, but we are just making our worldview the default. We are not learning, as we should, by questioning the systems and forces that have resulted in the deeply inequitable, damaging, marginalizing situation the intervention is supposed to address.
An example
Let’s take for example implementing environmental interventions in areas affected by mining or other extractive industries. When the evaluators see little progress, they will likely fault the interventions for not achieving their outcomes. But in fact, the mining company may be doing most of the polluting, driving low wages and displacement, and damaging community roles and relationships that once protected environmental assets. The interventions may be nothing more than a BandAid on a hemorrhage. Evaluating without investigating those aspects might be considered “objective” but only by the dominant group (the funder). The affected communities’ perspectives must be weighed alongside those of whoever is playing the tune.
As another example, and rather an extreme one, look at Global North medical research. For decades, mostly male doctors and professors and researchers have used male test subjects and then “generalized” the results to understand women’s dimensions, metabolism, reactions to treatments, and other characteristics. Women are fully half the population, yet the male-dominated medical establishment convinced themselves that their perspectives represented objectivity. But even today, when these practices are presumed to have vanished, current research by male medical professionals (and I use the term lightly) can still contain assessments of women’s “looks”, ostensibly for “medical” reasons! Also, disparities in treatment resulting from this inequality continue: “women wait longer than men for both a diagnosis and pain relief, and are more likely to be misdiagnosed or discharged during serious medical events.[2]
The belief that we are objective or impartial disguises uncomfortable truths. And it’s not functional for us as evaluators, where we need to understand the world from the perspective of affected populations. Otherwise our results won’t be useful for them – not just for the donor or the mining company or the government. This CGIAR blog post by Monica Biradavolu and Nilgün Uygun of QualAnalytics gives examples of how respecting the knowledge, perspectives and opinions of affected populations improves qualitative research findings and interpretations or analysis. In fact, they argue, understanding the complexity of perspectives, interventions and contexts allows us to co-produce stronger research responses in concert with affected populations.
Nerdy book club, anyone?
And if you’re still wondering how to reconcile what you’ve learned about being objective and impartial with what I’m saying, read this book with me. The book is called Qualitative Literacy: A Guide to Evaluating Ethnographic and Interview Research, by Mario Luis Small and Jessica McCrory Calarco. The link is to Bookshop.org. You can also get it at Amazon for a bit less but I am doing my best to avoid Amazon and it has been easier than expected thanks to digital books on Bookshop.org. Share your insights in comments to the blog and we’ll get a conversation going!
[1] Stame’s article, titled “Democratic principles which should underscore evaluation” in the UK Evaluation Society’s Evaluative Practice journal, Edition 04, 2025, references Barry MacDonald’s work on evaluation in the 1970s and 80s as well as other sources on this topic. Unfortunately the blog post is behind a paywall, but you can look up Stame and MacDonald independent of this article and there seems to be a lot.
[2] Lea Merone et al., 2022; Published by Mary Ann Liebert, Inc. This Open Access article is distributed under the terms of the Creative Commons License [CC-BY] (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
