What is a realistic “standard of evidence”? In response to this question, a methodologist might begin by sketching out a simple continuum of data sources. At one end, one might put uncorroborated self-reports from grantees, coupled with semi-random Program Officer observations. At the other end, one would find true experimental designs, featuring double-blind random assignment of “subjects” to “treatments.” In between those two poles would fall an array of intermediate alternatives, ranging from qualitative observations by evaluation professionals from outside the organization to quasi-experimental time-series designs utilizing comparison groups. 

When I was studying evaluation as a graduate student, there was no question in my instructor’s mind as to which method was most desirable. It was experimental rigor or nothing! But the field of evaluation has evolved over the past 25 years, and now even the experts disagree on the answer to this question. The key word here is “realistic.” It depends on what outcomes you are hoping to achieve.

Some health outcomes are relatively easy to measure in a fairly short time frame¾for example, whether a prenatal outreach program has been able to reduce the number of women delivering babies with inadequate prenatal care in a particular community. But long-term changes in individual behavior, or some of the community-wide outcomes to which we aspire as an organization, can be virtually impossible to track with precision, let alone during the limited life of a funding initiative. 

Moreover, many of the most methodologically ambitious attempts to evaluate long-term program impact have yielded disappointing results, feeding the perception in some quarters that “nothing works.” Yet if we step back a bit from our work, it stands to reason that it’s rather unrealistic to expect time-limited programs to engender long-term change, particularly in communities with few other support systems in place. That is why we and others have invested in longer term, multifaceted funding initiatives. But it only makes the challenge of evaluation that much more complicated.

Even with a relatively sophisticated evaluation design in place, there remains the challenge of attribution. How do we know that the results observed are due to the program we’ve funded? As we’ve noted before, most of our grant programs are being implemented in “high noise” settings, where multiple interventions are simultaneously taking place. Even if we were able to employ methodologies such as random assignment and control groups, there’s no guarantee that we would be able to unequivocally attribute observed outcomes to our funding.