...continue

I would recommend reserving such efforts for only the most “mature” programs, where data of this sort might be critical to assessing their potential for widespread replication. As an example, we cofunded an ambitious random assignment field trial in San Diego of a previously tested model of home visiting for families with newborns. One of our partners was the California State Department of Social Services, which was considering this model for significant statewide funding. It made sense to take the time and expense to underwrite a sophisticated experimental design in that case, since the model, although successful in other settings, had not yet demonstrated its effectiveness with underserved populations in California.

Going back to the reasons for engaging in evaluation for a moment, there’s only one – advancing knowledge¾that demands the “gold standard” type of experimental design described above as the acceptable standard of evidence. Rarely in the worlds of policy and practice are such “textbook” standards decisive¾for better or for worse. Judgments tend to be made on other forms of information, whether they are quantifiable, intermediate measures of success, other forms of documentation or even well-told anecdotes. Does that mean we should abandon rigor for pragmatism? To me, as in other aspects of philanthropic decisionmaking, it’s a matter of institutional values and tradeoffs, as expressed in this schematic:

As an organization, we tend to place a higher value on strategically meeting the health needs of underserved communities than we do on contributing to knowledge. Those endeavors are not mutually exclusive, of course, but their prioritization is instrumental in shaping the design of our grant programs and what we ultimately fund. It also determines the potential audiences for our work. Whom do we hope to influence, and what forms of data will be most persuasive?