Evaluation: Did You do it? Did it Matter?

A Logic Model Expresses the Theory

Organizations that collaborate to address community problems have a double challenge: to evaluate both their collaboration and their intervention—and the interplay between the two. In general, evaluations of collaborations focus on process (Did we do what we said we’d do?) and, to some extent, on immediate outcomes (Is a viable collaboration in place? What changes have occurred in the way organizations in the community view and respond to the issue of youth violence?). Assessing the effectiveness of an intervention focuses more on the long-term outcomes, or “impact” (Did we make a difference?), and requires examining specific behavioral and attitudinal changes in participants. In the case of violence prevention interventions for youth, this examination looks at improvements in identified resiliency factors and reductions in risk factors. Looking at the violence prevention initiative as a whole and the ways the collaboration and intervention interact involves a bit more.

The Substance Abuse and Mental Health Services Administration (SAMHSA) has funded more than 500 community partnerships and collaborations, and, partly through national cross-site evaluations of these programs, has developed frameworks for evaluating multilevel prevention initiatives, with a focus on documenting outcomes. The hypothesis was that a link exists between effective community partnerships and reduced negative behavior. The basis of the conceptual framework for the partnerships was a logic model that articulates program goals, objectives, and activities prior to any service implementation. In the early years grantees learned that:

It takes time to coalesce, often up to 2 or 3 years.
Collaborations are complex, multifaceted open systems, which makes programming and evaluating challenging.
Involving all key players in developing the conceptual framework generates collaborative energy.

The conceptual framework that evolved is a model any community can adopt and adapt to measure progress and outcomes. Each collaboration can further define the elements in the framework to fit their own reality. Its components are causally linked as follows:

Partnership characteristics

Eligibility rules
Number of partners
Governance structure
Organizational structure
Staff size and diversity
Age, ethnic, and racial diversity of partners and population

Partnership capacity

Human resources
Organizational resources
Planning
Internal and external communication
Managerial capability
Institutional knowledge of the “system”

Are Presumed to Lead to...
Communitiy actions and prevention activities:

Incentive activities
Strategic activities
Policy and legislative changes
Outreach activities
Community development

Which in turn produce...

Immediate process and activity outcomes:

Coordination and collaboration not present prior to the partnership
Spin-offs (from the partnership) of new services
Reduction in the duplication of existing services
Promotion of an appropriate, comprehensive mix of multilevel services
Non-competition with existing services
Managerial effectiveness

Long-term outcomes

Changes in risk perception
Increases in community protective/resiliency factors
Changes in community health and socioeconomic conditions

And finally...

Positive impacts occurring in the presence of contextual conditions (e.g., sociopolitical, community health, and community infrastructure conditions).

Source: Center for Mental Health Services, School and Community Action Grantee Meeting, December 4, 2000. Presentation Panel: Evaluating Collaborations and Program Implementation, Shakeh Jackie Kaftarian, Ph.D., Acting Director, Office of Knowledge Synthesis, Center for Substance Abuse Prevention, SAMHSA.

Each item of the logic model above can be evaluated. Evaluators simply need to pose questions item by item and gather data to answer those questions. Guidelines are available online and in print to help communities conduct useful evaluations.

Goals are broad, idealized statements that describe the desired measurable outcomes you want to accomplish in your target population. They are the major changes you must bring about to achieve the desired end—that is, reductions in (or prevention of) youth violence in your community. Your objectives will reflect the actions (the processes) you intend to take and the anticipated and progressive changes (the outcomes) you expect to see. Each objective should identify only one result and must be attainable, observable, measurable, and time defined.

You will have goals and objectives for your collaboration, and goals and objectives for your intervention. Moreover, you will have developed these in the process of developing your strategic plan. (See Getting to Outcomes in ACTION Pamphlet 1.) As the logic model above depicts, successful implementation of your intervention depends to a substantial degree on the success of your collaboration; thus, failure to have clear collaboration goals and objectives can result in a poorly conceptualized and implemented intervention.

Previous | TOC | Next

Back to Community Action Pamphlet Menu