Evaluation: Did You do it? Did it Matter?

Assessing Your Intervention

The bottom line question when assessing your intervention is “Have we made a difference?” Generally, you will compare (in the case of youth violence prevention interventions) changes in risk and protective factors in your target population. More specifically, you would want to compare changes in your population with changes that may or may not have occurred in a similar group of youth who were not involved in the intervention.

Evaluation Designs

The four most frequently used evaluation designs are:

Pretest–posttest designs—testing participants before and after the intervention.
Experimental designs—randomly assigning members of your target population into a service or a control group. Both groups would complete a pretest; the service group would receive the intervention; then both groups would complete a posttest. Comparisons between the changes experienced by both groups would indicate the extent to which the intervention was effective.
Quasi-experimental designs—dividing a group into three subgroups: Intervention A (the science-based program), Intervention B (a modified version, less intensive, with fewer components), and Intervention C (no intervention or a limited program). At the end, all would be posttested to compare, for example, their attitudes about violence. A pretest on each group would lend additional credence to the results.
Time series designs—testing participants at several points in time can be done if control or comparison groups cannot be arranged. Some time series designs involve pretesting two or three times before an intervention and posttesting several times after the intervention is over. A variation of that design is to pretest participants, enroll them in the intervention, and then conduct follow-up tests at pre-specified points during the intervention, again at the end of the intervention, and at least 1 and preferably 2 months (or even years) after the intervention.

An Example

With a clearly formed goal in mind and reasonable objectives, you will avoid promising the moon or, alternatively, expecting so little that when your program succeeds—as measured against your minimalist objectives—you really cannot say as much as you may wish in your defense. A rational way to set objectives is to use your experience or the results of other implementations of the same science-based program as a guide.

Goal: Reduce expulsions resulting from fights in middle schools.

By 2003, offer a 25-lesson program in sixth grade classes to help students develop social skills and learn nonaggressive responses appropriate for dealing with conflict.
Who: Prevention specialists
What: 1-hour sessions offered twice a week for 1 school year on topics such as self-understanding, conflict resolution, anger control, and prosocial actions
How much: All sixth grade classes
When: By 2003
Where: County schools

By 2004, implement a schoolwide program to mediate behavior problems and disputes among adolescents.
Who: Teachers and peer mediators
What: Weekly mediation clinics
How much: All sixth, seventh, and eigth grade students
When: By 2004
Where: County schools

The following example (drawn from a manual on best practices in violence prevention) delineates the links between a collaboration’s intervention goals and its anticipated outcomes.

(Outcome) Objectives:

By 2005, reduce the number of fights among eighth grade students from five per month (at baseline) to two per month (after the intervention).
Who: Middle school students
What: Incidents of physical aggression
How much: Reduce by 60 percent (from a 2003 baseline)
Where: County schools

By 2007, reduce by half (from pre- to postintervention) the number of middle school students (grades 6 through 8) expelled because of fights or other disruptive incidents in the schools.
Who: Middle school students
What: Expulsions related to fights in schools
How much: Reduce from an average of two per month to one per month
Where: County schools

Source: T. N. Thornton, C. A. Craft, L. L. Dahlberg, B. S. Lynch, & K. Baer. (2000). Best Practices of Youth Violence Prevention: A Sourcebook for Community Action (pp. 1–13). Atlanta, GA: Centers for Disease Control and Prevention, National Center for Injury Prevention and Control.

For the example, you might gather process and outcome data from the following sources:

Process: Implementation of the 25-lesson curriculum and the weekly mediation program. Curriculum and related materials, list of dates and locations of the class, enrollment and attendance sheets, staff training and qualifications, observations, and surveys.
Outcome: A simple pre- and post-test design. Participant surveys, records review, trend data, school incident data, and school expulsion data (assuming that the schools maintain data in such a way that you can link incidents of aggression to expulsions and to the students involved in the interventions).

The Importance of Fidelity

The extent to which you adhere to the choosen program model will affect the evaluation you conduct. Implementers are often caught between a desire to maintain program fidelity (the prescribed duration or intensity of services) and a stretched budget, but adhering to the model can save money in the end. A program implemented with high fidelity needs only a process evaluation documenting fidelity to assert the link between the intervention and the outcomes observed. If an implementer chooses to diverge in several ways from the model, any observed changes are less likely to be a result of the intervention, and the implementer must conduct an outcome evaluation (usually expensive and often involving a control or comparison group) to document results.

Program fidelity is even more important as the pressure to adopt evidence-based practices intensifies. In the field, it is difficult to adhere to research conditions. Deviations from the model occur; sometimes they are slight and do not affect the program; at other times, deviations are considerable. But practitioners try to build on the best knowledge they have and also gather enough information on the different programs and strategies to develop the next generation of interventions. Programs are not individual entities; they are learning laboratories whose results can contribute to a larger base of knowledge.

Evaluation In Action

In Tucson, Arizona, a violence prevention initiative based on the concept of resiliency and how to infuse it into the community is conducting an ongoing implementation evaluation to track progress, using forms and tools created with the “guided input” of community members. The Tucson model is based on what schools can do to promote resiliency in six areas: provide opportunities for meaningful participation; increase prosocial bonding; set clear, consistent boundaries; teach life skills; provide caring and support; and set and communicate high expectations. Extensive research has found that all these activities are effective in improving resilience, but this community’s evaluation efforts will add greatly to the knowledge base.

Process evaluation is woven throughout program planning, and teachers complete questionnaires based on the six-point model to assess program implementation. The outcome is improved school climate and a focus on resiliency. Leaders say that the community would not have been involved if packaged instruments had been used to assess the process and outcomes of the endeavor. However, the community was involved and helped develop the “short and sweet” instruments now in use.

Indications of success are that, since 1998, three of the five Tucson school districts have adopted the program at all academic levels; more than 70 educators and School Resource Officers have been trained; the commitment of school principals is strong; interest has been expressed by school districts outside the Tucson area; and one participating school is featured in an NEA video. The subcommittee of the Mayor’s School District Task Force continues to this day with its members “zealously committed to resiliency.”

Previous | TOC | Next

Back to Community Action Pamphlet Menu