RESOURCES FOR RESEARCHERS AND PROGRAM EVALUATORS

Six Steps to Program Evaluation

There are six basic steps to program evaluation, all of which are related. The first steps provide a foundation for the later ones. The steps (Box 3) are: engage stakeholders, describe the program, focus the evaluation design, gather credible evidence, justify conclusions, and ensure use and share lessons learned.

Engage stakeholders

Most suicide prevention efforts involve partners, all of whom are stakeholders in the effort. Stakeholders must be involved in the evaluation; without their involvement, an evaluation might miss important parts of a program. Three groups of stakeholders are important:

• Those involved in program operations (e.g., sponsors, collaborators, coalition partners, funding officials, administrators, managers, and staff)
• Those served or affected by the program (e.g., clients, family members, neighborhood organizations, academic institutions, elected officials, advocacy groups, professional associations, skeptics, opponents, and staff of related or competing organizations)
• Those who will be the primary consumers of information produced by the evaluation
• Stakeholder involvement is different for each program evaluation. The objective of the evaluation will help define which stakeholders to involve. Without agreement from stakeholders, the evaluation may be of limited use.

Describe the program

The program description details the mission and objectives of the program. Descriptions should help evaluators understand the program goals and strategies. The description should discuss how the program works to effect change, describe the program’s stage of development, and show how the program fits into the larger organization and community. Stakeholders should review and agree with the program description. Program descriptions will vary for each evaluation.

Aspects to include in a program description are a statement of need, expected effects, activities, resources, stage of development, context, and a logic model. A statement of need describes the problem that the program addresses. Expected effects are what the program must do to be successful. Program activities are what the program does to effect change. Resources include the time, talent, technology, equipment, information, money, and other assets available to conduct program activities. The program’s stage of development reflects its maturity. The context should describe the setting within which the program operates. Some programs have used a logic model as a planning tool to outline resources and to describe the order of the program activities and associated outcomes. Logic models require a planned, sequenced thought process to design a program.

Focus the evaluation design

The evaluation must focus on issues of greatest concern to stakeholders. A funder may be concerned about efficient use of resources. The designer of an intervention may be concerned about fidelity of delivery. Most evaluations have limited time and resources. A good evaluation design is directly related to knowing its intended uses, that is, whether it is designed to demonstrate effectiveness to policymakers, acquire resources, or some other type of use. Significant items to consider in focusing an evaluation are purpose, users, uses, questions, methods, and agreements.

Describing the purpose of the evaluation (i.e., its intent) will guide how it should be conducted. The stage of a program’s development will define the purpose of the evaluation. For example, a program just starting may focus its evaluation on perceptions of the use of the materials being developed. A more developed program may attempt to conduct an outcome evaluation to see if the program is effective. The context of the program also affects the purpose. Evaluations of prevention programs have four general purposes. The first is to gain insight: We try to learn if doing something makes sense. A second purpose for program evaluation is to change practice. A third purpose is to assess the effects of the program. The fourth purpose is to effect change in the persons participating in the management and delivery of the program. Participating in an evaluation can provide insight for program staff that can be a catalyst for self-directed change.

Users are the persons who will receive evaluation findings, such as the program manager, funders, and other stakeholders. Intended users should participate in choosing the evaluation focus.

Uses are the ways in which information from the evaluation will be applied, as in deciding to continue funding or expand an effort or determining if program procedures should be modified for better delivery. Stakeholders should help identify, plan, and prioritize uses in advance of implementing the evaluation process. The program’s stage of development and current context should also be considered when identifying uses.

Evaluation questions clarify what aspects of the program will be addressed. Creating evaluation questions identifies what the evaluation should answer. The questions will suggest how and what information should be gathered to enable adequate answers.

The methods for an evaluation refer to scientific research options. Methods for suicide prevention evaluations are generally developed in the social, behavioral, and health sciences and include experimental, quasi-experimental, and observational designs. Choosing the best design depends on the circumstances. The method should enable the evaluation to address stakeholders’ questions. Experimental designs randomly assign persons to intervention and non-intervention groups. By using random assignment, the two groups are usually very similar, with the exception that one receives the intervention and the other receives something else. When the intervention is complete, the differences we see in the two groups in the outcome of interest is likely caused by the difference made by the intervention. Quasi-experimental methods are similar to experimental, except that the groups are not similar because they are not randomly assigned. This type of method is used when randomization is not feasible. Observational methods tend to be descriptive and attempt to understand differences, similarities, and processes within a group. Monitoring a sample of telephone calls on a crisis telephone line and describing how well procedures are being followed is an example of an application of an observational method.

The choice of design has implications for what will be used as evidence. The design will determine how evidence will be gathered and what kind of claims can be made. The design also determines how data sources will be selected, what data collection instruments will be used, who will collect the data, and what data management systems will be needed.

Each evaluation method has its own limitations. Evaluations that mix methods are generally more effective. Methods might need to be revised or modified over the course of an evaluation: The intended use of an evaluation might shift, or changing conditions might require redesign of methods to keep the evaluation on track.

Agreements are explicit written statements that summarize the procedures and clarify roles and responsibilities among those who will execute the evaluation plan. Written agreements ensure understanding among stakeholders and evaluators in terms of their expectations and help to avoid numerous problems that can develop from misunderstandings.

To focus an evaluation design, stakeholders should review the evaluation questions and determine if they will address concerns of interest groups. A list of evaluation uses could be circulated among stakeholders to determine which is most important. Intended users could also be interviewed to understand what they need or want to know and to focus the evaluation so it responds to those needs.

Gather credible evidence

The information collected from a suicide prevention evaluation should provide a useful understanding of the program that addresses stakeholders’ needs. Stakeholders must find the information believable and relevant. Having credible evidence strengthens the recommendations. Credibility can be improved by using multiple procedures and by involving stakeholders in defining and gathering data. The following factors affect people’s perceptions of the credibility of your evaluation evidence: indicators, sources, quality, quantity, and logistics.

• Indicators are aspects of the program that can be examined to address the questions of the evaluation. Examples of indicators that can be defined and tracked include the program’s capacity to deliver services, the participation rate, levels of client satisfaction, the efficiency of resource use, and the amount of intervention exposure. Other measures of program effects, such as changes in participant behavior, community norms, policies or practices, health status, quality of life, and the settings or environment around the program, can also be tracked.
  
  
• Sources of evidence are persons, documents, or observations. More than one source might be used to gather evidence. Use of multiple sources provides different perspectives. In an evaluation where the question is, "Does a training of trainers result in a desired number of new gatekeepers being trained?", the indicator might be a "number of persons trained by each trainer over the course of a year."
  
  
• Quality refers to the correctness and integrity of the information. Quality data are representative of what they intend to measure and are informative for their intended use. Good indicators make it easier to collect quality data. Instrument design, data-collection procedures, training of data collectors, source selection, coding, data management, and routine error-checking all influence the quality of your data collectors, source selection, coding, data management, and routine error-checking all influence the quality of your data. For example, a data collection effort where data collectors are trained on procedures and where there are checks to monitor adherence to procedures will produce higher-quality data than one with inconsistent or minimal training of data collectors.
  
  
• Quantity refers to the amount of evidence gathered. The amount of information needed should be estimated in advance. All evidence collected should have a clear and anticipated use, with only minimal burden placed on respondents. Information gathered from participants in an intervention should be kept to a minimum. Each indicator should be clearly justified in terms of addressing an evaluation question. Another example of quantity would involve determining how many persons must provide information to adequately address the evaluation question. The burden on persons to provide information should always approach the minimum needed.
  
  
• Logistics encompass the methods, timing, and physical infrastructure for gathering and handling evidence. Each technique selected for gathering evidence must be suited to the source(s), analysis plan, and strategy for communicating findings. Cultural issues should influence decisions about acceptable ways of asking questions and collecting information. Procedures for gathering evidence should be sensitive to cultural conditions in each setting and must ensure that the privacy and confidentiality of the information and sources are protected.

Justify conclusions

Evaluation conclusions must be drawn from the evidence gathered and then compared to the standards set by the stakeholders. Stakeholders must agree that conclusions are valid: otherwise, the evaluation results will be of limited use. Justifying conclusions on the basis of evidence requires the following: standards, analysis and synthesis, interpretation, judgment, and recommendations.

• Standards reflect what stakeholders think is important and are the basis for forming judgments concerning program performance. Using standards distinguishes evaluation from other approaches, in which priorities are set without reference to any sort of specific statement about what is important.
• Analysis and synthesis of an evaluation’s findings might detect patterns in evidence. Analysis involves isolating important findings, while synthesis involves combining sources of information to reach a larger understanding.
• Interpretation is the effort of figuring out what the findings mean. It increases understanding of the evidence gathered in an evaluation. Merely uncovering facts regarding a program’s performance is not sufficient to draw evaluative conclusions; one must interpret the results based on criteria set before the evaluation begins.
• Judgments are statements concerning the merit, worth, or significance of the program. They are formed by comparing the findings of and interpretations regarding the program against one or more selected standards. Because multiple standards can be applied to a given program, stakeholders might reach different or even conflicting judgments. For example, a program that increases referrals to a mental health clinician by 10 percent from the previous year might be judged positively by program managers who are using the standard of improved performance over time. However, community members might feel that, despite improvements, a minimum threshold of access to services has not been reached.
• Recommendations are actions for consideration resulting from the evaluation. Recommendations that lack sufficient evidence or that deviate from stakeholders’ values can undermine an evaluation’s credibility. Sharing early drafts of recommendations and asking for feedback from multiple stakeholders during the process can increase the likelihood that the recommendations will be accepted. Additionally, when possible, presenting options instead of directive advice will make recommendations more acceptable.

Adequate justification of the conclusions and recommendations of an evaluation is an essential part of creating information that is useful. However, having useful information does not ensure that it is actually used.

Ensure use and share lessons learned

Unfortunately, despite the best intentions of evaluators and practitioners, lessons learned during an evaluation are not always used. For findings to be used effectively, they must be disseminated appropriately. This requires strategic thinking and should begin in the earliest stages of planning an evaluation and engaging your stakeholders. The goal of dissemination is to achieve full disclosure and impartial reporting. Additional uses for evaluation flow from the process of conducting it. Persons who participate in an evaluation can experience changes in thinking and behavior. Evaluation sometimes increases staff understanding of program goals.

Five critical elements for ensuring that an evaluation is used are as follows: design, preparation, feedback, follow-up, and dissemination.

• Design refers to the construction of evaluation questions, methods, and overall processes. The design should be organized to achieve intended uses by stakeholders.
• Preparation refers to sound evaluation planning and following the steps described in this booklet to ensure effective use of evaluation findings.
• Feedback is the communication that occurs among all parties to the evaluation. Giving and receiving feedback creates an atmosphere of trust among stakeholders.
• Follow-up refers to the technical and emotional support that users need both during the evaluation and after they receive evaluation findings.
• Dissemination is the process of communicating to others the lessons learned. The reporting strategy should be discussed with intended users and other stakeholders.

Sharing information learned from implementing and evaluating suicide prevention programs is one of the most important responsibilities we have in our effort to prevent suicide. Without sharing information, we are isolated in our efforts, and adoption of innovative and promising practices is limited. Effective diffusion of prevention information can have revolutionary effects on the practice of prevention efforts.

The most important aspect of applying lessons learned from evaluation is ensuring that the program is improved as a result. Recommendations should be specific regarding suggested shifts in program emphasis or in specific procedures. A specific written plan should be developed for implementing accepted recommendations. Subsequent evaluation cycles should examine the implementation of the changes and their impact.

Table of Contents | Previous | Next