This Web site is a component of the SAMHSA Health Information Network

This Site Mental Health at HHS All Mental Health

Hurricane Katrina and     Disaster Relief Information About the Program Tips For Talking About Disasters Crisis Counseling Program Guidance SAMHSA DTAC Mental Health Services Locator Related Mental Health Topics Featured Publications In The News Related Links Bioterrorism Links Emergency Mental Health and Traumatic Stress Homepage  CMHS Programs:   Go to... Children's Campaign Community Support Emergency Services Financing HIV/AIDS Homelessness Managing Anxiety Mental Health Statistics Protection & Advocacy Special Programs State Planning Traumatic Events Suicide Prevention Programs Homepage  CMHS Activities:   Go to... 15+ School Violence Stigma Suicide Prevention Surgeon General

printer friendly page e-mail this page bookmark this page shopping cart  current or new account

This Web site is a component of the SAMHSA Health Information Network.

Emergency Mental Health and
Traumatic Stress

Recommended Approaches to Evaluation of Crisis Counseling Grant Projects

Note: This is the seventh in a series of program guidance documents developed to ensure consistency in addressing key program issues in the Crisis Counseling Training and Assistance Program (CCP). The Crisis Counseling Training and Assistance Program is funded by the Federal Emergency Management Agency (FEMA) under the authority of the Robert T. Stafford Disaster Relief and Emergency Assistance Act. On behalf of FEMA, the Center for Mental Health Services (CMHS), Emergency Services and Disaster Relief Branch (ESDRB) provides technical assistance, program guidance and oversight.

Contents

Purpose

What This Guidance Includes

Evaluation Activities Encouraged of All Grantees

How Process Evaluation in the Crisis Counseling Program

Some Common Concerns About Evaluation in Crisis Counseling

Organizing and Implementing a Process Evaluation–Key Questions

Conclusion

Appendix A: Resources

Appendix B: Logic Model

References

Purpose
This Program Guidance is primarily intended for State and local mental health disaster coordinators, program planners, administrators and directors of local crisis counseling projects. It is designed to provide information that will assist grantees of the FEMA/CMHS Crisis Counseling Assistance and Training Program in designing and implementing effective evaluations. Evaluation, as described in this guidance, refers to the systematic collection of information designed to answer important questions about activities, characteristics, and results of a grant project. While formalized program evaluation is currently an optional activity for Crisis Counseling Program grantees, both FEMA and CMHS are urging grantees to devote additional attention to ensuring that projects are carefully developed and monitored and that program decisions are evidence-based. Experienced Crisis Counseling Program managers may recognize that many of the evaluation activities described in this guidance build upon existing program monitoring and reporting requirements. The goal of the guidance is to encourage grantees to collect and analyze data more systematically based on specific evaluation questions and to encourage grantees to utilize evaluation approaches designed to provide useful feedback as the program is being conducted.

When implemented appropriately, evaluation should not interfere with the delivery of services and can improve the quality of services by providing ongoing feedback to staff and managers. Evaluation can help document achievements and can provide information regarding the overall quality of services, helping assure that the most effective practices in one grant are replicated in future grants. Documentation of results has assumed critical importance for all Federal departments and agencies with passage of the Government Performance and Results Act of 1993. In addition, evaluation is needed to guide policy and programmatic decisions concerning disaster crisis counseling at the national level.

What this Guidance Includes
This Program Guidance describes program evaluation activities that can be implemented building upon basic data collection and reporting activities already expected of all Crisis Counseling Program grantees. It is intended as a starting point for evaluation planning in individual projects and will be most useful when used in consultation with the CMHS/ESDRB project officer. It discusses concerns and misconceptions individuals sometimes have about evaluation and shows how evaluation can assist crisis counseling projects and improve program reports and activities. The guidance provides a starting framework for conducting process evaluations of disaster crisis counseling projects and summarizes the steps in implementing a process evaluation. Finally, a list of resources is provided.

A Note on Terminology: Evaluation has been defined in many ways. For purposes of this Program Guidance, evaluation is discussed primarily in terms of "process evaluation." A process evaluation focuses on program implementation. It describes how the program operates, how it delivers services, and how well it carries out its functions. By documenting a program's development and operation, a process evaluation provides some understanding of the performance of the program and information for potential replication, as well as for improvement. Although a monitoring system is essential for process evaluation, the evaluation goes beyond monitoring for purposes of accountability. Process evaluation seeks to improve the quality of services as they are delivered and seeks to document the overall performance of the program in meeting its objectives.

While the evaluation techniques described and encouraged in this guidance may involve the collection and analysis of data on the recipients of crisis counseling services, data collection with crisis counseling recipients should be limited in scope and must include extensive protections of any private information.

Process evaluation as described in this guidance does not seek to demonstrate that specific interventions are linked to a specific clinical outcome for individuals. While such studies are clearly of potential value in the field of disaster mental health, they are often methodologically challenging to implement in community-wide programs in the unpredictable environment of a disaster response and recovery effort. While some grantees may collaborate with other community partners in studies of clinical outcomes, studies of this nature are not expected of all grantees and must be approached with attention to a number of human subject protections.

While detailed outcome studies are not emphasized in this Program Guidance, the general steps in implementing an outcome evaluation are similar to those for a process evaluation, such as engaging stakeholders, developing appropriate questions and analyzing data. These steps are outlined throughout the guidance. Agencies with an interest in outcome evaluation may decide to collaborate with researchers external to the project, following approval of the CMHS/ESDRB project officer.

How Process Evaluation Can Help the Crisis Counseling Program
Carefully designed and implemented evaluation activities can lead to improved services, can identify individuals and groups not being adequately served, and can possibly help garner additional resources for a community following closure of the Crisis Counseling Program funding. Important reasons for conducting evaluations are discussed below.

Improving Program Implementation and Ensuring Quality Services
To ensure project effectiveness and maximum use of resources, ongoing information collection is essential. This is sometimes referred to as continuous quality improvement. The goal is to ensure that a project stays on course and is faithful to the initial model, while identifying problems or gaps that need attention. This aspect of the evaluation can help redirect resources and modify aspects of the plan that are not working as originally intended; it assists in making midcourse corrections. Some of the information in the initial needs assessment, for example, may have been incorrect–continually assessing needs and resources will guide changes to meet program goals and objectives.

Documenting Achievements
Another reason for collecting information is to provide a narrative history–a record of program activities. Grantees are required to report to FEMA and CMHS on program accomplishments and expenditures. In addition to meeting a grant requirement, a report that incorporates evaluation results accomplishes several goals: it may enhance the ability of a State or local agency to obtain support in the future, if needed; it provides program staff with a tangible record of accomplishments, documenting achievements; and it can also assist FEMA and CMHS in two ways: (1) helping future projects by sharing lessons learned, and (2) reporting to Congress on their stewardship of Stafford funds. When competent stewardship is documented, the opportunity for program survival and growth are improved. Some grantees have also found that describing a program's accomplishments is professionally satisfying and can enhance staff morale.

Measuring Performance
Evaluation helps a project demonstrate accountability by comparing its actual performance with expectations and by explaining reasons for differences. Analysis of data on outreach contacts, services provided, and satisfaction with services can help a program to assess effectiveness and understand why some activities were more useful than others. This information is valuable as a way of improving services in the future.

Evaluation Activities Encouraged of All Grantees
All Regular Service Program grantees are strongly encouraged to conduct process evaluations designed to improve service delivery and to ensure that the most vulnerable populations are receiving needed services. Evaluation should be conceived of as an integral and ongoing aspect of a crisis counseling project--not a stand-alone activity. Since every disaster is unique and every community is unique, it is anticipated that the evaluation process will be different for every grantee. Just as the crisis counseling program must be tailored to its special set of circumstances, so must its evaluation.

For purposes of the Crisis Counseling Program, process evaluation is defined in four distinct but overlapping phases: 1) an initial needs assessment, 2) ongoing quality improvement activities, 3) a detailed description of the program, and 4) a final review of activities and ongoing needs. Much of the information needed for a strong process evaluation is already being collected by many grantees in order to meet the basic reporting requirements of the Crisis Counseling Program regulations. A package of basic data collection instruments for program monitoring purposes is provided in a new CMHS Data Management Tool Kit, which is designed to help grantees meet the basic reporting expectations established under FEMA regulations. The data collection and analysis approaches described in this guidance seek to improve the quality and sophistication of data collection and assure that information gathered is used to improve project performance.

Initial Needs Assessment
Experienced crisis counseling grantees are familiar with the initial needs assessment, as it has long been required. A needs assessment in the Crisis Counseling Program has two purposes. First, the needs assessment helps document the fact that crisis counseling services are necessary and that the community does not have the capacity to respond with exisiting State and local resources. Second, the needs assessment is used in developing both a plan and budget for services.

Needs Assessment Data Collection Techniques

• Collect and analyze disaster statistics on loss related to the disaster
• Review resource guides to community services
• Conduct key informant surveys
• Analyze census and economic statisticscounseling services are necessary and that the community does not have the capacity to respond with exisiting State and local resources. Second, the needs assessment is used in developing both a plan and budget for services.

The key importance of a needs assessment is to drive and guide the program plan. In the needs assessment, the numbers of individuals in need of emotional assistance are specified, the kinds of problems they are facing are identified, and special issues in the community are described, such as the presence of particular cultural or linguistic groups. The needs assessment should be seen as developing the "road map" for the program. It also becomes the foundation of further data collection and analysis to ensure that services are provided appropriately.

Quality Improvement Process
A quality improvement system is designed to collect and organize management and programmatic information. It includes collecting data on both the internal and external operations of a program and periodically analyzing those data. Such a monitoring system provides ongoing feedback on the attainment of service delivery objectives and can ensure high quality reports on program activities and accomplishments. It helps to identify where corrective action needs to be taken, including possibly the reallocation of resources. The monitoring system may include collection of the following four types of data:

• Who: Information on the background characteristics of impacted populations with special emphasis on any high risk populations and their emotional needs;
• What: Data describing the activities of the program (i.e., is the program doing what it was supposed to do and do these relate to the original goals and objectives of the program);
• How: The ongoing human and financial resources allocated to the program; and
• Impact: Information on results (e.g., what is known about program participants and community linkages, what aspects of project implementation appear to be working well, how well the program is reaching the population intended for services, and what are the problem areas of the project).

Quality Improvement Data Collection Techniques
1. Compile information from logs and contact sheets
2. Organize focus groups
3. Procure and analyze staff reports from the field
4. Conduct key informant interviews
5. Compile statistical indicators
6. Conduct disaster survivor satisfaction surveys
7. Conduct interviews and/or surveys of staff in partner agencies

This aspect of the evaluation may also be thought of as a continuous needs assessment. It can be a powerful tool for identifying both the strengths and weaknesses of a program, help to ascertain the quality of services provided by the project, and provide guidance on phase-down activities.

Description of Program Activities
An important and sometimes overlooked component of process evaluation is a detailed description of all activities and services provided. Information on "input" is necessary for documenting the linkage between specific service activities and program results. Thus, a good program description is key for both replication of a program and for its evaluation.

Crisis Counseling Program grantees currently provide descriptions of activities required for quarterly and final program reports. Programs may also develop a more detailed and systematic "case study" for evaluation purposes. It could include descriptions of the service providers and their qualifications, description of training provided, detailed descriptions of outreach and crisis counseling approaches (i.e., did counselors work individually or in teams, were specific outreach and counseling approaches targeted to specific populations), descriptions of where services were provided, numbers and demographic characteristics of people receiving crisis counseling services, and other aspects of the program.

Description of Program Activities Data Collection Techniques

• Review original grant application
• Analyze meeting minutes
• Summarize quantitative information collected in Quality Improvement process
• Conduct staff interviews
• Review quarterly reports and other internal reporting mechanisms

A comprehensive description of program activities is qualitative in nature and provides rich descriptive data. It also captures important information about a program's social, cultural, and political context. Understanding the community is often key to understanding how a program works, and the program description needs to include such information. Disaster crisis counseling programs are multi-faceted, incorporating a variety of activities, and every program will have a different mix of these. Moreover, each type of activity has a different result. For example, a support group accomplishes something quite different from a newsletter. The process evaluation needs to explicate these activities and assumed results.

Final Review of Activities and Ongoing Needs
A review of activities and ongoing needs should be undertaken at the end of the project in order to document ongoing services that disaster survivors may need following the termination of the program, to identify any infrastructure from the project that was left in place, and to identify existing or additional resources to provide the continuing needs. This activity could be thought of as "renegotiating the future" since, if done correctly, it may generate additional and ongoing services for disaster survivors still in need following the termination of the Federally-funded program. This review may incorporate many of the same types of data sources used in the initial needs assessment and in the quality improvement process.

Final Review/Ongoing Needs Data Collection Techniques

• Conduct focus groups
• Collect updated disaster statistics
• Conduct interviews of staff in partner agencies
• Analyze statistical indicators

Some Common Concerns about Evaluation in Crisis Counseling
Program managers with a desire to conduct evaluation may encounter resistance. Some managers view evaluation as intimidating, or they may expect an evaluation to generate few if any helpful findings. Common concerns include the possibility of burdening the survivors; a belief that skills necessary for evaluation are not available; fear that the results of the evaluation will damage the program; and a belief that the specific nature of disasters, with their unpredictability, makes it impossible to conduct any evaluation. Each of these concerns is briefly discussed below, and the methods recommended in this Guidance are designed to address them.

Ethical Issues
In a disaster situation, by definition, the disaster survivors being served have suffered in many ways. Program providers may feel that gathering information which is to be used for evaluation as opposed to service delivery is unethical, that it places additional, unnecessary stress on survivors and is therefore counter to the goals of the Crisis Counseling Program. Moreover, mental health services are often viewed as stigmatizing, and gathering information while maintaining the privacy of disaster victims may be perceived as a barrier to evaluation. It is important to remember, however, that a key goal of process evaluation is to improve the capacity of the program to assist survivors, and the evaluation can be designed to minimize its impact on disaster victims while maintaining their privacy. The involvement of stakeholders at the outset can assist in designing an evaluation that addresses these concerns.

Some program administrators and community members may also be concerned that "research" will result in denial of services due to use of control groups. They should be assured that studies that withhold state-of-the-art services as a part of a control group will not be used in connection with the Crisis Counseling Program.

Limited Evaluation Expertise
Crisis counseling program managers may believe that evaluation is too complicated to undertake and that the necessary expertise is not available. However, evaluation can be undertaken with various levels of sophistication. Many aspects of process evaluation are already required for program reporting, are not mysterious, and can be implemented by an interested program manager. In fact, it is incumbent on program managers to undertake program assessments as a part of accountability.

For more technical evaluation activities, such as analysis of large data sets, program managers can build partnerships with experienced evaluators in their own agencies or in their communities. It should be noted, however, that even when evaluation consultants are brought into the program, it is expected that program managers will continue to play an active role in the design and implementation of evaluation activities. Considerations for program managers in selecting an evaluation consultant are discussed later in this document.

Concern about Results
Sometimes program managers actively resist undertaking evaluation, in the belief that it will bring criticism of the program. They may be concerned that an evaluation will show that a program is not quite as effective as it could be. Conversely, they may feel that it is obvious that the program is effective, so why waste time and energy studying it?

However, evaluation does not need to imply a simplistic judgment of good or bad; it can be more sophisticated and useful. When program managers, staff, and evaluation consultants work together as a team, a commitment to quality can emerge and defuse anxieties related to the evaluation. If an evaluation is designed with all the stakeholders involved–including staff and program managers–the evaluation can pose appropriate questions, enhancing the quality and value of staff activities and thus their satisfaction.

Chaotic Nature of Disasters
Evaluation is most efficiently accomplished when it is included in the preliminary planning for a program. Disasters preclude all but the most general of planning. They are unpredictable in location and in impact. Therefore, developing an evaluation plan in the midst of a disaster may be extremely difficult. However, with some pre-planning, it should be possible to implement a well-designed evaluation effort during the Regular Services Grant project. If evaluation is conceived of as an integral part of service provision, not as an adjunct, it can and should be included in the development of the Regular Services Grant project from the beginning and may build on basic data collected during the Immediate Services Grant phase.

Organizing and Implementing a Process Evaluation
Key Steps in Conducting a Process Evaluation

The key steps in evaluation are as follows:

• Engaging staff and other potential stakeholders in the evaluation process.
• Implementing feedback mechanisms.
• Developing evaluation questions.
• Determining the methods for data collection.
• Collecting the data.
• Analyzing the data.

These steps are described in more detail below.

Involving Staff and Other Stakeholders in Evaluation Process
There are three categories of stakeholders for any crisis counseling program: Stakeholders are key. They provide new perspectives, can help in implementation, and enhance credibility.

• Those Involved in Program Operations (Administrators, Managers, and Staff): Those who are most closely involved in the program have a very strong interest in its assessment.
• Collaborators and Partner Agencies: A crisis counseling program requires partnerships to be effective. Thus, considering the values and concerns of partners is essential in the evaluation as well as in the development of the project. For example, voluntary agencies in the community, housing agencies, disaster managers, and members of the unmet needs committees are groups that have an interest in a disaster crisis counseling program.
• Beneficiaries of the Program: Disaster survivors have an interest in ensuring a strong and useful program, one that is culturally competent and respects local and cultural values.

Stakeholders are key. They provide new perspectives, can help in implementation, and enhance credibility.

Involving stakeholders in the evaluation–as in the development of the program–ensures that their perspectives are understood. If they are not engaged, the evaluation might overlook important elements of the program. Stakeholders can also help to implement the evaluation. They can improve the credibility of the evaluation and help the project address ethical concerns, such as minimizing the impact on victims and ensuring their privacy.

There are several ways to involve stakeholders in the evaluation. These include consulting with representatives from as many stakeholder groups as possible; developing an evaluation task force and including representatives of the stakeholder groups; and providing timely feedback on the process of the evaluation. Grantees might also consider developing an advisory committee to function throughout the life of the project. A committee that includes representatives of the different partner agencies might provide innovative ideas that can help a program improve and be more effective as well as advising on the evaluation. If a stakeholder committee already exists, such as an unmet needs committee, it might be used for consultation for the evaluation.

Implementing Feedback Mechanisms
Since the most important reason for conducting a process evaluation is to improve the project, it is important not to conceive of it as being an activity that is essentially divorced from the ongoing operations of the project, only to result in a final report that is completed after the termination of the program. There are several ways to integrate the evaluation into the core of the project. At the beginning of the planning for the evaluation, the project administrators, staff, and other relevant stakeholders should begin considering feedback mechanisms for the evaluation. This could include weekly meetings with the evaluation team and administrators; monthly discussions or roundtables with a larger group; and biweekly memos from the evaluator(s) on insights and reflections for response and comment. Ongoing dialogue and frequent communication are essential elements in ensuring that the information is used to improve the project and should also assist the evaluation team to refine the design and interpretations.

Early and continuous feedback on evaluation findings can improve a project.

Identifying an Evaluation Team
The responsibilities for evaluation activities will vary among programs, depending on available resources. However, there are many separate tasks involved in the evaluation process requiring a variety of roles. Therefore, many stakeholders and partners will be involved in conducting the evaluation. Most programs will develop an evaluation team, which will include program stakeholders as well as one or more persons trained in evaluation methods. Certain aspects of the evaluation, such as gathering data on disaster survivors, may be undertaken by program staff, while the data may be analyzed by a statistician.

Some States have considerable in-house expertise in evaluation and can use existing staff to assist local programs with evaluation. An advantage is that the evaluation consultant is available for assistance. However, because many mental health evaluators are experienced primarily with programs targeted to people with serious mental illness, they will need an orientation to the crisis counseling program goals and activities.

In some cases, local programs are able to collaborate with a university, in which a class may assist with the evaluation as a way to provide students with applied experience; or a professor may serve as a consultant to the program staff. For highly technical evaluations, or where in-house or in-kind resources do not exist, the project may wish to contract with a professional evaluation organization for assistance in various aspects of the evaluation. When contracting out such work, it is important to ensure that the evaluators fully understand and share the philosophy of the program as well as the intent of the evaluation and how the results will be used. It would be important to consult with the CMHS/ESDRB Project Officer and FEMA staff at an early phase prior to selecting an evaluation consultant.

The evaluation team may include many individuals, including project management. Technical expertise can be acquired.

Depending on the questions asked, there are several considerations in deciding who will conduct different aspects of the evaluation. For maximum credibility, the stakeholders must have confidence in the skills of the evaluators. In addition, the evaluators must be as free from bias as possible so as to conduct an objective evaluation that will not be viewed with suspicion. An outside evaluator may generally appear to have greater objectivity, but this is not always the case. The question of potential bias and how to minimize it should be considered prior to the implementation of the evaluation.

The program management needs to be intimately and consistently involved in the evaluation–in determining the questions that will be asked and in deciding upon the best methods. If the evaluation is viewed as being outside the project, it is less likely that the results will be used to improve the program.

Identifying Evaluation Questions
The most important aspect of any evaluation is the identification of the questions that will be asked: The evaluation questions drive the study. After defining one or more important questions, one must then determine whether or not it is possible to answer them. For example, a question may not be clearly stated or its elements adequately defined. It may also be the case that there is no methodology that can be used to answer the question. Or, while it may be theoretically possible to design an evaluation study to answer a particular question, it may be quite expensive to conduct the study and sufficient funds may be unavailable. Determining whether or not a question can be asked clearly, whether there is a way to study it, and whether there is sufficient money to undertake an appropriate study is sometimes referred to as an "evaluability assessment."

Many experts now believe that using a "logic model" is a good way of identifying evaluation questions. A logic model is simply a diagram (perhaps a flow chart or a table) that shows the relationships between program resources and presumed outcomes; it represents the theory of how and why the program is assumed to work. By developing such a diagram, program stakeholders can sometimes identify areas of particular interest for the evaluation. A worksheet for developing a logic model and an example of a completed logic model are included as Appendix B.

It is important to ensure that the questions being asked in an evaluation are the ones that have the most salience for the program and can help improve and strengthen services.

Examples of some important process evaluation questions for disaster crisis counseling programs might include the following:

• Was the program implemented in accord with the initial program plan? If not, why not? What was the process of development and implementation? Why did it evolve the way it did? What were the challenges encountered in implementing the program plan?
• Were service recipients satisfied with the services provided? How and why did they react to the program the way they did? Did those who received services believe that these services were effective and important and helped them to recover from the disaster experience?
• What results can be identified for the program? Why were these results achieved? Were there unanticipated outcomes?
• Did the program reach those most in need? What criteria were used to identify and reach individuals and groups in need?
• What lessons have been learned about developing and implementing a disaster crisis counseling project that might be useful elsewhere?
• How have environmental factors affected the development, implementation, success and limitations of the program?

Determining Methods of Data Collection
Both qualitative and quantitative data collection techniques should generally be employed in evaluating crisis counseling projects. Quantitative methods employ counting and measuring; qualitative methods are descriptive and may be used to show how a project functions and what it means to those being served. Both quantitative and qualitative data should be used in process evaluation.

A beginning point for data collection is the "Data Management Tool Kit," developed for States by CMHS.

A starting point for quantitative data collection is the Crisis Counseling Program Data Management Tool Kit, developed by CMHS. The data collection instruments included in the Data Management Tool Kit were developed in consultation with State Mental Health Authorities and are designed to provide basic information on service contacts, services provided, and issues observed by Crisis Counselors. The forms in this data toolkit include forms on individual service contacts, group contacts, and material distribution. The forms are recommended, but not required. They may be used throughout the Immediate and Regular Service programs to compile statistics required in the application and the quarterly and final program reports. The State is free to alter these forms or replace them with their own forms in consultation with the CMHS/ESDRB project officer.

To assure accurate and high quality data, it is critical that all Crisis Counselors receive training on the use of data collection instruments and that data are provided in a timely fashion. Areas of data collection for basic program monitoring should include the following:

• Individual Service Contacts - Demographic Information,
• Individual Service Contacts - Observed or Reported Reactions,
• Individual Referrals,
• Group Contacts, and
• Material Distribution.

There are many additional techniques for collecting information that may be appropriate to process evaluations of crisis counseling projects. Some of these are briefly described below.

Qualitative Techniques:

• Observation: This approach consists of carefully observing a group. The evaluator identifies and notes certain processes or outcomes of interest and also attempts to describe and understand events in a particular context. For the crisis counseling program, direct observation of disaster survivors and how they interact with crisis counselors might be employed in selected situations. Such observation must be conducted in a manner that is sensitive to privacy and does not interfere with services.
• Interviews: Interviews can range from the very informal, in which the evaluator poses general questions and uses probing to ascertain an individual's story from his or her own perspective, to the highly structured, in which questions have clear and direct answers that can be easily coded. If an interviewer is highly skilled, a less structured interview can generate much more information, but such information is considerably more difficult to analyze. The questions might relate to ongoing needs of the community and/or the perception of the quality of the crisis counseling program. For process evaluation, interviews may be conducted with disaster survivors and/or staff in partner agencies.
• Discussion Groups: Discussion groups, also called focus groups, are small group discussions in which generally three to six questions are posed and the answers subsequently analyzed. Discussion groups are not designed to secure a consensus on an issue, but rather to gather information and perceptions. There are no wrong answers, and a diversity of opinions is encouraged. Discussion groups can provide a rich source of qualitative information, for example elucidating both strengths and problems in a program. They tend to be most successful when limited to eight individuals. Participants are selected because they share certain characteristics which make their opinions particularly relevant to the project. The interaction of a focus group is often stimulating, and participants may think of things that they would not have thought of individually.
• Analysis of Internal Documents: Meeting minutes and other documents that describe the program and how it evolved may be analyzed for information, such as the type of program components delivered, how decisions were made, and the impact of organizational structures.
• Analysis of Resource Guides to Community Services: State and local agencies should have access to information on mental health and community resources which can be used to generate an analysis of available community resources and the extent to which they can or cannot address the additional needs caused by the disaster. It may also be necessary to contact mental health organizations directly to obtain assessments from them post-disaster.
• Analysis of Staff Reports: The staff can provide program administrators with a good picture of what is happening and where there may be gaps. Information can be obtained from them at daily or weekly group meetings and/or in written formats. Interviews of staff could also be considered.

Quantitative Techniques:

• Surveys: A closed-ended questionnaire administered to a particular group of individuals can elicit information that is easily coded and analyzed. Surveys can be self-administered (e.g., through a mail survey) or administered by the evaluator (e.g., through a telephone contact). Disaster survivors or consumers can be surveyed to assess their degree of satisfaction with services and their perceptions as to the impact of the assistance received through the crisis counseling program. Staff in partner agencies should also be surveyed as well as actual program staff. Depending on the size of the program, a sample may be selected for the survey rather than the entire set of disaster survivors. Before conducting a survey, the program manager should contact his or her CMHS/ESDRB project officer.
• Compilation of Statistics: Disaster statistics on loss related to the disaster: These data are available from various sources, such as the American Red Cross, the FEMA disaster field office, and key informants.
• Census, economic statistics, and statistical indicators: A community assessment should include demographic, cultural, geographic and economic data in order to target services to the most needy and in order to design culturally appropriate services. Information can also be collected on different community indicators at different times to gauge the level of stress of the community. By using data collected from recipients of services along with social indicator and census data, a program may be able to link provision of services to a wide set of variables.

Identifying Appropriate Data Collection Instruments
In addition to selecting a method for collecting information, the evaluation team will need to decide on the instruments for collection. The term "instrument" includes survey questionnaires, interview protocols, coding forms, and other standard tools for data collection and analysis. In some cases, it is possible to conduct process evaluation without use of additional data collection instruments beyond those provided in the Crisis Counseling Data Management Toolkit. In other cases, grantees may need to need to find additional data collection instruments or develop their own instrument to help address key evaluation questions. The resources section of this Program Guidance lists instruments which have been developed in previous disaster crisis counseling programs that may be used or modified to fit a particular situation.

An evaluation may use existing instruments for collecting data or, if none exist that are appropriate, a program may need to create its own instruments.

If no data collection instrument exists that is appropriate to address the key questions of the evaluation, the evaluation team may need to create its own. The key issue in developing any new data collection instrument is to be clear on the information needs and ensure that the instruments used will actually obtain appropriate information from that source. Advantages of existing instruments are standardized (i.e., scores on particular items have been rated as "normal" and "non-normal"), and established validoty and reliability (validity means the instrument measures what it is supposed to measure and reliability means that responses are consistent over time). The disadvantages of using existing instruments is that they may not be appropriate for the Crisis Counseling Program. For example, existing instruments may be diagnostically oriented or refer to services not provided through the program. They may be inappropriate for the cultural or ethnic groups that comprise a community. If new data collection instruments such as surveys and interview schedules are developed, it will be critical to pilot test them. If necessary, an evaluation consultant can be used to assist in the process of developing evaluation instruments.

Analyzing the Data
In the data analysis phase of the evaluation, the information is interpreted and a judgment made about the meaning of the data that has been collected–what are the answers to the questions that have been posed and what do these answers mean?

Interpretation of data is the final, key element of an evaluation.

Different analytic techniques are used for quantitative vs. qualitative data. Many resources are available which describe in detail various aspects of data analysis, for both quantitative and qualitative data. Resources for technical assistance and additional information are provided as an attachment to this document.

Analysis of Quantitative Data
Organizing quantitative data can be accomplished in a variety of ways, depending on the scale of the project and the resources of the agency. In small programs, periodic manual tabulations might be adequate. For larger programs with complex evaluation questions, data might be handled with statistical software packages. In very large programs, data might also be sent to external firms for data entry and analysis.

Complex statistical analyses that have little meaning for the program staff are less likely to be useful than simpler analyses that provide an answer to a meaningful question.

Basic descriptive statistics should be considered first; perhaps the most meaningful data are simply percentages or averages, such as the demographic breakdown of people receiving services and how that compares to the original outreach objectives of the program. Frequency distributions of the most important variables, relationships between variables, and cross-tabulations can also be used. In a process evaluation, complex statistical analyses that have little meaning for the program staff are of considerably less use than simpler analyses that provide an answer to a meaningful question.

Analysis of Qualitative Data
Qualitative data, such as that obtained from focus groups or unstructured interviews, may present a greater challenge for analysis, but techniques do exist to assist in converting large amounts of qualitative information to useful data. If the evaluation team does not include anyone experienced in data analysis, it may be useful to consider using a consultant for this purpose. Qualitative data is more than anecdotal: it can be analyzed for common themes or recurring issues. Qualitative analysis may include categorization, coding, and contextual analysis of information. Evaluators may review program materials for themes and then categorize the themes and patterns. It then becomes possible to sort by the key categories. Case studies or narrative summaries are additional techniques that may enable an evaluator to put the themes and patterns into an appropriate context.

Drawing Conclusions - Developing Standards of Success
Program managers and evaluators must agree upon standards that will be used to judge the meaning of the findings. For example, if an evaluation question addresses program success in outreach, is an outreach program successful that reaches 15% of disaster victims? The answer depends on what the program and the community define as adequate and appropriate. When different stakeholders have different standards, there may be differences as to what one concludes from the data analysis. To the extent possible, community stakeholders should develop standards and objectives at the outset of an evaluation and continue to update as the project progresses.

Conclusion
This Program Guidance was developed to focus attention on the need for evaluation of crisis counseling projects. It has provided a brief summary of evaluation methodology. Project planners should use this information to enhance and improve their current approach to evaluation as a way to better serve disaster survivors. The extent of the evaluation effort will vary among communities and among disasters. As FEMA and CMHS further examine their goals related to evaluation, they will develop additional ways of improving knowledge about the Crisis Counseling Program, in partnership with States, communities, and established researchers. This may include supporting local projects to undertake more rigorous evaluations than have been possible in the past. Grantees are also encouraged to collaborate with researchers; for example, those who are applying for grants through the Rapid Assessment Post Impact of Disaster (RAPID) program of the National Institute of Mental Health. However, it is critical that grantees consult with their CMHS/ESDRB project officer to ensure that such projects are in keeping with the program's regulations. If program managers have questions regarding additional research they should contact their project officer to discuss these. Additional resources are provided in appendices.

Appendix A
SOURCES OF ADDITIONAL INFORMATION AND TECHNICAL ASSISTANCE

CMHS/ESDRB can assist grantees in a number of ways to improve their ability to conduct process evaluations, and additional technical assistance materials are planned for development in the future.

Data Collection Instruments

Data Collection Tool Kit: The data collection instruments included in the Data Management Tool Kit were developed in consultation with State Mental Health Authorities and are designed to provide basic information on service contacts, services provided, and issues observed by Crisis Counselors. The material in this data toolkit include forms on individual service contacts, group contacts, and material distribution. The forms are recommended, but not required. They are used throughout the Immediate and Regular Service programs to compile statistics required in the application and the quarterly and final program reports. The State is free to alter these forms or replace them with their own forms in consultation with their project officer.

CMHS Technical Assistance Resources: Technical assistance in evaluation may be available from CMHS or its partners. In addition, CMHS has developed an outside technical assistance evaluation capacity available to all CMHS grantees. The Evaluation Center@HSRI is a technical assistance center funded by the Center for Mental Health Services (CMHS) and operated by the Human Services Research Institute (HSRI). While the staff have limited specific experience with the Crisis Counseling Program, they do have strong expertise in evaluation The internet web site for this organization is as follows:

http://www.eval.hsri.org

Contact information for the Evaluation Center at HSRI is provided below:

Human Services Research Institute
2269 Massachusetts Avenue
Cambridge, MA 02140
Phone: (617) 876-0426
Fax: (617) 492-7401

Examples of Useful Web Sites
Centers for Disease Control and Prevention
http://www.cdc.gov/eval

This site, supported by the U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, provides a description of the public health approach to evaluation, in a clear and straightforward manner. It includes links to other good web sites with additional information on program evaluation, including numerous on-line publications that can be downloaded.

Bureau of Justice Assistance Evaluation Web Site
http://www.bja.evaluationwebsite.org

This site, supported by the U.S. Department of Justice, Bureau of Justice Assistance, provides a primer on evaluation, including process evaluation. While the examples are oriented to projects of the Department of Justice, the text is generic to evaluation of community-wide programs.

W.K. Kellogg Foundation
http://www.wkkf.org/

This site includes a downloadable version of the evaluation handbook developed by the W.K. Kellogg Foundation for its grantees. It provides much useful information for evaluating projects that are community-based.

University of Kansas
http://ctb.ukans.edu/

This site includes over 3,000 downloadable pages on evaluating community programs. User friendly and comprehensive, it is maintained by the University of Kansas.

Useful Publications
Afifi AA, Clark V. Computer Aided Multivariate Analysis. 2d ed. New York: Chapman and Hall, 1990.

Description: A text on linear models and regression analysis.

Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999;48(No. RR-11):1-40.

Description: A description of all the elements of a good evaluation for public health programs, including standards for effective evaluation.

Kvale, S. InterViews: An Introduction to Qualitative Research Interviewing. Thousand Oaks, CA: Sage, 1996.

Description: A comprehensive book written for practitioners but also useful for researchers.

Lunnenberg C. Modeling Experimental and Observational Data. Boston: PWS-Kent, 1994.

Description: A text on linear models and regression analysis.

Maxwell, JA Qualitative Research Methods. Thousand Oaks, CA: Sage, 1996.

Description: A comprehensive look at qualitative research.

Patton, MQ Utilization-Focused Evaluation. 3rd ed. Thousand Oaks, CA: Sage, 1997.

Description: A clear and readable text providing good guidance on evaluation strategies that can lead to useful–and used–evaluations.

Rossi, PH & Freeman HE Evaluation: A Systematic Approach. 5th ed. Thousand Oaks, CA: Sage, 1993.

Description: A highly regarded text on evaluation.

Weiss, C H. Evaluation, Methods for Studying Programs and Policies. 2nd ed. Upper Saddle River, NJ: Prentice Hall, 1998.

Description: An easy-to-read and comprehensive text on evaluation.

Weitzman, EA & Miles MB. A Software Sourcebook: Computer Programs for Qualitative Data Analysis. Thousand Oaks, CA: Sage, 1995.

Description: A sourcebook addressing the relative advantages of different computer programs for qualitative analysis.

Appendix B
Example of a Logic Model

Program Logic Model Crisis Counseling Example

Note: The following logic model is provided for illustrative purposes only. For each individual evaluation, a logic model is recommended to address the specific issues and questions being addressed in the evaluation.

References
Bond SL, Boyd SE, & Rapp, KA. Taking Stock: A Practical Guide to Evaluating Your Own Programs. Chapel Hill, NC: Horizon Research, Inc., 1997.

Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999;48(No. RR-11):1-40.

Scheirer MA. Designing and using process evaluation. In Wholey JS, Hatry HP, and Newcomer KE (eds), Handbook of Practical Program Evaluation, San Francisco: Josey-Bass, 1994.

Weiss CH. Evaluation, Methods for Studying Programs and Policies. 2nd ed. Upper Saddle River, NJ: Prentice Hall, 1998.