This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
2001 Annual Report to Congress on the Evaluation
|
|
CHAPTER SUMMARY In summary, special studies conducted as part of the national evaluation to help address the question of effectiveness of systems of care have found that
|
The national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program is designed to provide information about the impact of systems of care at multiple levels using a standard approach and instrument package. However, there are also areas that are of importance in the study of systems of care that are not addressed in the standard evaluation. To allow the flexibility needed to pursue some of these additional questions, the national evaluation has, over time, included a variety of special studies. Three special studies are currently underway. The first of these is a study of families' service experiences in systems of care that is being conducted by family members. The second is a study of the factors that facilitate and impede the sustainability of systems of care beyond their funding periods. The third study is an examination of the effectiveness of specific treatments and services within systems of care. This chapter describes the current status of these ongoing special studies.
The goal of the family-driven research study is to support family members and youth as they conduct an investigation on how families are experiencing systems of care. The results of this study are intended to be used to make service systems more responsive to families. Family members often have reported that the areas being studied by researchers are not topics that are of high priority to families. This study will endeavor to determine what is important to families and how best to obtain information from them that can lead to improved service delivery systems. The perspectives and interests of families raising children with serious emotional disturbance who are enrolled in system-of-care grant communities govern all aspects of this project. This study is being conducted under the leadership of the Federation of Families for Children's Mental Health and the Georgia Parent Support Network in collaboration with ORC Macro.
This study operates under the guiding principle that "Systems of care need to ensure that families have all the information they need about all of their options. Only then can families choose the right path to healing that they can follow." |
This study is being conducted by a field-based team of family members and youth who have experience with communities funded by the CMHS grant. Researchers from the national evaluation team are also on the team. This group works together through monthly conference calls, e-mail and faxes, and occasional face-to-face meetings to accomplish the five primary stages of the project. They are as follows:
The team has nearly completed the first stage and is working toward the second stage. A summary of the project's accomplishments and future plans is provided below.
Although the general goal of this project is to study how families experience systems of care, it was necessary to refine the question and to focus on a particular area of service experience. The family-driven study used a modified Delphi process to determine specifically what research questions would be addressed in this project. This process was followed by three adult and one youth focus group to finalize the research questions and to shape the study design and data collection procedures. The two rounds of the Delphi process and the focus group procedures are described below.
The first round of the Delphi process was a brainstorming activity designed to obtain a large pool of possible topics from a broad set of individuals. In this round, the project team identified experts on systems of care for children with serious emotional disturbance and their families. These individuals came from multiple disciplines and represented a variety of perspectives, cultural backgrounds, and regions of the country. The final list of potential respondents included family members, service providers, administrators, and advocates. They were sent a brief description of the project and asked to provide written responses to each of the following two questions:
Top 14 Themes Identified in Round One of the Delphi Process
|
Of the 100 individuals contacted, 47 responded to the first round of the Delphi process. Among them were youth, family members, community leaders, and other advocates. There was a total of 47 responses to the first question and 43 responses to the second question. These responses were entered into a qualitative data management software program. The team organized these responses and identified over 55 themes that emerged from the roughly 900 ideas expressed by the participants. The team then tabulated the number of ideas under each theme to determine which themes the respondents mentioned most frequently. The 10 most often-mentioned themes were selected to be used in the second round of the Delphi process. Four additional themes were selected that were mentioned frequently by respondents and that project team members determined were too essential to families' service experiences to leave out.
The purpose of the second round of the Delphi process was to distill further the themes identified in round one by having the original 100 Delphi participants prioritize the importance of a subset of the themes generated by the ideas they had previously provided. The project team developed short descriptions for the 14 selected themes. A ballot with 14 themes and their descriptions was mailed to the same experts who received the round 1 materials, whether or not they participated in the first round. They were instructed to read all the themes carefully and then distribute 15 "votes" among the ones they thought most important. Fifty-eight ballots were returned in round 2, and the votes were tallied. Five themes clearly rose to the top:
Based on the results of the Delphi process and following lengthy discussions of the themes, the project team concluded that the study should focus on families' engagement in systems of care. The idea of "engagement" was chosen because it encompasses both empowerment and involvement. The project team defined engagement as "the act of doing something for your child, your self, or your family that determines or derives from a care plan or supports the delivery of services and supports." Engagement can also be associated with "participation of families and youth with the intention of improving or enhancing service planning and delivery of treatment, services, family supports, or care."
Following the conclusion of the Delphi Process, focus groups were planned to obtain additional feedback. The purpose of the focus groups was (a) to learn whether the idea of family and youth engagement in systems of care was salient to family members, (b) to discern whether family members perceived a connection between family and youth engagement and child outcomes, (c) to identify barriers to obtaining candid responses from family members and youth about their experiences with engaging in services, and (d) to receive input on data collection procedures that would maximize participation of families in the study.
Some Respondents' Comments Included . . .
|
In July and August 2001, a series of small focus groups were held with family members and youth in Florida, Georgia, Utah, and Montana. The Montana focus group was held with youth who have received services through systems of care. Each focus group contained no more than nine members. Data including audiotapes currently are being transcribed, and the text will be analyzed. Feedback from the focus groups are being used to refine the research question, to develop the research design, and to shape data collection instruments and procedures.
A timeline has been developed to guide project activities. In the fall of 2001, efforts will be devoted to developing the research design, instruments, and procedures and to obtaining clearance from the Federal Office of Management and Budget to collect data. Data collection is scheduled to begin in 2002 and will continue for 10 months. Data cleaning and analysis will then be conducted, with dissemination of findings scheduled for 2003.
Critical challenges for systems of care are sustaining the infrastructures and services developed or enhanced during their funding periods and continuing to develop and evolve the systems after the Federal funding cycles are completed. Observations during the system-of-care assessments conducted as part of the national evaluation suggest that there are significant variations among grant communities in their experiences and expectations regarding their ability to maintain their services and infrastructure when Federal funding ends. System leaders allude to many factors that may affect the sustainability of the systems of care developed with the support of Federal CMHS grants. Some of these are related to the approaches used to develop and finance the systems of care, while others are related to the larger policy contexts and environments in which the systems of care operate.
This special study is designed to assess the potential of grant-funded communities to sustain their systems of care beyond their Federal funding periods and to gather information that will be useful to Federal and State policymakers as well as to local systems of care. The study approach involves learning about sustainability from the experience of earlier grantees in order to assess the potential sustainability of the systems of care and services of newer grantees. By examining grantees after and near the end of their funding periods, the study will identify factors that contribute to or impede the ability of communities to sustain their systems of care. Grant communities at earlier stages in their funding cycles will then be examined against these factors to assess the degree to which these systems of care may be sustainable beyond the Federal funding cycle.
This study includes (a) review of relevant literature and previous work on the sustainability of systems of care, (b) review of information about sustainability collected through system-of-care assessments, and (c) three phases of telephone interviews. Interviews will be conducted on a voluntary basis in a sample of grant communities. Telephone interviews will be conducted at both the community level and the State level with both grant communities whose funding has ended (or soon will end) and those currently funded. Key stakeholders to be approached for interviewing at the community level will include the grant community director, other key staff, and family members. Key stakeholders to be approached at the State level will include the State children's mental health director and other key staff.
Interviews at the community level will focus on identification of aspects of systems of care that are likely to be sustained and aspects that are less likely to be sustained, factors affecting sustainability, and what effects these factors have had in each grant community. The first phase of interviews conducted at the State level will explore factors affecting the sustainability of the CMHS-funded local systems of care and the relationship between sustainability and State policy. Information gathered during the first phase of data collection will be used to revise items related to sustainability on the system-of-care assessment protocol.
At the conclusion of the interviews, a work group meeting will be held to review the existing literature, data from system of care assessments, and findings from the recent telephone interviews. Using this information, the work group will identify the most significant facilitating or impeding factors. The group also will assist in developing a process for assessing the potential sustainability of newer grant communities. The work group will include the directors and key staff of grant communities and representatives of the Child, Adolescent and Family Branch of CMHS, the Technical Assistance Partnership for Child and Family Mental Health, the Federation of Families for Children's Mental Health, the National Technical Assistance Center for Children's Mental Health, the Research and Training Center for Children's Mental Health, ORC Macro, and other key stakeholders. The interview protocol used in the first phase of interviews will be revised to focus on aspects of systems of care that are the most and least likely to be sustained, factors affecting sustainability, and effects these factors may have. Telephone interviews using the revised protocol will then be conducted with stakeholders in currently funded grant communities at both the local community and State levels. Interview data obtained in this round will be used to describe and assess each community's potential for sustaining the programs, services, structures, and philosophies developed with program grant support.
In the fourth and final year of the study (projected to be conducted in FY 2003), a third phase of interviews will be completed with the same grant communities interviewed in the second phase. The goal of these interviews will be to explore further issues and progress related to the potential sustainability of their systems of care and services. In order to track change over time, the revised protocol used in the second phase will also be used in the third phase. All information gathered across grant communities during the course of the study then will be synthesized, and a final report will be written. The report will be geared to leaders of local systems of care as well as to Federal and State policymakers. The report will enable these individuals to consider sustainability issues in the earliest phases of system development and to learn about strategies and approaches that may enhance their efforts to build enduring systems of care. Based upon the study findings, the sustainability-related items on the system-of-care assessment site visit protocol will be further refined and revised.
To date, the research literature and information pertaining to sustainability collected through the system-of-care assessment site visits have been reviewed. A discussion group was held at the Comprehensive Community Mental Health Services for Children and Their Families Program Spring 2001 Grant Community Meeting. Issues pertaining to sustainability in both the services and infrastructure domains were explored with a diverse group of stakeholders in the system of care. About 45 representatives from current and previously funded grant communities participated. Regarding the services domain, participants noted that because system-of-care grant funds have been used largely to plug holes in local systems rather than wholly pay for new service systems, there is hope that services may continue beyond the Federal funding period through other sources of funding. Continuation of respite services was of major concern to the group, as it is one of the preventive services that parents consider very important.
Regarding the infrastructure domain, participants expressed varying opinions about what will happen in the future with respect to systems of care. Several participants perceived that agencies would become less collaborative if system-of-care services were not maintained, and there would be an overall reversion to the way mental health services were provided in the past. Many agreed that if grant communities were given more time and possibly less money to implement systems of care, more positive change could be observed at all levels. Many noted that at the current stage of system-of-care development, structures still are not formally and institutionally changed, but that infrastructures are being strengthened by establishing trusting relationships with new partners across communities. Several participants also noted that certain services such as wraparound services are saving money and thus will be maintained by systems because their value is recognized.
A meeting will be held at the beginning of May 2002 with the director of the study, a national evaluation team staff member, and the participating consultants. The meeting will be used to discuss the review of the literature and previous work, the review of the sustainability-related information in the system-of-care assessment site visit reports, and the creation of the interview protocol for the first phase of data collection.
The evidence-based practices in systems of care initiative was funded as part of the national evaluation beginning in 1999. The intent of this initiative was to begin the process of identifying and studying the effects of evidence-based practices for children's mental health disorders within systems of care. Initial designs were proposed to evaluate the effectiveness of evidence-based practices in funded systems of care and matched comparison communities that had not received Federal funding. These designs called for random assignment of children in the systems of care who met specific diagnostic criteria into an evidence-based practice or other system-of-care services alone. A similar process would occur within the comparison community, which would provide services as usual.
Because funds were not available to support the implementation of the evidence-based practices in either the system-of-care or comparison communities, it was necessary to alter these designs to study only evidence-based practices already underway in the system-of-care communities.
In addition, a growing body of literature has drawn attention to problems inherent in the transportability of evidence-based practices that were developed through a clinical trials approach. Previous research has found that evidence-based practices have been shown to lead to improved outcomes for children in clinical trials, but transporting those treatments to community settings has not had the same positive impact (e.g., Burns, Hoagwood, & Mrazek, 1999; Weisz, 2000). Several observers have noted compromises in treatment fidelity as a reason for limited effectiveness in community settings (Schoenwald & Hoagwood, 2001). Another explanation offered is that the samples used in the clinical trials were carefully selected to maximize the success of the evidence-based practice (Hoagwood, Burns, Kiser, Ringeisen, & Schoenwald, 2001). Children with comorbidities and disadvantaged families experiencing multiple day-to-day stressors typically were excluded from the trials, but they cannot be excluded in community implementations. Moreover, it is likely that in community settings children are enrolled into evidence-based practices who have different diagnostic characteristics than those targeted by the treatment (e.g., treating depression with an intervention designed for conduct disorder). Several articles have recently entered the literature noting that the field may not yet be ready for comprehensive treatment effectiveness studies for evidence-based practices (Burns, 2001; Burns et al., 1999; Farmer, 2000).
Instead of focusing on the implementation and resulting effectiveness of an intervention that has been established previously as evidence based, the evidence-based practices study has been altered to investigating the effectiveness of evidence-based practices in conjunction with existing services within systems of care. The thrust of the newly labeled evidence-based practices study is to identify and investigate the combined effectiveness of a specific service or group of services and an evidence-based practice within systems of care. At present, these studies will be conducted only within funded communities and will not involve testing the effects of evidence-based practices in comparison communities.
An incremental approach has been instituted for the evidence-based practices study. In the last year, the national evaluation has begun to focus on evaluating the effectiveness of specific services delivered to children and their families within systems of care. Much of this work has been driven by the increased interest in evidence-based practices and the issues associated with successfully moving practices that have been supported by controlled efficacy trials into community settings. The process of implementing and sustaining effective practices in community settings is complex but critical to addressing the mental health needs of children in the future, as noted in the SGR. ORC Macro has worked actively with staff at the Child, Adolescent and Family Branch and other partners to develop designs for evaluating the effects of specific services and evidence-based practices delivered within system-of-care programs. This intense and time-consuming process evolved over the last year to the point of identifying evidence-based practices that are candidates for implementation and evaluation at study sites over the next several years. The nominations process for selecting candidate practices has involved contacting professional associations, researchers, policymakers, service delivery personnel, and families. Personnel in the field are in the process of submitting nominations of specific practices for inclusion in these studies. This inclusive process will identify practices that target prevalent difficulties encountered by children and adolescents with serious emotional disturbance and their families. These practices could be treatment oriented or preventive in orientation. These practices will have evidence of research support. In addition, practices will be identified whose relationships to system-of-care principles have been investigated, at least on a preliminary basis. Practices that already have developed measures of implementation fidelity are also of high priority.
Once evidence-based practices are selected for study, the investigation process will begin with small-scale pilot studies over the next year to evaluate feasability. These studies eventually will be supplanted by larger-scale effectiveness trials over the next several years within funded communities. This iterative process should allow for the development of a rigorous yet realistic approach for evaluating the effectiveness of services within system-of-care programs.
Special studies are a component of the national evaluation that assist with developing timely and important information relevant to evaluating the effectiveness of systems of care. Current studies support an expanded perspective on family involvement in systems of care and the development of policy-relevant information for sustaining systems of care. In addition, the evidence-based practices studies address the important topic of integrating research-based practices within comprehensive services provided to children and families at the community level. The continued progress of these studies over the next several years will help to address fully the critical question of effectiveness of services at multiple levels: the level of family experiences, the level of policies that assist with sustaining service systems, and the level of integrating cutting-edge research in the treatment of children's mental health disorders within systems of care.
|
| Home | Contact Us | About Us | Awards | Accessibility | Privacy and Disclaimer Statement | Site Map |
