This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
2001 Annual Report to Congress on the Evaluation
|
|
The Comprehensive Community Mental Health Services for Children and Their Families Program is our nation's principal response to the service needs of the estimated 4.5 to 6.3 million children who have a serious emotional disturbance (Friedman, Katz-Leavy, Manderscheid, & Sondheimer, 1999) and their families. It provides grants to States, communities, territories, and American Indian tribes to improve and expand their systems of care to meet the needs of children and adolescents with serious emotional disturbance and their families. These include children and youth from birth to age 21 who currently have, or at any time during the past year had, a mental, behavioral, or emotional disorder of sufficient duration to meet diagnostic criteria specified in the Diagnostic and Statistical Manual for Mental Disorders, Fourth Edition (DSM-IV; American Psychiatric Association, 1994) that resulted in functional impairment that substantially interferes with or limits one or more major life activities. The program is administered by the Child, Adolescent and Family Branch within the Substance Abuse and Mental Health Services Administration's (SAMHSA) Center for Mental Health Services, U.S. Department of Health and Human Services. The first four grant communities were funded by this program in 1993. By 2001, the initial investment of $5 million had grown to $464 million, the largest Federal investment ever in community-based mental health services for children and their families. To date, 67 grants have been awarded, each for a period of at least 5 years ( see Figure 1 and Table 1). As of April 2001, 47,303 children and their families had been served.
|
It is useful to provide an historical context before discussing what has been learned about the effectiveness of systems of care. Over the past two decades, there has been dramatic change in the mental health service delivery system for children and their families. Twenty years ago, Jane Knitzer described children with serious emotional disturbance as being "unclaimed" by the public agencies responsible for serving them (Knitzer, 1982). Her survey of 50 States and the District of Columbia in 1981 found that the most readily available type of service for these children was inpatient hospital care, in spite of the fact that at least 40 percent of placements were deemed inappropriate. Even though intensive, community-based nonresidential services had been found to be cost effective, 18 State mental health departments continued to increase the availability of residential care, no State department had the capacity to provide nonresidential services such as day treatment, and just seven State departments had even begun to take initial first steps toward creating a range of service options for children with mental health needs. Knitzer found that few State mental health departments had policies that focused on children's issues, fewer than half of the States responding to the survey (21 of 44) indicated they had a special unit within the State department that was staffed by people solely responsible for and knowledgeable about children's mental health issues, and just 15 percent had a separate State budget for children's mental health services.
The Federal government responded to Knitzer's findings by authorizing the Child and Adolescent Service System Program (CASSP) in 1984 to assist states and communities to initiate a planning process to improve services for this population. CASSP was built on the system-of-care approach as outlined by Stroul and Friedman (1986) and described above. This was followed by other initiatives, both foundation and State supported, using a similar approach and culminated in 1993 with The Comprehensive Community Mental Health Services for Children and Their Families Program.
By 1993, when the status of children's mental health service systems in the States was revisited through a study that basically replicated Knitzer's 1982 study, considerable change in the philosophy, administration, and operation of services was found (Davis, Yelton, Katz-Leavy, & Lourie, 1995). Inpatient hospital care no longer appeared to be the most readily available and used service option, most States reported that they had targeted State funds to develop alternative resources, and day treatment services were being used with greater frequency. In addition, most State departments of mental health (40 of 48 reporting) had formed separate units to focus on children's services, every State had designated a person to be a member of the Children's Division of the National Association of State Mental Health Program Directors, and 70 percent of States had separate budgets for children's mental health services.
The advent of the Comprehensive Community Mental Health Services for Children and Their Families Program has resulted in widespread implementation of the system-of-care approach and principles. Since its inception, the potential for children and their families to receive mental health services and supports in their own communities has grown exponentially, as has the number of providers and stakeholders knowledgeable about and committed to delivering services using a system-of-care approach. Grant-funded communities have actively expanded their service arrays, adding new services such as mentoring, respite, and family support and tailoring others to meet the specific needs of their communities (Vinson, Brannan, Baughman, Wilce, & Gawron, 2001). Breaking with the past, the norm in the grant communities is for families to be partners in service planning and provision and, in many grant communities, in evaluating the services (Osher, van Kammen, & Zaro, 2001). There is a growing recognition of the importance of natural support systems within culturally diverse communities and the advantage of adapting services to be congruent with them. More is known about how to embed new interventions in service organizations (Burns, in press), and many child-serving agencies across the country have become committed to doing business in a new way, one that involves interagency collaboration and leads to more individualized and less restrictive services. In some cases, changes in policies at the State and Federal levels have led to legislation that supports system change, both within and beyond the grant communities (Holden, De Carolis, & Huff, in press). Finally, system-of-care proponents have been able, in some instances, to use the advent of managed care to further system-of-care goals such as improving interagency collaboration, expanding the array of services and providers, and improving access to mental health care (Stroul, Pires, Armstrong, & Zaro, in press).
Several studies have shown that the change in the way services and supports are delivered using the system-of-care approach has resulted in (a) more children and youth being served in more timely and appropriate ways, (b) more coordination and integration among agencies serving this population, (c) less exiting of services prematurely, and (d) generally more satisfaction with services expressed by families (Bickman et al., 1995; Burns, Farmer, Angold, Costello, & Behar, 1996; Rosenblatt, 1998). In addition, studies have shown that children served within systems of care have improved behaviors and functioning over time (Burns et al., 1996; Center for Mental Health Services [CMHS], 1997, 1998, 1999, 2000; Clark, Lee, Prange, & McDonald, 1996; Rosenblatt, 1998).
Mental Health: A Report of the Surgeon General documents the progress that has been made and the resources devoted to transforming the nature of service delivery for children with serious emotional disturbance and their families (U.S. Department of Health and Human Services [DHHS], 1999). The report (hereinafter called the SGR) calls for a refocusing of research from system-level changes and performance to an examination of the effects of system change on individual outcomes. It questions the degree to which changes at the system level have affected clinical outcomes and provides a context for the further refinement and evaluation of systems of care. Having a better understanding of this question of effectiveness is especially important in an era of managed care, accountability, and budget cuts.
In shifting the focus from system-level change to individual-level change, it is important to keep in mind the historical context briefly described above. Initial system building efforts were focused on the philosophy, organization, and processes of delivering services. In terms of this focus, these efforts have been very effective: The goals have been taken seriously; political support, legislation, and resources have been forthcoming; and the service delivery system for children with serious emotional disturbance has greatly improved over two short decades.
This shift in focus, however, requires clarification about what is meant by outcomes and effectiveness. Hoagwood and colleagues have identified five outcome domains that represent significant areas in which to look for evidence of effectiveness: symptoms, functioning, consumer perspectives, environments, and systems (Hoagwood, Jensen, Petti, & Burns, 1996). Most recently, Farmer (2000) details some areas in which answers to current questions would be of benefit in highlighting the effectiveness of these programs. For example, what interventions are most likely to be effective for which outcomes? Is improved consumer satisfaction an indication of effectiveness? Are short-term or long-term changes in individual child and family functioning required for a system or service to be effective? How are outcome domains linked together, and are some more important than others? She points out the need to develop a realistic consensus about appropriate outcomes for this seriously impaired population and realistic timeframes for achieving these outcomes. She also notes the lack of information on the overall cost effectiveness of systems of care.
As the number of system-building initiatives grows and the research and evaluation base develops, the knowledge base also grows and develops. The purpose of this report is to describe what has been learned over the past 8 years about the development of systems of care, the progress made by those children and their families served by the system, their service experiences, and service costs.
The Center for Mental Health Services has a strong commitment to promoting knowledge development and application. Reflecting this commitment, a national, cross-site evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program began in 1994 and is being conducted by a national evaluation team led by ORC Macro. The evaluation responds to the legislation that authorizes the program and calls for an annual evaluation to (a) describe the children and families served by the system-of-care initiative; (b) assess how systems of care develop and what factors impede or enhance their development; (c) measure whether children served through the program experience improvement in clinical and functional outcomes and whether those improvements endure over time and why; (d) determine whether the consumers are satisfied with the services they receive; and (e) measure the costs associated with the implementation of a system of care and determine its cost effectiveness. Besides responding to the legislation, the evaluation serves as a laboratory for addressing many of the questions described above. Findings from the evaluation will help illuminate the community parameters that may facilitate or impede the effectiveness of services and can provide information upon which to base future treatment, programmatic, funding, and policy decisions (Holden et al., in press).
The evaluation also provides data on program progress toward meeting goals set in the Government Performance and Results Act (GPRA) measures for this program. These measures include the number of children receiving services, interagency collaboration, utilization of inpatient and residential treatment programs, functional outcomes, family satisfaction, stability of living arrangements, and clinical outcomes after 6 months of service delivery. In FY 2001, a new measure on number of children served was added to the GPRA. Table 2 provides a summary of results on the key indicators of the seven measures for FY 2001. Performance targets for FY 2001 were met or exceeded, with the exception of the following four indicators: (a) percent referred from non-mental health agencies, (b) percent with regular school attendance, (c) percent with no law enforcement contact, and (d) percent satisfied with services received. More information about program performance on these measures is presented in detail throughout this report.
Any evaluation must be undertaken with the recognition that a complex set of factors determines the outcomes for a particular child and family. Key questions often posited in mental health services and research are: What works? For whom? And under what conditions? These questions are not easy or straightforward to answer. The service system is one critical factor, but others such as child and family characteristics and the quality of treatment must be taken into consideration as well (Burns, 1996). The national evaluation of the Federal grant program is designed to address many complex and related dimensions of effectiveness. It is longitudinal in nature; thus, children and families are followed over time, so that changes in outcomes can be understood from a developmental point of view. It includes a comprehensive assessment of outcomes across several domains. The service delivery systems also are assessed over time, so that their developmental trajectories can be better understood. This includes identifying the ingredients necessary to sustaining systems of care, whether system-level changes result in concomitant practice-level changes, and how families engage in systems of care. Other critical questions addressed by the evaluation include whether systems of care are more effective than traditional service systems in improving outcomes for children with serious emotional disturbance and whether providing community-based mental health services and supports to this population is cost effective. This level of complexity is necessary to understand the relationships between system, practice, and individual outcomes, recognizing that levels of change can occur simultaneously.
The evaluation is comprehensive and includes the 67 grantees, the children and families served by the programs, service providers, and partner agencies (Holden, Friedman, & Santiago, 2001). It also includes information from non-funded comparison communities. The number of grantees, participants, components, and methodologies incorporated into the evaluation make it the most extensive study ever undertaken of a children's mental health services initiative. The core components of the evaluation, as mandated by the legislation, are briefly described below.
The evaluation includes multiple cohorts of grantees funded over the period 1993 to 2000. Figure 3 illustrates the sequencing of cohorts and the number of grantees in each cohort. It also shows the sequencing of the two longitudinal comparison studies and the evidence-based practices study, and the number of grant-funded and non-funded communities included in each study.
In addition to the core components of the study described above, other studies have been added to the evaluation as additional questions about effectiveness have emerged over time. The Federation of Families for Children's Mental Health has collaborated with the national evaluation team at ORC Macro to conduct a family-driven research study that examines how youth and family engagement in a system of care affect child and family outcomes. A sustainability study was designed to assess the potential of funded grant communities to sustain their systems of care beyond their Federal grant period and to provide information that will be useful to Federal and State policymakers as well as local systems of care to enhance sustainability.
The remainder of this report presents evaluation findings as of FY 2001. Findings from the evaluation of the first cohort of grant communities, funded in 1993 and 1994, have been presented in previous Reports to Congress (CMHS, 1996, 1997, 1998, 1999, 2000). This report includes findings from the cohorts funded in 1997, 1998, and 1999 with the exception of findings from the first longitudinal comparison study, which was conducted in grant communities funded in 1993-94. Chapter II presents findings from the system-of-care assessments. Chapter III provides an in-depth analysis of the Phase I comparison study conducted from 1997 to 2000, including overview, methods, and results. Chapter IV provides the most recent findings from the descriptive and outcomes studies and preliminary findings from the second comparison study that began in 1999. Chapter V presents a progress report on the Phase II comparison study, which is being conducted in communities funded in 1997. Preliminary results from the special studies currently being conducted are found in Chapter VI. Chapter VII presents the activities of partners affiliated with the Comprehensive Community Mental Health Services for Children and Their Families Program. References are found in Chapter VIII.
|
| Home | Contact Us | About Us | Awards | Accessibility | Privacy and Disclaimer Statement | Site Map |
