SAMHSA's National Mental Health Information Center

This Web site is a component of the SAMHSA Health Information Network

Online Publications

Order Publications

National Library of Medicine

National Academies Press

Publications Homepage

printer friendly page

e-mail this page

bookmark this page

shopping cart

current or new account

This Web site is a component of the SAMHSA Health Information Network.

2000 Annual Report to Congress on the Evaluation
of the Comprehensive Community Mental Health
Services for Children and Their Families Program

Home | Summary | Table of Contents | Figures | Tables | I | II | III | IV | V | VI | VII | VIII | IX | Appendix

I. INTRODUCTION AND PROGRAM BACKGROUND

INTRODUCTION

The first Surgeon General's report, Mental Health: A Report of the Surgeon General (U.S. Department of Health and Human Services [DHHS], 1999), is a landmark publication detailing the significant resources that have been devoted to the research and development of mental health services over the past several decades. The report confirmed that approximately 20 percent or 1 in 5 children are affected at any one point in time by a mental health disorder. Although much progress has been made in addressing the mental health needs of children and their families, they continue to be an underserved population, and many questions remain regarding the development of effective community-based mental health services for youth. These questions span a wide range of issues, including such areas as the accurate identification of children and families in need, the integration of evidence-based practices into community mental health services, the efficient financing of services, and the reduction of the stigma associated with mental health disorders among children and their families.

Mental Health: A Report of the Surgeon General (hereinafter called the SGR), in part, reflects a continuing national response to the needs of the estimated 4.5 to 6.3 million children in the United States who have a serious emotional disturbance (Friedman, Katz-Leavy, Manderscheid, & Sondheimer, 1999). Prior to the development of a system-of-care-approach, these children were typically underserved or served inappropriately by a fragmented mental health system. In response to these findings, Federal leadership, along with a growing family movement, began to emerge and promote a new paradigm for serving these children and their families. Since first articulated by Stroul and Friedman in 1986, this system-of-care approach has evolved into the principal organizing framework shaping the development and delivery of community-based children's mental health services in the United States. Hallmarks of this approach include the following:

The mental health service system is driven by the needs and preferences of the child and family using a strengths-based, rather than deficit-based, perspective.
Family involvement is integrated into all aspects of service planning and delivery.
The locus and management of services are built upon multi-agency collaboration and grounded in a strong community base.
A broad array of services and supports is provided in an individualized, flexible, coordinated manner, and emphasizes treatment in the least restrictive, most appropriate setting.
The services offered, the agencies participating, and the programs generated are responsive to the cultural context and characteristics of the populations that are served.

The Center for Mental Health Services (CMHS) within the Substance Abuse and Mental Health Services Administration (SAMHSA) has had the primary responsibility for translating this framework into a program of services and supports that now exists in 67 grant communities around the country. Beginning in 1986 with a funding mechanism to support the creation of a service system infrastructure called the Child and Adolescent Service System Program (CASSP) and culminating with the passage of the Children's and Communities Mental Health Services Improvement Act in 1992, CMHS has championed the development of community-based, family-focused systems of care. Funding for these systems of care is provided through CMHS's Comprehensive Community Mental Health Services for Children and Their Families Program.

The SGR provides a context for measuring the progress that is made in the next several years on improving the effectiveness of children's mental health services. The report highlights the valid and reliable diagnostic systems and empirically supported treatments found effective for specific childhood mental health disorders. Evidence from these efficacy trials indicates that mental health treatment delivered in clinical settings can have a significant positive impact on children and families when compared to control groups. These results, however, stand in contrast to the more limited information available to document the differential effectiveness of interventions in community settings. A major concern in the field of mental health services research is the feasibility and suitability of implementing and evaluating evidence-based practices in complex community systems where rigorous methods and controls can be compromised. Despite these challenges, the SGR calls for further research to focus on the effectiveness of evidence-based practices in fully integrated community settings such as the CMHS-funded systems of care.

THE COMPREHENSIVE COMMUNITY MENTAL HEALTH SERVICES FOR CHILDREN AND THEIR FAMILIES PROGRAM

The Comprehensive Community Mental Health Services for Children and Their Families Program, now in its seventh year, provides grants to States, communities, territories, and American Indian tribes to improve and expand their systems of care to meet the needs of children and adolescents with serious emotional disturbance and their families. These include children and youth with a serious emotional disturbance from birth to age 21 who currently have, or at any time during the past year had, a mental, behavioral, or emotional disorder of sufficient duration to meet diagnostic criteria specified in the Diagnostic and Statistical Manual for Mental Disorders, Fourth Edition (DSM-IV; American Psychiatric Association, 1994) that resulted in functional impairment that substantially interferes with or limits one or more major life activities. Administered by CMHS's Child, Adolescent and Family Branch, funding for the initiative has grown from an initial $5 million in its first year to a cumulative level of $464 million ($83 million in FY 2000). From FY 1993 to FY 2000, a total of 67 grants were awarded for 5 years each ( see Figure 1 and Table 1).

THE NATIONAL EVALUATION OF SYSTEMS OF CARE

The Comprehensive Community Mental Health Services for Children and Their Families Program represents the largest Federal investment ever in the development of community-based mental health services for children and their families. This investment has resulted in the rapid proliferation of the system-of-care model within a swiftly changing social and political landscape. Strong evaluation components at both the individual grant community and the national level have undergirded the program since its inception.

CMHS's commitment to promoting knowledge development and application has resulted in a national, cross-site evaluation of the program that begun in 1994. The evaluation is conducted by ORC Macro (formerly Macro International Inc.) and its partners, the Louis de la Parte Florida Mental Health Institute of the University of South Florida and the Federation of Families for Children's Mental Health.

The evaluation is designed to maintain accountability and to address the pressing need for information upon which to base programmatic, funding, and policy decisions. It also responds to the mandate in the authorizing legislation to conduct an annual evaluation of the program's effectiveness. A basic requirement of the mandate is to describe the children and families served by the system-of-care initiative. Other requirements include an assessment of how systems of care develop, what factors impede or enhance their development, whether children served by the program experience improvement in clinical and functional outcomes, whether improvements endure over time, and whether consumers are satisfied with the services they receive. Besides responding to the original mandate in the authorizing legislation, the evaluation provides additional information upon which to shape the delivery of mental health services to the Nation's children. Government Performance and Results Act (GPRA) measures set for this program include increases in collaboration among child-serving agencies, stability of children's living arrangements and family satisfaction, decreases in the use of inpatient or residential services, and improvement in child clinical and functional outcomes as indicators of program effectiveness. Table 38, near the end of this Report, provides a summary of program performance according to the indicators established for GPRA measures. Information about program performance on these measures is presented in detail throughout the 2000 Annual Report to Congress.

While the SGR provides documentation on the growing knowledge base about children with serious emotional disturbance and about system-of-care services and treatments, many important questions remain. For example, understanding of child and adolescent mental health disorders has significantly improved since the establishment of standard diagnostic categories. Nevertheless, variation in how children with a diagnosable disorder display significant impairment in their day-to-day functioning has not been clearly ascertained. A clearer understanding of this is critical to accurate identification of children with serious emotional disorders. Perhaps even more important are the policy, services, and treatment implications that derive from an understanding of the impact of serious emotional disorders on child and adolescent functioning. Equally significant, but less well documented, are the implications of child mental health disorders on family functioning and well-being. In addition, there is a high rate of stability and persistence of serious emotional disorders that potentially has long-term effects in many domains of functioning, often persisting into adulthood (Burns, 1999). A better understanding of these long-term effects is necessary for developing appropriate services and supports that transition children and adolescents with serious emotional disturbance into an adult world.

Another area for further examination relates to the populations which systems of care serve. Some studies have provided data on racial or ethnic backgrounds, gender, specific diagnosis, and level of impairment about a population of youth frequently underserved and under-examined (Berndt et al., 1995; Cullinan, Epstein, & Sabornie, 1992; Landrum, Singh, Nemil, & Ellis, 1995; Rosenblatt & Attkisson, 1993; Silver, Unger, & Friedman, 1994). Other studies have made attempts to draw comparisons between characteristics found in the study sample and information about the community or population from which the sample was drawn (Rosenblatt & Attkisson, 1993; Santiago, Grosser, & Rejino, 1997). However, a complete understanding of whom systems of care serve or where referrals originate is lacking. Nor is there an adequate understanding of the unmet need or the degree to which children are underserved (Coutinho & Denny, 1996).

Finally, there are critical questions to be answered about the overall effectiveness of systems of care. The SGR concluded that while findings are encouraging, few studies in the children's mental health field have included comparison populations with long-term follow-up of multiple outcomes. Those that have been conducted have yielded results suggesting that systems of care are effective in achieving important system improvements (e.g., reduction of restrictive placements, increased satisfaction of parents) (Bickman et al., 1995; Bickman, Lambert, Andrade, & Peñaloza, 2000; Bickman, Noser, & Summerfelt,1999; Bickman, Summerfelt, Firth, & Douglas, 1997; Bickman, Summerfelt, & Noser, 1997). However, additional work must be done to determine whether greater improvements in children's clinical outcomes can be attributed to services delivered within a system of care compared to services delivered within more traditional systems. The SGR recommended that greater attention be paid to the relationship between changes at the system level and changes at the practice level.

The evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program provides a living laboratory for addressing many of the issues raised by the SGR. The evaluation provides rich descriptive information about children with serious emotional disorders and their families. It also assesses the children's clinical and functional outcomes, and it describes how services are organized and delivered within the comprehensive systems. Most critically, with the addition of an evidence-based practice substudy, the national evaluation enhances our understanding of community parameters that may facilitate or impede the effectiveness of evidence-based practices for children, adolescents, and their families.

EVALUATION DESIGN AND EVALUATION COMPONENTS

Underlying the design of the evaluation is a program theory model, which has evolved to explain how systems of care are intended to work, that is, a delineation of what must be done to achieve desired goals (see Figure 2). The model illustrates the connection between resources allocated to the program, program activities, and intended outcomes at the system, practice, and child and family levels. The system of care provides a fertile context for developing the infrastructure that supports program activities and integrated, community-based practices to produce successful child and family outcomes. The model depicts the intended relationship between funding systems of care, development of the systems, and demonstrations of their effectiveness and sustainability so that system reform is the ultimate outcome.

The evaluation is designed to provide an understanding of the implementation and development of the system of care as portrayed in the program theory model. As the model shows, it is also designed to provide feedback to continuously improve the quality of the program. Broadly speaking, evaluation findings provide

data on the mental health and other service needs of children and families,

data on the effectiveness of systems of care that serve children with mental health needs,

an increased understanding of the interactions between providers and families,

an increased understanding of implementation and development of systems of care,

information to guide service delivery approaches at the national and local levels, and

empirical information to help inform policy and funding decisions.

The evaluation design is complex and includes multiple components that involve a variety of quantitative and qualitative methods. Figure 3 illustrates the sequencing of the major components within the evaluation and shows the number of grant communities (and non-CMHS-funded comparison communities) that are included in special studies and analyses.

The major components of the evaluation are briefly described below.

System-of-care assessment study. This study examines whether systems of care have been implemented in accordance with the program theory and documents how systems develop over time to meet the needs of the children and families they serve.
Child and family descriptive study. This study describes the children enrolled in the systems of care in terms of their demographics, functional status, living arrangement, diagnosis, risk factors, and mental health service history. Family demographics, socioeconomic status, and composition are also described.

Child and family outcomes study. This study examines how the system affects child clinical and functional status and family life. For a subsample of families enrolled in the child and family descriptive study, outcomes data on child clinical and functional status are collected at intake and at subsequent intervals to assess change over time in the following areas: symptomatology, diagnosis, behavioral and emotional strengths, social functioning, substance abuse, school attendance and performance, delinquency and juvenile justice involvement, and stability of living arrangements.

Services and costs study. This study describes the types of services used by children and families, their utilization patterns, and the associated costs. The relationship between service use and outcomes is also explored.

Longitudinal comparison study. This study addresses the central question, "Can greater improvements in children's behavior and functioning be attributed to the system-of-care approach compared to a traditional service delivery approach?" It provides an understanding of what happens to children and families after services are terminated, which will increase the knowledge base about stability of service effects over time. Other areas for examination include educational outcomes; substance abuse; involvement with the juvenile justice system; and family life, family involvement in service delivery, and satisfaction with services. Substudies in grant communities funded in 1997-98 monitor system performance and assess whether children's and families' experiences with services and provider practices are in keeping with system-of-care principles.

Study of evidence-based practices in systems of care. This study assesses the effectiveness of evidence-based practices within systems of care. Findings from this study will allow for the determination of the additive contributions of focused, evidenced-based practices to the overall effectiveness of the systems of care.

Besides these core components, special studies were or are being conducted in each of the cohorts of the evaluation. In grant communities funded in 1993-94, a study on the effects of managed care assessed the impact of managed care initiatives on the development and operation of six local systems of care, and it identified aspects of the structure, design, organization, and characteristics of managed care systems that have facilitated or hindered mental health service delivery to children with serious emotional disturbance and their families. This study was completed in September 1999. A second special study, called a family scan, was conducted with the Federation of Families for Children's Mental Health. This study used survey methods to examine how extensively families are involved in evaluating systems of care. Findings delineated the roles family members perform in the national evaluation and tapped the perspectives of three major groups: families, project directors, and evaluators at the local sites.

Additional studies were conducted in the non-CMHS-funded comparison communities in the longitudinal comparison studies to examine practice-level parameters and cultural factors that may influence the impact of systems of care. The first involved family-centered interviews to compare information from families and their providers about the degree to which funded systems of care and traditional service systems address and work to meet the needs of individual children and families. The second study used an ethnographic approach to develop an understanding of the cultural factors associated with raising a child with a serious emotional disturbance and utilizing the service delivery system in an inner city.

In grant communities funded in 1999-2000, a family-driven research study, conducted by the Federation of Families for Children's Mental Health in collaboration with ORC Macro, will help illuminate how families are experiencing systems of care as implemented by the grant communities. The research questions will be evolved through a Delphi process with families participating in systems of care. Conducting a "family driven" study such as this is a new concept for the children's mental health community and, as such, demonstrates how families and researchers can work together to investigate questions of significance in ways that are scientifically sound and responsive to family needs and priorities.

CURRENT STATUS OF THE EVALUATION

The evaluation of grant communities funded in 1993-94 concluded in December 1999. This report includes combined descriptive data from grant communities funded in 1993-94 and 1997-98, final outcomes data from the grant communities funded in 1993-94, and preliminary outcomes data from grant communities funded in 1997-98. Two longitudinal comparison studies have been undertaken. In the grant communities funded in 1993-94, the Phase I comparison study was implemented in FY 1997 and completed in December 2000. Preliminary data are included in this report. The Phase II longitudinal comparison study, which is being conducted in grant communities funded in 1997, will be completed in FY 2003. Data describing the children and families enrolled to date are included in this report. The evidence-based practice study began in FY 2000. The work group that oversees the study has begun to evaluate potential evidence-based practices and has conducted a survey of the grant communities to determine the feasibility of implementing evidence-based practices in each grant community. The family-driven research project also began in FY 2000. The research team has met to discuss study design and has initiated a Delphi process to determine study questions.

During 1999 and 2000, the national evaluation team visited each of the 14 grant communities funded in 1998 to establish relationships and to conduct training on data collection. Initial system-of-care assessment visits were made to 12 of these 14 grant communities, and a second round of system-of-care assessment visits was made to the nine grant communities funded in 1997. In May 2000 the Federal Office of Management and Budget approved the design and instrument package for the evaluation of grant communities funded in 1999-2000, and an evaluation and technical assistance needs assessment was conducted with 19 of these 22 grant communities. The national evaluation team has established relationships with all of the grant communities funded in 1999-2000 and has initiated data collection training visits. ORC Macro developed a customized software package available to each grant community for their local data entry and transmission to a central Web site to ease data management and to enhance the accuracy of longitudinal analyses. Most grant communities devote the first year of the grant funding period to the development or refinement of system-of-care services, hiring, and other start-up activities.

ABOUT THIS REPORT

The remainder of this report presents the FY 2000 evaluation activities and findings. Chapter II presents system-of-care assessment findings from grant communities funded in 1997-98. Chapter III presents aggregate child and family descriptive data from grant communities funded in 1993-94 and 1997-98, outcomes data from grant communities funded in 1993-94, and baseline and 6-month data from grant communities funded in 1997-98. Chapter IV presents preliminary results from the Phase I comparison study and provides preliminary results from the Phase II comparison study. Chapter V presents information about services and costs. Chapter VI profiles the graduating grant communities, Chapter VII highlights programmatic and evaluation features of the American Indian grant communities, and Chapter VIII profiles CMHS-funded activities in support of the services program.

Table of Contents | Previous | Next