This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
2000 Annual Report to Congress on the Evaluation
|
|
CHAPTER SUMMARY In summary:
|
A significant aspect of the national evaluation is the longitudinal comparison study. The purpose of the comparison study is to answer the primary question: "Do children with behavioral and functional impairments improve more when served with a system-of-care approach compared to services as usual?" A multi-method approach is being employed to evaluate outcomes, system characteristics, services delivered, and family service experiences. The results of the comparison study will contribute to determining whether CMHS-funded systems of care lead to better outcomes for children and families than receiving services as usual.
As the previous chapters have demonstrated, the Comprehensive Community Mental Health Services for Children and Their Families Program has been effective in achieving significant changes in system structure and function and child and family outcomes. However, the relationship between changes in the structure and function of a system of care and changes in clinical outcomes for children remains unclear. Findings from the Fort Bragg Evaluation Project indicated that services delivered through a continuum of care did not produce significantly better clinical outcomes than traditional services for military dependents (Bickman et al., 1995). Access to services was greater in the demonstration site, with accompanying increases in costs. This study was limited to a military population, however, and its methods of integrating and individualizing services were ambiguous. Therefore, the findings may not generalize to other populations, especially in managed care environments practicing cost-containment strategies. A subsequent investigation utilized a randomized control group design to evaluate the effectiveness of system-of-care services for children with serious emotional disturbance and their families (Bickman, Summerfelt, Firth, et al., 1997; Bickman, Summerfelt, & Noser, 1997; Bickman et al., 1999). This latter effort also found no significant clinical and functional differences between children served in a system of care and those who received treatment as usual. However, the children enrolled in both the experimental and control conditions were found to receive very similar service delivery and planning mechanisms in a State greatly influenced by system-of-care principles. In the face of these findings, the SGR concluded that while results of several studies on the effectiveness of systems of care seem promising, more research is necessary to determine conclusively whether changes at the system level will result in better clinical outcomes for children in systems of care than for children in more traditional service systems.
The Phase I comparison study involved selection of three systems of care from among the 22 communities funded in 1993-94 and three matched comparison communities delivering services as usual. Selection of the three comparison study pairs was initiated in the summer of 1997. The Phase II comparison study involved the selection of two additional systems of care from among the 9 communities funded in 1997 and their matched comparison communities. These two comparison pairs were selected in the fall of 1998. In each case, system-of-care communities were matched with non-CMHS-funded communities that used a different approach to serving children with mental health problems. All comparison study communities were selected based on the following criteria:
The final selection criteria for comparison study communities was willingness to participate. A few potential community participants were unwilling to participate because of other planned activities during the proposed study period. Based on these criteria, the matching pairs listed in Table 9 were selected for the comparison studies. The selection of multiple comparison pairs provides the opportunity to examine multiple replications of the comparison study design with pairs that have different geographical and demographic characteristics.
The Stark County system of care has been refining its children's mental health care delivery system over many years since an initial Child and Adolescent Service System Program (CASSP) grant in the 1980s. The Stark County Family Council (SCFC) is the governing body that organizes the service delivery system for all children and families in need of mental health services. The SCFC structure was developed in the mid-1980s as a result of a State-mandated effort to develop a more collaborative, interagency approach to providing services. The SCFC has a long history of multi-agency collaboration and thus has made possible the infusion of system-of-care principles. With the award of Federal grant funds, the SCFC was able to implement a full-scale system of care successfully. Mental health services have expanded to include community-based case management and mental health therapy, which are provided by staff stationed in schools and child welfare offices.
In Mahoning County, children were recruited from a private agency in Youngstown. The agency is a traditional outpatient facility that provides office-based, as opposed to community-based, mental health services. Referrals come from a variety of child-serving agencies and organizations, including hospitals, homes for battered and abused women, foster care agencies, primary care physicians, and more traditional institutions, including juvenile justice and child welfare.
The East and West Baltimore neighborhoods offered the opportunity for a comparison within an inner-city urban setting. The system-of-care community in East Baltimore is a neighborhood-based project that limits its catchment area to specific ZIP Codes. The program contracts for services with Johns Hopkins Hospital, and the program director and staff are Johns Hopkins Hospital employees. The East Baltimore program serves children from Baltimore city school districts, the juvenile justice system, child protective and placement agencies, and other mental health providers. East Baltimore has liaison staff stationed in the different child-serving agencies and schools who refer children to the program. Through the school-based component, children and families receive direct clinical services, educational support, and therapeutic after-school and summer programs. A Family Resource Coordination Unit provides families with intensive outreach services. "Neighborhood liaisons" provide individual and family therapy, emergency support services, case management, and individualized wraparound services that are provided in the home and in other community settings.
Children in West Baltimore are served by a variety of private and public agencies. No one centralized agency serves children throughout the entire West Baltimore catchment area, so the comparison study enrolled children through different agencies, including juvenile justice, child protective services, schools, and a children's mental health provider. Children are served primarily within the agency that identifies the child's behavioral problems, with little coordination across agencies. The service providers primarily serve children only in the West Baltimore neighborhoods, which minimizes potential contamination effects of the East Baltimore system-of-care initiative.
The Santa Cruz County system of care is operated by the county mental health agency. Children are referred to the system of care from a variety of community organizations: schools, the county juvenile justice system, the county child placement agency, and a private substance abuse treatment agency. Services are provided by one of several teams that have fashioned a set of services and supports to meet a particular population's needs. However, the Santa Cruz County system of care places special emphasis on serving children from the juvenile justice system. Two key services are the residential facility for court wards and the juvenile probation program to bring court wards home or divert placement. Besides these services, Santa Cruz offers a comprehensive array of services to children with serious emotional disturbance, including outpatient therapy, case management/care coordination, crisis intervention, special education, reunification support, wraparound, family support, and respite.
Because California was moving toward a statewide adoption of the system-of-care service delivery approach during Phase I community selection, finding a non-system-of-care community for the Santa Cruz County system of care within California that was using a different service delivery approach was not possible. Travis County, Texas, was selected as the non-system-of-care community because it had similar demographic characteristics. In particular, both counties have a large Hispanic population, which adds to the diversity of populations included in the comparison study as a whole.
The participating agency in Travis County is the county children's mental health agency. Children served in Travis County also come from a variety of sources: the county juvenile justice system, the State child protective agency, and the education system. Access to Travis County children's mental health services is through a single point of entry with a toll-free telephone number. Travis County provides services such as assessment/evaluation, medication support, case management, mobile crisis outreach, in-office young children's groups, individual and group therapy, and transportation.
A complicating factor in this comparison pair is the awarding of a CMHS system-of-care grant to Travis County, Texas, in October 1998. This may dilute the pre-existing differences in infrastructure and service delivery between Santa Cruz and Travis County across time. However, the existence of a matched non-system-of-care community in Travis County prior to CMHS funding offers the opportunity in the future to compare changes in service delivery and outcomes prior to grant initiation to the outcomes that are obtained after a system of care is fully in place.
This comparison pair offers a unique opportunity to examine the impact of services within a rural State where the system-of-care and non-system-of-care communities each span 22-county areas of approximately 15,000 square miles. The Central Nebraska Initiative for Families and Youth (CeNIFFY), housed at the Human Services Region III Behavioral Health office in Kearney, is one of nine grant communities that received CMHS funding in the fall of 1997. Referrals to the Region III office come from the departments of parole and probation, schools, the Department of Health and Human Resources, self-referrals, and referrals from other child-serving agencies. A unique aspect of the project is the Professional Partnership Program (PPP). Although this is a statewide program specifically designed to serve children and youth with serious emotional disturbance who are at risk for out-of-community placement, in Region III the ability of this program to serve children has been enhanced by the grant program. PPP staff function as case managers who use a "wraparound" process to assist families in building a team dedicated to flexible, nontraditional, and unconditional care. Available wraparound services include respite, transportation assistance, and recreational therapy. Additional services include outpatient, intensive inpatient, acute inpatient, day treatment, psychiatric residential treatment, and multisystemic therapy.
Human Services Region IV Behavioral Health encompasses four service delivery centers located in Sioux City, Columbus, Norfolk, and O'Neill in northeastern Nebraska. In this service delivery system, children are referred directly to the mental health department and a large number of contracted private providers throughout the 22-county service area. The majority of referrals to mental health services come from child protective services or juvenile justice agencies or are self-referrals. Available outpatient services include individual, couples, family, and group counseling; psychological and psychiatric evaluations and consultation; pre-admission screens; aftercare; emergency services; and medication evaluations. Substance abuse services, home-based therapy, and therapeutic foster care also are provided. For children with serious emotional disturbance, the region has limited infrastructure and resources for providing services locally; consequently, many children with serious emotional disturbance become wards of the State and are sent to residential facilities outside the region. Children and families have been recruited into the study from two county-based agencies located in different cities and from several private counseling offices located throughout the region.
In Jefferson County, the system-of-care community, the Jefferson-Blount-St. Clair Mental Health and Mental Retardation Authority is the grant recipient. The Jefferson County Community Partnership (JCCP), the governing body of the grant program, brings together the grant recipient with other community agencies in the COPE (Children Overcoming Problems Everyday) program. A range of services is available through JCCP, including assessment, case management, crisis follow-up, liaison or collateral services, in-home services, supportive therapy for foster parents, respite beds, and case management for transition into adulthood. Outpatient, day treatment, inpatient, and emergency services are obtained through contractual arrangements with other community providers. Children and families enter JCCP services through school-based diagnostic and evaluation units located in two regular middle schools, two alternative middle schools, Family Court, and the Department of Human Resources. Diagnostic and evaluation specialists conduct intake and assessment and refer children to appropriate COPE teams stationed in these locations. Each COPE team includes a combination of mental health liaisons, diagnostic and evaluation specialists, case managers, outpatient therapists, substance abuse counselors, domestic violence counselors, psychiatric consultants, in-home services teams, and parent advocates. These teams work collaboratively with partnering agencies to ensure children and families receive needed services.
In Montgomery County, the non-system-of-care community, children are recruited from a similar county-based agency, the Montgomery Mental Health Authority, that serves a four-county area. Mental health services are coordinated through one main center and two satellite centers. All children requiring mental health services are referred directly to the Montgomery Mental Health Authority. Outpatient mental health services are provided on-site by the outpatient therapy team. The service array provided through the center includes emergency/crisis management, therapy, case management, in-home intervention, in-home therapy, court liaison, pre-hospitalization screening, psychiatric services, medication monitoring, and diagnostic and evaluation testing.
In both comparison studies, children participate in the study for up to 2 years during and after their service experience. Children and families participating in the comparison study present for mental health services or have been referred by participating agencies such as juvenile justice, school systems, and child welfare.
In the system-of-care communities, children enrolled into services have serious emotional problems according to the CMHS grant guidelines. Thus, all children receiving services in the system-of-care communities are eligible for the study. In the non-system-of-care communities, no such formal guidelines regarding severity of problems exist. As a result, children in these communities are screened through an eligibility process to ensure only children with serious emotional problems are enrolled into the study in an effort to equate the comparison study samples. In addition, one of the following four selection criteria for determining severity has to be met:
Although a clinical or functional assessment score above the clinical range would be enough to determine severity, multiple criteria are used because information on any one criterion is not consistently available across communities. Besides these criteria, children had to be 6-17.5 years old and the child and caregiver had to be willing to participate. All children and families who met the above criteria were selected until sample size was reached or the enrollment period had ended.
Many components of the comparison studies are the same as those in the overall national evaluation. The primary component of the comparison study is the interviews conducted with children and caregivers about child and family outcomes. Once entered into the study, all children and families participate in a baseline intake interview and follow-up interviews at 6-month intervals for up to 2 years. Field staff in each community interview families in the convenience of their homes regarding child and family behavioral and functional issues. System-of-care assessments are used to evaluate the development of service delivery systems. Evaluators make site visits to each community to assess the system's infrastructure and service delivery practices compared to system-of-care principles. Management information system data are obtained from mental health agency databases to evaluate service use and service costs across time.
The Phase I comparison study also expanded upon the national evaluation with smaller study components that provided additional data about families' experiences. In each community, a sample of families and their primary service providers was selected to participate in system-of-care practice reviews to evaluate how the service system was meeting the individual needs of children and families. The perspectives of caregivers, children, and their primary providers were combined with record reviews to address this question. In selected communities, samples of families were also selected to participate in ethnographic interviews to assess cultural factors that may impact upon the provision of care.
The Phase II comparison study design includes several modifications. To ensure sample size is sufficient, a sample of 225 children is being recruited in each community. In addition, a practice-level assessment component has been added to understand better the interaction between service providers and the children and families they serve. Service diaries will be completed by selected families and providers to evaluate practice-level interactions. Questions to be answered by the practice-level assessment include: Are planned services actually received? What factors contribute to adherence to service plans? Are children's and families' experiences consistent with the system-of-care philosophy? Are providers' attitudes and practices consistent with the system-of-care philosophy? The Phase II comparison study also differs in that system-of-care practice reviews and ethnographies will not be conducted, given budgetary constraints and the relatively high cost of these intensive qualitative data collection approaches to practice-level assessment.
Prior to the start of child enrollment for the Phase I comparison study initiated in August 1997, enrollment goals were set based on two primary criteria: (a) the number of study participants needed to generalize results from this study to other populations of children with severe behavioral and emotional disorders, and (b) the capacity of local mental health agencies to enroll children into services. Given these criteria, an enrollment goal of 1,100 children was set for the group of six study communities, and individual community enrollment goals were set at 150 to 200 children depending upon the local agency's enrollment capacity.
Through July 31, 2000, a total of 1,042 children and caregivers have been interviewed at intake for the comparison study (see Table 10). Enrollment of children and families ended in January 2000, but follow-up interviewing will continue through the end of the study in December 2000 (ensuring the opportunity for completion of 12-month follow-up interviews with children and families enrolled most recently). The 6-month wave of interviews, which included 869 families, was completed in June 2000. Through July 2000, 777 families were interviewed for the 12-month wave, 562 families for the 18-month wave, and 355 families for the 24-month wave. Because enrollment ended in January 2000, not every family will be eligible to complete their 18-month and 24-month interviews. At the end of December 2000, 86 percent of families will have been approached for their 18-month interview and 63 percent of families will have been approached for their 24-month interview.
Besides the number of children and families initially enrolled into the study, the number of families retained in the study over time can also affect the representativeness of the participants and the ability to generalize findings to a wider array of communities. Although all caregivers and youth voluntarily consented to participate in the study at the time of enrollment, some families decided not to participate in the ensuing waves of interviews. Additional families could not be located or contacted. Families missing two consecutive interviews were withdrawn from the study.
The overall data completion rate was strong compared to other longitudinal studies:
|
Overall, family retention rates were high (see Table 11). Through July 2000, 86 percent of families had been retained in the study. Retention rates varied somewhat from 75 to 93 percent across different communities; however, the overall retention rate is high compared to other longitudinal studies. Although total data completion rates across communities ranged from 73 to 84 percent, the overall data completion rate of 80 percent across the four follow-up waves of data collection was comparable to or higher than similar studies. For example, in the Fort Bragg Evaluation Project, completion rates at 18 months ranged from 65 to 81 percent for key outcomes measures (Hamner, Lambert, & Bickman, 1997). In the Stark County study, Bickman, Summerfelt, and Noser (1997) reported a data completion rate of 76 percent at the 6-month follow-up interview. Authors of the Great Smoky Mountain Study (Angold, Costello, Burns, Erkanli, & Farmer, 2000) reported a 70 percent completion rate across four data collection waves in the Great Smoky Mountains Study.
All data analyses presented below for the Phase I comparison study include interview data collected before April 2000 in all communities.2 Follow-up data collection continues for the 12-, 18-, and 24-month data collection waves. The analyses that follow include all 1,042 children and families who completed intake interviews at intake and 864 children and families who completed their 6-month interviews.
To address the central question of this evaluation, "Are greater improvements in children's behavior and functioning associated with a system-of-care approach?" it is critical to know the degree of system development in the CMHS-funded system-of-care communities as well as in the matched comparison communities. Preliminary discussions with agency administrators and directors during the community selection process indicated that non-system-of-care communities were using approaches to serving children that were different from the system-of-care approach. However, children's mental health service delivery approaches can change over the course of a 4-year study. In addition, understanding the differences in service delivery systems can help to explain how children and families benefit from services. Before examining child and family outcomes, the question of whether service systems are different must be addressed so the assessment of whether one system is more effective than another can be made. The system-of-care assessment and the system-of-care practice reviews examined the differences between service delivery approaches at the system level and the family level.
The system-of-care assessment was designed to assess the extent to which the comparison study communities embody system-of-care principles. The assessment was conducted using an instrument which is organized in terms of the generic components of any service system (e.g., governance, entry into the system, service planning) to allow comparable assessments of both CMHS-funded and non-CMHS-funded communities. Development of the system-of-care assessment is described in more detail in Chapter II, "System-of-Care Implementation."
Data for the assessments was obtained through semistructured interviews with key stakeholders such as grant principal investigators, program directors, case managers, counselors, and family members. A review of background information provided by the communities, such as staffing arrangements, program goals and plans, and budget information, was also used.
Results of the assessment of the Phase I comparison study communities were presented in the Annual Report to Congress on the Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program, 1999 (CMHS, 1999). In summary, comparisons across all of the system-of-care communities and non-system-of-care communities revealed the following trends:
Assessment of service experience at the practice level was included to determine whether system-of-care principles were being directly expressed in practice-level interactions among service delivery personnel and families. System-of-care principles may be fully implemented at the overall community and participating agency levels, but incomplete infusion of these principles into practice will limit the effectiveness of system-level reform. For example, in CMHS-funded communities, if individual providers deliver services utilizing an approach that is not consistent with system-of-care principles, then outcomes can only be attributed to the specifics of the individual provider's approach rather than the system-of-care model. Conversely, in matched comparison communities, system-of-care principles may be adopted by individual providers and directly influence the services that are provided, but they may not be characteristics driving development at the overall community and specific agency levels. An assessment of service experiences at the individual child and family level provides an opportunity to assess differences between system-of-care and matched comparison communities and to understand how service experiences may influence the relationships between service delivery systems and outcomes.
The system-of-care practice review (SOCPR) portion of the comparison study was conducted under subcontract to the Child and Family Studies Department at the Louis de la Parte Florida Mental Health Institute of the University of South Florida, an active collaborator and partner in the national evaluation since its inception in 1994. The SOCPR has been referred to previously as the Family-Centered Interview in earlier versions of this report (c.f., CMHS, 1999). The case study approach used to evaluate service experiences for this study (Hernandez, Gomez, Lipien, Greenbaum, Armstrong, et al., in press) provided both qualitative and quantitative information about families' experiences of services provided as well as the needs and strengths of the children and families served.
Samples of families were selected from system-of-care and matched comparison communities for participation in system-of-care practice reviews. The protocol for each participating family consisted of multiple data collection components, including a case record review, primary caregiver interview, child interview, provider interviews, and "informal helper" interviews (e.g., with a relative who may have been living with the family). Interviews and case record reviews were conducted by data collection teams who typically conducted an extensive 1- to 2-week data collection site visit to obtain information for all participating families. Information was combined across the case record review and interviews. The overall objectives and their underlying measurement areas are listed in Table 12.
The results of the system-of-care practice reviews conducted in the Stark County-Youngstown and the Santa Cruz County-Travis County (Austin) comparison pairs were summarized in the Annual Report to Congress on the Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program, 1999 (CMHS, 1999). In summary, those results indicated that the service experiences of families were more consistent with system-of-care objectives in the CMHS-funded systems of care than in the matched comparison communities. The results for the East Baltimore-West Baltimore comparison pair are presented below.
System-of-care practice reviews were conducted in the East Baltimore system-of-care community and the West Baltimore non-system-of-care community during September-October 1999. Eleven families participated in East Baltimore, and 12 families participated in West Baltimore. Figure 61 displays the results for these two communities across the four major objectives of the SOCPR. Results indicated significantly higher levels of implementation for the key system-of-care principles (i.e., child centered and family focused and community based) at the service experiences level in the East Baltimore system-of-care community. Differences between communities on the objectives of cultural competence and impact were not significant. Within the objectives, significant differences were observed for the measurement areas of individualized services (t = -5.26, df = 21, p < .001), full participation of the family in the service process (t = -2.96, df = 14, p = .005), case management (t = -2.26, df = 21, p = .018), access to services (t = -3.57, df = 12, p = .002), restrictiveness of services (t = -2.19, df = 20, p = .021), integration and coordination of services (t = -3.45, df = 21, p = .001), and agency culture (t = -2.56, df = 21, p = .009). In each case, the differences supported greater implementation in the East Baltimore system-of-care community. These results indicate that the service experiences of families are more consistent with system-of-care principles in the CMHS-funded system than in the matched comparison community.
Results of system-of-care practice reviews in all three pairs of communities indicated that the service experiences of families were more consistent with system-of-care objectives in the CMHS-funded systems of care than in the matched comparison communities. |
Combined findings from the system-of-care assessments and the system-of-care practice reviews indicate that the grant initiative differentially influences system development across the communities and, furthermore, systems of care may be having a significant impact at the level of service provision. A report is being prepared that summarizes results of the system-of-care practice reviews across the three system-of-care communities and compares them with results across the three matched comparison communities. The summary report will provide a comprehensive picture of the expression of system-of-care principles at the services level within the Phase I comparison study.
One of the challenges in comparing the effects of different service delivery approaches (e.g., system of care vs. non-system of care) is minimizing other influences that could account for differences in child and family outcomes such as demographic and behavioral characteristics. For instance, different outcomes might occur for older children who have more serious behavioral problems than younger children. Examining the demographic and behavioral characteristics of children and families in the comparison study communities will assist in addressing these issues and accounting for differences in data analyses. Results are presented for each comparison pair separately.
A comparison of basic demographic characteristics collected at intake revealed that children enrolled into the study are similar in two of the three pairs of communities in terms of their gender, age, and stability of previous living situations (see Table 13). About two-thirds of children enrolled into the study were males, and children's gender was not significantly different among any of the three pairs. The ages of children in the Stark County-Youngstown pair were not significantly different, nor were there significant differences in the East Baltimore-West Baltimore pair. However, children in Santa Cruz County were significantly older than children in Travis County. Half of the children in Santa Cruz County were 15-17 years old when enrolled into the study, while only 11 percent of children in Travis County were in that age range. This difference in ages reflects, in part, the difference in referral sources between the two communities. Santa Cruz County placed special emphasis on serving children referred from juvenile justice. Travis County had no special emphasis on a particular referral source and served children referred from a variety of sources.
The samples of children selected from each of the communities represent an ethnically and racially diverse group. |
The purposive selection of communities helped to ensure an ethnically and racially diverse group of children in the study. Forty-two percent of the 1,042 children enrolled are African-American, 36 percent are White, 11 percent are Hispanic or Latino, and another 8 percent have a mixed racial and ethnic background. However, the objective of enrolling children with similar racial and ethnic backgrounds in paired communities was met with only partial success. While children enrolled from the East and West Baltimore communities are almost all African-American, children enrolled from the other two pairs differ significantly in race and ethnicity. Children in the Youngstown and Travis County communities are more likely to be African-American, whereas children in the Stark County and Santa Cruz County communities are more likely to be White. However, similar proportions of children with Hispanic backgrounds were successfully enrolled in Santa Cruz and Travis Counties.
The stability of the child's living situation in the 6 months prior to intake was examined. Caregivers were asked to report how many places their child had lived in the past 6 months, and children were grouped into categories based on their total number of living situations. About two-thirds of children in the Stark County and Youngstown communities lived in only one place during the 6-month period leading up to intake into services. Many children in East and West Baltimore (80 percent and 75 percent respectively) also had only one living arrangement in the 6 months leading up to intake. While children in Travis County also had fairly stable living situations, 32 percent of children in Santa Cruz County had two placements during the 6 months prior to intake and another 20 percent had three placements during that period, a significantly greater proportion than in Travis County, where 26 percent of children had two placements and only 7 percent had three placements during the same period.
The demographic characteristics of the caregivers and families participating in the Phase I comparison study were also examined at intake (see Table 14). Caregivers who had spent the most time with the child in the past 6 months were sought as interview respondents. As a result, biological mothers were the most frequent respondents, and 91.6 percent of all caregiver respondents across the six communities were female. About 5 to 10 percent of respondents in each community were the children's biological fathers. Within each pair, the relationship of the caregiver respondents to the children was fairly similar. However, in East and West Baltimore, twice as many grandmothers and "other" family and friends were respondents compared to the other two pairs. When comparing the percentage of married and "living as married" caregivers versus single caregivers within each pair, no significant differences were found. When comparing across pairs, caregivers who responded in the two Baltimore communities were less likely to be married or living as married than respondents from other pairs.
In regard to caregiver respondents' ability to support their children financially, no consistent pattern existed across the six communities or within the pairs of communities. Respondents in West Baltimore and Santa Cruz County were more likely to have completed their high school education than the respondents in their paired communities. Caregivers who responded in Stark County, West Baltimore, and Santa Cruz County were less likely to be unemployed and more likely to have a higher family income than in their matched comparison communities. Overall, poverty was higher in all six communities compared to the nation as a whole. Fifty-nine percent of all families participating in the study made less than $15,000 per year. According to the 2000 U.S. Department of Health and Human Services poverty guidelines, a family of four with two children is living in poverty if their annual income is below $17,050. National figures indicate that 15.1 percent of all families with children under 18 lived in poverty in 1998 (U.S. Census Bureau, 1999b).
Similar to the outcomes study sample of the national evaluation, the primary behavioral and functional measures for the Phase I comparison study are the Child and Adolescent Functional Assessment Scale (CAFAS; Hodges, 1990b) and the Child Behavior Checklist (CBCL; Achenbach, 1991). The CAFAS was administered to provide a broad assessment of how children function in eight different life domains, and the CBCL provided parent reports of levels of emotional and behavioral symptoms.
Results from both the CAFAS and the CBCL administered at intake show that the typical child in the study has serious behavioral and emotional problems. The CAFAS criteria for marked or severe behavior is a score of 70 or higher. The mean CAFAS score for all children was 73.8. Children with a score of 63 or higher on the CBCL Total Problems scale are considered to fall in the clinical range, and the mean score of participants in the comparison study was 67.8.
Figure 62 presents children's mean level of functioning in each of the paired communities. Children in five of the six communities had average CAFAS scores in the marked or severe range. The mean score for children from West Baltimore was slightly below the marked/severe range at 66.3, but their mean score was not significantly different from the mean score of children in the East Baltimore community. Children from Santa Cruz County and Travis County were also comparable on the CAFAS. However, children enrolled from the Youngstown community had significantly more serious levels of functional impairment than the children from Stark County, even though both groups of children were rated within the marked/severe range.
Generally, caregivers of children in each of the six communities rated their children in the clinical range on the CBCL Total Problems scale (see Figure 63). While mean scores indicate similar levels of behavioral and emotional problems, the scores from one pair were found to be significantly different. Children from Travis County were rated on average with more serious clinical symptoms than children in Santa Cruz County.
A similar pattern of results was found on the Youth Self-Report (YSR; Achenbach & Edelbrock, 1987), the companion measure to the CBCL that is administered to youth aged 11 and older. The mean YSR Total Problems scale scores show that children reported similar non-clinical levels of symptomatology in the Stark County-Youngstown and East Baltimore-West Baltimore paired communities (see Figure 64). Children from the Travis County community, however, reported significantly higher mean levels of symptomatology than children from the Santa Cruz County community, though still below clinical range.
As Table 15 shows, most children in each individual community presented with serious mental health problems at intake. Although some variability existed across sites, a majority of children (53-74 percent) were rated within the ranges of marked or severe functional impairment on the CAFAS at intake. Although Santa Cruz County and Youngstown appeared to serve more children with marked or severe functional impairment than their matched communities, neither difference was statistically significant. Similarly, on the CBCL Total Problems and Externalizing scales, every community had a majority of children (62-79 percent) in the clinically significant range at intake. Travis County served significantly more children in the clinical range on both the CBCL Total Problems scale and the Externalizing scale than its matched system-of-care community in Santa Cruz County, but this is confounded with the fact that the two communities differed significantly in age. A cross-cultural study of the CBCL found that, consistently across cultures, parent report of externalizing problems decreased with increasing age of the child (Crijnen, Achenbach, & Verhulst, 1997). Further, for children in the longitudinal comparison study, discrepancies between parent and child reports of internalizing and externalizing problems decreased for older versus younger children due, in part, to fewer reported problems by parents of older children (Stephens & Holden, in press). Although there were only slight and nonsignificant differences, all three systems of care served fewer children with serious internalizing behavior problems on the CBCL at intake than the non-system-of-care communities.
The pair of comparison communities with the largest differences in the percentage of children rated to have serious mental health issues is the Santa Cruz County-Travis County pair. About 11 percent more children in Santa Cruz County were rated on the CAFAS as having marked or severe functional impairment. About 13 percent more children in Travis County, however, were rated on the CBCL Total Problems and Externalizing scales as having clinically significant behavioral problems. Some of these differences as measured by the CAFAS may be attributed to the previously mentioned age bias introduced by the intentional focus of the Santa Cruz County program to serve older children and those involved with the juvenile justice system. Although the program enrolls children into services from a variety of sources, the local probation department and juvenile residential facility are two major referral points. Two determinants of a higher CAFAS score are delinquent behavior in the community and residential placement in a juvenile detention facility. Thus, higher CAFAS scores for children in Santa Cruz County than in Travis County were not unexpected. In addition, data from the outcomes study of the national evaluation (CMHS, 1999) has revealed a similar relationship between CAFAS and CBCL Externalizing scores as a function of increasing age.
In addition to measures of children's behavior and functioning, the comparison study protocol also includes indicators of family functioning. The Caregiver Strain Questionnaire (Brannan et al., 1998) was administered to all caregivers. It measures the degree of strain experienced by a caregiver as a result of his or her responsibilities related to caring for a child with behavioral problems. For instance, did the caregiver have to miss work as a result of the child's problems? Did the caregiver feel angry toward the child as a result of the child's behavioral or emotional problems? Did the caregiver worry about the child's future?
Caregivers with children who have more serious behavioral and emotional problems experience more strain in their lives. Overall, caregivers of children in the Stark County and the East Baltimore systems of care did not experience significantly more strain in their lives than caregivers of children in the respective comparison communities of Youngstown and West Baltimore. However, caregivers in the Santa Cruz County community reported more overall strain on average than caregivers in Travis County (t = -2.47, df = 244, p = .014). Factors affecting these higher mean scores may include the higher average CAFAS ratings of functional impairment and the older ages of the children served in Santa Cruz County--two predictive variables of caregiver strain.
Children and caregivers in paired communities generally were well matched by gender, marital status, and caregiver education. However, some communities did not match on age, racial identity, family employment, and family income. |
The attempt to match child and family demographic characteristics, behavior, and functioning at intake was met with mixed success. Populations across all six communities were very diverse. While children and caregivers in paired communities generally were well matched by gender, marital status, and caregiver education, some community pairs did not match on age, racial identity, family employment, and family income. The attempt to match children and families on demographic characteristics was not conducted as part of the child selection process, but rather as part of the less rigid overall community selection process. Although communities were fairly well matched during the community selection process, the local children's mental health agencies' enrollment processes did not always mirror predictions or census information used to match communities. As a result, some significant differences were also reported in functional and behavioral measures across matched pairs. Children from Youngstown exhibited significantly more functional impairment on average as measured by the CAFAS, and children from Travis County presented with more serious emotional and behavioral problems at intake as measured by the CBCL. These baseline differences will need to be taken into account on any analyses of behavioral and functional change within and across communities.
All of the intake data and 96 percent of the 6-month data were available for an analysis of outcomes in July 2000. Because data collection continues for later data collection waves, only 6-month outcomes are appropriate to analyze at this time. The lack of complete follow-up data prohibits the use of sophisticated statistical analysis procedures that will be appropriate for the complete longitudinal data set. Analysis techniques that are appropriate for the preliminary data are relatively limited in their ability to examine individual differences in change across time or between groups, and any differences that are found may be spurious due to unstable patterns that may not be substantiated in the final data set. The presence of unequal sample sizes for the groups exacerbates these limitations. The following preliminary results should be viewed with these caveats in mind.
To assess the relative effectiveness of service delivery systems to affect functional and behavioral changes in children with serious emotional disturbance, CAFAS total scores and CBCL and YSR total problems scores were analyzed at intake and 6 months in each comparison pair (see Table 16). Preliminary analyses of change in CAFAS total scores from intake to 6 months in the Stark County-Youngstown comparison pair revealed that, overall, children in the non-system-of-care community were significantly more functionally impaired on average than children in the system-of-care community. The mean CAFAS total score, averaging across intake and 6-month data collection waves, was 76 for children in Youngstown compared to an average across waves of 68 for children in Stark County, and both communities showed significant improvement from intake to 6 months. There were no differences in the rates of improvement between communities. In the East Baltimore-West Baltimore comparison pair, children in the East Baltimore system-of-care community were significantly more functionally impaired than children in the West Baltimore non-system-of-care community. As in Stark County and Youngstown, children in both East Baltimore and West Baltimore showed significant improvement. Again, there were no differences in rates of improvement between communities. In the Santa Cruz County-Travis County comparison pair, no differences in average CAFAS total scores were observed between the communities. Children in both Santa Cruz County and Travis County improved significantly from intake to 6 months at comparable rates (see Table 16).
Preliminary analyses of change in CBCL Total Problems scores from intake to 6 months indicated no significant differences in rates of change between the Stark County and Youngstown communities. Children in both communities showed a significant reduction in behavioral and emotional symptomatology from intake to 6 months, and the communities did not differ overall. For the East Baltimore-West Baltimore pair, CBCL Total Problems scores decreased to below the clinical range, but they decreased significantly more in West Baltimore. This finding should be interpreted with caution, however, given the preliminary nature of these analyses. For the Santa Cruz County-Travis County pair, CBCL Total Problems scores overall were significantly higher in Travis County than in Santa Cruz County, but both communities showed significant improvement from intake to 6 months at comparable rates.
Analyses of mean YSR Total Problems scores, as reported by youth 11-18 years old, revealed findings similar to those reported by caregivers on the CBCL. For the Stark County-Youngstown pair and the East Baltimore-West Baltimore pair, children in both the system-of-care and the non-system-of-care communities reported reductions in behavioral and emotional problems from intake to 6 months, and there was no difference in the rates of change between the systems of care and the non-system-of-care communities. For the Santa Cruz County-Travis County comparison pair, overall YSR Total Problems scores were significantly higher in Travis County than in Santa Cruz County. Scores improved significantly in both communities, however, and there was no difference in their rates of change.
The outcomes assessment included analyses of caregiver reports of strain. The global strain subscale scores of the CGSQ were analyzed in each of the comparison pairs. For all three pairs, global strain was reduced from intake to 6 months, and there were no significant differences between systems of care and non-system-of-care communities in the rate of change.
The preliminary findings for the CBCL Total Problems scores reported above suggested the need for further analytic investigation concerning the differential rates of improvement. Because CBCL Total Problems scores are a composite of scores that summarize both internalizing problems (e.g., depression, anxiety) and externalizing problems (e.g., delinquency, aggression), additional analyses were conducted to examine the potential for differential change of mean scores for internalizing and externalizing problems over time in each comparison pair (see Table 17).
For the Stark County-Youngstown and Santa Cruz County-Travis County pairs, CBCL Internalizing and Externalizing problems scores dropped from intake to 6 months, and there were no differences in the rates of change between the systems of care and their matched comparison communities. For the East Baltimore-West Baltimore pair, Internalizing problems dropped significantly more from intake to 6 months in West Baltimore than in East Baltimore. Externalizing problems improved significantly from intake to 6 months as well, but there were no differences between the two communities in terms of their rates of change. This suggests that the finding for CBCL Total Problems for the East Baltimore-West Baltimore pair reported above is accounted for primarily by changes in Internalizing problems scores; however, these preliminary results should be interpreted cautiously.
The behavior and functioning of children in both systems of care and non-system-of-care communities improves from intake to 6 months. The results are preliminary and should be viewed with caution. |
Overall, the outcomes results at 6 months suggest that children served both in systems of care and in more traditional service delivery systems improve in behavior and functioning over time. However, the need to interpret these preliminary outcomes results with caution cannot be overstated. As described previously, the need to complete follow-up data collection at the 12-, 18-, and 24-month waves limits the analyses that can be performed on the data that are currently available. The analysis options appropriate for the intake and 6-month data are limited in their ability to examine individual differences in change over time. The analysis strategies employed with the currently available data are designed to examine average group differences in outcomes. The examination of change in the current analysis has been limited to a comparison of change conceptualized as an incremental rather than a continuous process. Two waves of data provide minimal information about the process of change. Once data collection for the subsequent follow-up waves is complete, the availability of multi-wave outcomes data will allow other options in the analysis of change. Analysis techniques exist which allow the individual trajectory of change for each child to be modeled mathematically. In addition, individual- and community-level characteristics can be used to explain the differences in individual change trajectories that are observed. Final assessment of the impact of systems of care on child and family outcomes will incorporate these more refined analysis techniques. In summary, conclusions regarding the effectiveness of systems of care in improving child and family outcomes based on the results presented in the current chapter would be premature.
The Phase II longitudinal comparison study (two grant communities funded in 1997 and two matched, non-system-of-care communities) began in spring 1999. Results presented below should be considered preliminary, and are subject to change as more children are enrolled into the study.
Through June 30, 2000, a total of 201 children and caregivers have been interviewed at intake and 43 at 6 months for the Phase II comparison study (see Table 18). About half of the interview data were collected from Region III in Nebraska, a current CMHS-funded system-of-care community.
The following preliminary data analyses focus on the demographic, risk factor, and clinical characteristics of the children and families participating in the Phase II comparison study at intake. The degree of similarity of children enrolled into the comparison study to children enrolled in the Comprehensive Community Mental Health for Children and Their Families Program is important to consider. A representative sample of children will allow comparison study results to be generalizable to a wider array of communities implementing the system-of-care approach. In addition, less variability in the characteristics of children enrolled will enable results in children's outcomes to be attributed to characteristics of the service delivery system rather than other factors. Currently, not enough information is available at 6 months to make valid inferences about changes across time for these children and their families.
Communities have enrolled 201 children in the Phase II comparison study, and data collection, cleaning, and entry have been completed for 192 children and families enrolled prior to June 2000. Subsequent analyses are based on this subsample. Table 19, which summarizes the demographic characteristics of the sample, shows that the children were predominantly young White males, and the majority of the children were from families with annual incomes below $25,000.
For over half of the children (55.2 percent), a single biological parent had legal custody, while 27.4 percent were in the joint custody of two biological parents or one biological and one step-parent. In those families where a single biological parent had custody, it was most often the biological mother of the child (95.2 percent).
Caregivers were asked whether their children had received mental health services prior to entry into the study. Across the four communities, 84.8 percent of the children who participated in the Phase II comparison study had received services related to their emotional and behavioral problems in the 12 months prior to intake data collection. Their previous service utilization is presented in Figure 65. As can be seen, the largest proportion of children received outpatient and school-based services, while the proportion of children receiving alcohol/substance therapy was the smallest.
Emotional disturbance and mental health problems are often associated with specific child risk factors. Risk factors cited most frequently for children in the Phase II sample included being a runaway and being physically abused (see Figure 66).
As in the Phase I comparison study, the primary behavioral and functional measures for the Phase II comparison study are the CAFAS and the CBCL. The CAFAS was administered to provide a broad assessment of how children function in different life domains, including school, work, and home environments. The CBCL provided reports of parents' perceptions of their children's emotional and behavioral symptoms.
Based on CAFAS scores, most children at entry into services in the Phase II comparison study communities likely will need some type of significant, long-term mental health services. |
CAFAS total scores at intake indicate that most (69.1 percent) of the children's mental health disorders in the comparison study can be classified as marked or severe (see Figure 67). The marked and severe levels of functioning identify children who are in need of intensive mental health services and multiple sources of supportive care (Hodges, 1990a). According to Hodges, the approximately 24 percent of children in the Phase II comparison study who were rated as functioning at the moderate level may be appropriately assisted with outpatient care, but they will likely need additional supportive services. Therefore, according to CAFAS scores at intake, 93.2 percent of children at entry into services in the comparison study communities likely will need some type of significant, long-term mental health services.
The CBCL scores revealed that most children in the Phase II comparison study sample present with serious mental health problems at intake. Overall means for the total sample and for gender subsamples were above the clinical cutoff score of 63 for the Internalizing, Externalizing, and Total Problems scores (see Table 20). In fact, 77 percent of children in the comparison study at intake had Total Problems scores that fell within the clinical range. However, a notable difference existed between caregiver and youth reports of problem behaviors on the CBCL and YSR. Youth tended to report lower rates of symptoms than caregivers across all three scales. In fact, on the YSR, overall means for the total sample and for each gender subsample were below the clinical cutoff on all three scales (see Table 21). Only 38.2 percent of youth perceived their behavioral and emotional problems to be within the clinical range (above the clinical cutoff), indicating a need for clinical care.
Assessment of service experiences at the practice level was included in the Phase II comparison study to determine whether system-of-care principles were integrated in practice-level interactions between providers and families. The practice-level assessment examines the congruence between the system-of-care program design and what is actually experienced by children and families, and whether service delivery is infused with system-of-care principles and values. Fidelity to program design is critical to theory-driven evaluation in which the study design is explicitly framed by the theorized relationship between the program's elements and activities and the desired client outcomes (Bickman, 1987; Chen, 1990). In services research, it is important to know not only whether the program was developed in accordance with the program theory and design, but also whether individual clients experienced those services as expected (Cordray & Pion, 1993; McGrew, Bond, Dietzen, & Salyers, 1994; Sechrest, West, Phillips, Redner, & Yeaton, 1979). Without understanding how well an individual client's actual services adhere to what was planned and intended, it is impossible to attribute outcomes to the service delivery model. A system of care may have key components (e.g., case management, multidisciplinary treatment teams, wraparound services) in place and make a wide array of service options available at the community or agency level, but individual clients may not experience those services as intended. That is, if the study does not find the desired outcomes, such as improvements in child functioning, it is difficult to know whether the system-of-care theory failed or whether the theory was not truly implemented as designed. To some extent, this was the case with the Fort Bragg Evaluation Project (Bickman et al., 1995). Some authors contend that the evaluation established that the service demonstration was developed in accordance with the program's theory and that it was, as a whole, implemented well (Heflinger, 1996), while others have argued that implementation was weak, and the program theory was not adequately tested (Friedman & Burns, 1996). Regardless, little was learned about whether individual children and families experienced the treatment regimens recommended by the multidisciplinary treatment teams. Therefore, the practice-level assessment allows the testing of potential differences between systems of care and services as usual that may influence child and family outcomes.
There are several components of the practice-level assessment. In the Phase II comparison study communities, as in the national evaluation outcomes study communities, information is collected at follow-up about services received by children and families and their satisfaction with those services (using the Family Satisfaction Questionnaire [FSQ] and the Youth Satisfaction Questionnaire [YSQ]). Caregivers and youth provide information about their service contacts, including types of services the child and family received during the past 6 months, their service settings, length and sequence of services, and extent to which the service met the family's needs.
Another component addresses whether families' experiences with the overall system and with individual providers were in accordance with system-of-care principles. Two measures were developed for this purpose: the Experience with Service System Questionnaire (ESSQ), which assesses, from the caregiver's perspective, the extent to which the system as a whole embodied system-of-care principles, and the Service Experience Questionnaire (SEQ), which asks caregivers and youth to rate their case management services and the service used most often. Data collection using these measures began in spring 2000 in the Phase II comparison study communities.
A third component of the practice-level assessment examines in detail whether services that were planned were in fact received, the obstacles that may have been encountered in service delivery, and family and provider perspectives about service experiences and the incorporation of system-of-care principles in these services. This substudy will be conducted in the four Phase II comparison study communities with 50 families and their providers during Years 3-5 of the grant period. Caregivers and their child's primary provider will complete diaries or logs of service activities and will be interviewed by telephone on a biweekly basis over 16 weeks to assess informal and formal services used, child and family involvement in decisions, coordination, contacts with other service providers, service experiences, and reasons why service encounters did not occur during each 2-week period. Case record reviews will provide information about established service plans, attendees at service planning meetings, services received, documentation of service coordination, explanations for nonparticipation, and any changes in service plans. Data collection for this study component will begin during FY 2001.
A fourth component of the practice-level assessment measures the attitudes and practices of a broad variety of providers of services to children and families in these communities. A survey will be mailed to at least 40 providers at each of the four Phase II comparison study communities twice during the grant period to appraise possible changes in their attitudes and practices as systems develop.
The transportability of evidence-based practices into the community has become an important topic of concern, both for researchers who have developed manualized interventions and conducted efficacy trials, and for those who are more directly invested in examining services at the community level. The Comprehensive Community Mental Health Services for Children and Their Families Program provides a potential laboratory for conducting the research necessary to understand community parameters that may facilitate or impede the effectiveness of evidence-based practices for children, adolescents, and their families. In response to an increased interest in evidence-based practices in mental health settings, future plans include the addition of evidence-based practice studies among grant communities funded in 1997-98 and in 1999-2000. The transportability of evidence-based practices out of laboratories and into communities and the interaction of systems of care and evidence-based practices are the major areas of focus. It is hypothesized that the inclusion of evidence-based practices in both the practice and evaluation of systems of care may strengthen and improve the system-of-care model, the quality of its services, and the outcomes for children and families. Results may provide evidence on the factors that are necessary to ensure that evidence-based practice protocols are modified appropriately and delivered with the highest possible fidelity within communities that serve children with emotional and behavioral disorders and their families.
The implementation of these studies is being driven by a collaborative process involving the stakeholders who are actively involved in the Comprehensive Community Mental Health Services for Children and Their Families Program. An Evidence-based Practice Work Group consisting of government personnel, members of the national evaluation team, expert external consultants, and representatives from grant communities and family advocacy groups has oversight for the study. The initial activity of the work group has involved surveying communities to determine their plans for implementing evidence-based practices and their level of interest in, and readiness for, participating in the evidence-based practice study. Results of a survey recently completed with the communities initially funded in 1999 revealed significant variability in grant communities' familiarity with evidence-based treatments. While a small number of grant communities funded in 1999 have plans to include an evidence-based practice in their service array, many grant communities expressed concern about the feasibility of random assignment of children to receive evidence-based treatment. An update on the progress of the study of evidence-based practices in systems of care will be provided in the next Annual Report to Congress.
Preliminary outcomes results of the Phase I comparison study indicate that CMHS funding of systems of care has effectively influenced system development across communities. Further, the implementation of system-of-care principles seems to have had a significant impact at the level of service provision. According to families and providers, service experiences reflect services that are community based and family focused and, for two of the three pairs, more culturally competent care. While 6-month outcomes data suggest that children in systems of care and non-system-of-care communities improve at similar rates, the possibility exists that these findings may be spurious due to unstable patterns of variability and may not be substantiated in the final data set. Further, this preliminary analysis only examined change from intake to 6 months. As the Phase I comparison study comes to a close, more conclusive results will soon become available regarding the evaluation of the effectiveness of the system-of-care approach in improving longer-term outcomes for children with serious emotional disturbance and their families. While the preliminary 6-month outcomes results are inconclusive, it seems likely that the question to be addressed by the comparison study will become, not if, but for whom, does a system of care improve outcomes? More sophisticated analyses that require additional follow-up data from the subsequent data collection waves will determine more conclusively the factors influencing individual improvement rates.
Preliminary descriptive results of the Phase II comparison study provide demographic and functional and behavioral status indicators at intake. Participants in the CMHS-funded systems of care and in non-system-of-care communities are typically White, male, 12 years old, and from families with annual incomes below $20,000. A majority of children have received outpatient and school-based mental health services prior to entry in the study, have experienced multiple risk factors, and present with serious emotional and behavioral problems. Further longitudinal data collection is planned and will include practice-level and evidence-based practices studies.
The SGR concluded, based in part on data from CMHS-funded communities, that systems of care are effective in reducing residential and out-of-home placements, that they have succeeded in improving child functioning and behavior, and that parents are more satisfied with the services they receive. However, the effectiveness of systems of care remains to be demonstrated conclusively by studies that include a comparison group, and the effect of system-of-care implementation on the cost of services remains unclear. Furthermore, examining the relationship between changes at the system level and changes at the practice level is critical to understanding the link between system characteristics and child and family outcomes.
This point was also underscored by Burns, Hoagwood, and Mrazek (1999) in a comprehensive review of the literature on effectiveness of treatments for children's mental health disorders. They concluded that the relationship between treatment components and the principles underlying service system delivery may be one of the most important elements influencing short- and long-term outcomes in community settings. Evidence from these efficacy trials indicates that treatment can have a significant positive impact on children and families when compared to groups that received no treatment, were placed on a waiting list, or were exposed to a placebo intervention. These results, however, stand in contrast to the more limited information on community-level outcomes, which provides less support for the differential effectiveness of interventions in community settings.
|
| Home | Contact Us | About Us | Awards | Accessibility | Privacy and Disclaimer Statement | Site Map |
