This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
2001 Annual Report to Congress on the Evaluation
|
|
CHAPTER SUMMARY In summary, findings from the Phase I longitudinal comparison study indicated that
|
While the previous chapter has documented the effectiveness of the Comprehensive Community Mental Health Services for Children and Their Families Program in achieving meaningful changes in system structure and function, the relationship between these changes and clinical outcomes for children remains a critical area of investigation. The longitudinal comparison study component of the national evaluation was designed to answer the primary question:
Secondary research questions that follow from this include:
The longitudinal comparison study of the national evaluation was first implemented in 1997 in three of the original 22 grant communities funded in 1993 and 1994. In addition, a Phase II comparison study involving the selection of two additional systems of care from among the 23 communities funded in 1997 and 1998 and their matched comparison communities is currently being conducted. The Phase II comparison is described in detail in Chapter V. These studies contribute to nearly a decade of research designed to determine the effectiveness of systems of care (and of the services provided within them) for improving child and family outcomes. While the evaluation and research literature have demonstrated consistently the positive effects of system reform, it is less clear whether these system-level changes translate into greater symptom improvement for children served in systems of care than for those served in non-systems of care (DHHS, 1999; Farmer, 2000). One limitation of previous work has been the narrow focus on clinical outcomes as the only relevant measure of a system of care's effectiveness. Of equal, if not more importance, are improvements in indicators of functioning in school and in the community (Rosenblatt, 1998). A need exists to develop evidence of the mechanisms through which system changes affect changes in service delivery which in turn affect changes in clinical outcomes for individual children and families. Farmer observed that there has been a shift in focus as system-building initiatives have grown from their historical roots in the Child and Adolescent Service System Program (CASSP). While the principles of a system of care, as outlined by CASSP (Stroul & Friedman, 1986), have continued to drive these initiatives, an emphasis on individual-level outcomes as measures of effectiveness has replaced the initial focus on macro-level changes and system performance. However, the complexity of the interventions, the lack of clarity in defining how effectiveness should be determined, and the lack of well-defined theories of change as the basis for the interventions have all contributed to the difficulty of evaluating the effectiveness of systems of care at the individual level (Farmer, 2000).
The longitudinal comparison study addresses the research questions listed above by comparing the outcomes of children and families served by CMHS-funded systems of care with the outcomes of those served by non-systems of care. The results of the comparison study will contribute to our understanding of whether systems of care lead to better outcomes for children and families than are found in non-systems of care and will begin to link system-level factors with practice-level factors.
A multi-method approach was employed to evaluate system characteristics, child clinical and functional outcomes, service delivery and costs, and family service experiences. Many components of the comparison study were the same as those in the overall national evaluation described throughout this report. The primary component was the child and family outcomes study that included functional and behavioral measures such as the Child and Adolescent Functional Assessment Scale (CAFAS; Hodges, 1990b) and the Child Behavior Checklist (CBCL; Achenbach 1991a) as well as other functional indicators (e.g., school placements, school attendance, contacts with law enforcement, family functioning, social support). A detailed description of the measures is presented in Appendix B. Once entered into the study, all children and families were asked to participate in intake (baseline) interviews and follow-up interviews at 6-month intervals for up to 2 years. Field staff in each community interviewed families in the convenience of their homes. Further, as described in the previous chapter, a system-of-care assessment was conducted in each community to assess the system's infrastructure and service delivery practices relative to the system-of-care principles. In addition, a select sample of children and families participated in a system-of-care practice review that assessed the extent to which their service experience conformed to system-of-care principles. Finally, management information system data were obtained from mental health agencies to examine service use and costs over time.
Selection of three grant communities from among the 22 communities funded in 1993-94 was initiated in the summer of 1997. In each case, the CMHS-funded system-of-care community was matched with a non-CMHS-funded community that used a different approach to serving children. All comparison study communities were selected based on the following criteria:
The final selection criterion was willingness to participate. A few potential communities were unwilling to participate because of other planned activities during the proposed study period. Based on these criteria, the matching pairs listed in Table 4 were selected for the Phase I comparison study. The selection of multiple comparison pairs provides the opportunity to examine multiple replications of the comparison study design with pairs that have different geographical and demographic characteristics.
Children and families recruited for the study were asked to participate for up to 2 years after entering the study. Participants presented for mental health services or were referred by agencies such as juvenile justice, schools, or child welfare. In the system-of-care communities, children enrolled into services had serious emotional problems in accordance with the CMHS grant guidelines. Thus, all children receiving services in the system-of-care communities were eligible for the study. Children from matched comparison communities were eligible for the study if they were between the ages of 6 and 17.5 and presented with serious emotional or behavioral problems. Specifically, at least one of the following four criteria had to be met: (a) a diagnosis of a mental health disorder and a clinical or functional assessment score above the clinical range, (b) a history of services received from multiple (i.e., two or more) child-serving agencies (e.g., juvenile justice, education, child protective services, or substance abuse), (c) currently at risk of or past history of out-of-home placement, or (d) participation in a special education program for children with serious emotional disturbance. All children and families who met the above criteria were selected until desired sample sizes were reached or the enrollment periods had ended.
The comparison study had an enrollment goal of 1,100 children across all six communities, and individual community enrollment goals were set at 150 to 200 children, depending upon the local agency's enrollment capacity. Overall, enrollment goals were met. Enrollment of children and families ended in January 2000, with a total of 1,036 families enrolled (see Table 5). Six-month interviews, which included 869 families, were completed in June 2000. Through December 2000, 842 families were interviewed at 12 months, 697 families at 18 months, and 475 families at 24 months. Because enrollment continued into January 2000, some children and caregivers were not eligible for their 18- and 24-month interviews before the end of data collection in December 2000. By the end of data collection, 87 percent of families had been approached for their 18-month interview, and 63 percent of families had been approached for their 24-month interview.
The overall data completion rate was strong compared to other longitudinal studies of children's mental health service delivery:
|
In addition to the number of children and families initially enrolled into the study, the number of families retained in the study over time affects the statistical power to detect differences across communities. The family retention rates in Table 6 were generally acceptable. Overall, through the end of data collection in December 2000, 84 percent of families had been retained in the study. Although total data completion rates across communities ranged from 73 to 84 percent, the overall data completion rate of 80 percent across the four follow-up waves of data collection was strong compared to other longitudinal studies of children's mental health service delivery. For example, in the Fort Bragg Evaluation Project, completion rates at 18 months ranged from 65 to 81 percent for key outcomes measures (Hamner, Lambert, & Bickman, 1997), and in the Stark County study, Bickman, Summerfelt, and Noser (1997) reported a data completion rate of 76 percent at the 6-month follow-up interview. Angold, Costello, Burns, Erkanli, and Farmer (2000) reported a 70 percent completion rate across four data collection waves in the Great Smoky Mountains Study.
Analyses presented throughout this chapter have focused initially on data from the matched communities of Stark County, Ohio, and Mahoning County, Ohio, because recruitment, data completion, and retention of children and families were most successful in this pair. Data from each of the remaining pairs are being analyzed separately, and the results of those analyses will be presented in the 2002 Annual Report to Congress. The research questions were addressed across each of the components of the longitudinal comparison study (i.e., system characteristics, child clinical and functional outcomes, service delivery and associated costs, and family service experiences). Specific analysis strategies for each component are discussed in the following sections.
To address system characteristics, quantitative scores were calculated from individual item ratings from the system-of-care assessment. These scores were used to determine the extent to which each community incorporated system-of-care principles in the domains of infrastructure and service delivery (a detailed description of the system-of-care assessment protocol is provided in Chapter II).
The system-of-care framework (see Chapter I) asserts that improved child and family outcomes can be expected from a service system in which the guiding system-of-care principles and values have been implemented fully and in which high-quality, effective treatment is provided. Given this assumption, a test of the effectiveness of systems of care requires an assessment of the degree to which these principles and values have been implemented.
As stated in Chapter II, the system-of-care assessment rates the extent to which system-of-care principles are operating within eight service system components. In addition, these eight service components can be categorized and viewed within two specific domains: infrastructure and service delivery. Infrastructure is defined as the organizational arrangements and procedural framework that support and facilitate service delivery. The four service components included in this domain are governance, management and operations, service array, and quality monitoring. The second domain, service delivery, refers to the activities and processes undertaken to provide services to children and families for the purpose of addressing and, to the extent possible, relieving the emotional and behavioral challenges experienced by the child. Within this domain, the four remaining service components can be found: entry into the system, service planning, service provision, and care monitoring and review.
Using this structure as a basis for comparison, the findings from the Stark County and Mahoning County system-of-care assessments are summarized below. In addition to comparing overall performance within each of the system-of-care principles, the degree to which the funded community successfully implemented the system-of-care principles within the infrastructure and service delivery domains relative to the non-funded community is also discussed. Note that the unit of analysis for this portion of the study was at the level of the service delivery system. A single score was generated for each principle in each domain for each community. Thus, statistical tests of significance are not appropriate.
The system-of-care assessment yielded summary scores on a scale of 1 to 4 (with 1 being a lesser degree of development) for each principle in the infrastructure and service delivery domains. Overall, the system-of-care assessment data documented that Stark County was successful in implementing the system-of-care principles. Stark County received higher scores than Mahoning County in seven of eight system-of-care principles, the exception being the provision of services in the least restrictive environments (see Figure 8). This difference in overall scores for the area of least restrictive resulted from differences between the two systems in the infrastructure domain (see next section). Across all principles, Stark County received scores ranging from 2.17 to 3.67. Five of the eight areas assessed received scores of 3 or above, indicating that effective efforts had been made toward the implementation of these principles. In contrast, Mahoning County scores ranged from 1.47 to 3.02, with only one principle score above 3.0. These differences suggest that the Stark County system of care was implementing system reform with a substantial degree of fidelity to the guiding principles of system-of-care development. The greatest differences between the two communities were in the areas of interagency involvement, community based, and collaborative/coordinated. Stark County's higher scores in these areas indicated that the grant community had made specific efforts to promote and enhance the involvement and partnership of core child-serving agencies in the system of care, to provide services within close geographical proximity to their targeted community, and to facilitate the coordination of services available within the service array. However, it must be noted that, although the measure also indicated that Stark County had effective efforts and mechanisms in place, these efforts were not entirely sufficient for full implementation of these principles. Further, it should be noted that the Mahoning County non-system of care was not completely failing to implement some of these principles. This differences are explored in further detail separately for each domain in the sections below.
Stark County's higher overall system-of-care assessment scores in the areas of interagency involvement, community based, and collaborative/coordinated indicated the grant community had made specific efforts to
|
Components of the infrastructure domain include governance, management and operations, service array, and quality monitoring. When comparing the two communities within this domain, the findings indicated that, in general, Stark County was successful at implementing the system-of-care principles. The Stark County system of care received higher scores than the Mahoning County non-system of care in five of the eight system-of-care principles for infrastructure (see Figure 9). The scores for Stark County ranged from 1.33 to a high of 3.12, with the lowest score for cultural competence and the highest for interagency involvement. Mahoning County, however, received their lowest score in the area of interagency involvement (1.2), reflecting a relative lack of focus in facilitating the involvement and partnership of multiple child-serving agencies. Mahoning County did, however, receive higher principle scores than Stark County for accessibility of services and provision of services in the least restrictive environments.
Mahoning County's higher accessibility scores are reflective of a restructuring of their intake and enrollment process as well as specific efforts made to minimize financial barriers to services. Although both communities had been somewhat effective at minimizing the use of inappropriately restrictive services, Mahoning County's slightly higher score in this area can be attributed to their routine monitoring of information related to the use of overly restrictive service options and their use of this information to reduce future occurrences of inappropriately restrictive services.
Components of the service delivery domain include entry into the service system, service planning, service provision, and case monitoring and review. When comparing the two communities within this domain, Stark County received higher scores than Mahoning County in all of the system-of-care principles except for provision of services in the least restrictive environments (for which both communities received the same score; see Figure 10). Principle scores for Stark County ranged from 2.44 to 4.0. Scores for Mahoning County ranged from 1.86 to 3.5. Stark County received scores above 3.0 in seven of the eight principles, while Mahoning County's scores within the service delivery domain averaged only 2.68, with six of the eight principle scores below 3.0.
The greatest difference between the two communities was in the area of interagency involvement. Mahoning County scored lowest in this area (1.86), indicating that limited focus had been given to establishing and enhancing interagency involvement in the service delivery domain. The Stark County system of care's higher interagency scores can be attributed to multiple agency involvement within the enrollment and intake process as well as during the planning of services provided to children and families.
In summary, findings from the system-of-care assessments illustrate the degree to which Stark and Mahoning Counties differ in terms of system development. Differences between the two service systems in terms of overall principle scores were reflected in both the infrastructure and service delivery domains, with the greatest differences occurring in service delivery. In general, Stark County's higher scores in nearly all areas assessed are reflective of their effective efforts to operationalize and implement the system-of-care principles within their service system, and they clearly distinguish service delivery in the grant-funded community from Mahoning County's non-system of care.
Demographic characteristics of the 449 children and families interviewed at intake (232 in the system-of-care community and 217 in the matched comparison community) are presented in Table 7. A comparison of baseline demographic characteristics revealed that children enrolled in the study were fairly similar in the two communities. Children's age and gender distributions did not differ; however, children in the two communities did differ with regard to race and ethnicity, and family income. Children in the system-of-care community were more likely to be White and to report family incomes greater than $15,000 per year.
Similar to the national evaluation, the primary behavioral and functional measures for the comparison study were the CAFAS and the CBCL. The CAFAS was administered to provide a broad assessment of how children function in eight different life domains, and the CBCL provided parent reports of levels of emotional and behavioral symptoms. Results from both the CAFAS and the CBCL showed that the typical child enrolled in the study had serious behavioral and emotional problems. The CAFAS criterion for marked or severe behavior using the CAFAS 5-scale total is a score of 70 or higher. Children from the Mahoning County community were rated as having significantly higher average levels of functional impairment than the children from Stark County, although both groups scored in the marked/severe range (t = 3.08, df = 440, p < .005).
While there were significant differences between children in the two communities at baseline, results showed that the typical child in both communities had serious behavioral and emotional problems. |
For the CBCL Total Problems scale, mean T-scores for children in the two communities were similar at intake. The mean scores for children in Stark County and in Mahoning County were 68.9 and 70.2, respectively, falling within the clinical range of 63 or higher (indicating a need for clinical care).
In order to reflect accurately both individual and group change over time, a series of growth curve analyses with linear and quadratic growth components was conducted using hierarchical linear modeling (HLM; Bryk & Raudenbush, 1992) and general growth mixture modeling (GGMM; Muthén & Muthén, 1998). These complex statistical techniques are designed to model individual variation in change over time and, in the case of GGMM, to identify different subgroups, or classes, of individuals based on different patterns of change over time. By including the service delivery approach as a predictor of these patterns of change over time, this technique presents a useful method for determining whether intervention effects exist for subgroups of children in the comparison study sample. Both of these statistical techniques allow inclusion of individuals in the analysis who do not have complete data for the outcomes measures at each wave of data collection, making them superior analysis techniques for measuring change in research using quasi-experimental designs (Bryk & Raudenbush, 1992; Muthén, 2001).
To assess the relative effectiveness of systems of care to effect change in emotional and behavioral symptoms and functioning for children with serious emotional disturbance, CAFAS total scores, CBCL broadband scores (i.e., Internalizing and Externalizing Problems), and one CBCL syndrome score (i.e., Delinquent Behavior) were analyzed across data collection waves.
Analyses of change in functional impairment (i.e., CAFAS total scores) from baseline to 24 months for children in Stark and Mahoning Counties revealed a complex relationship between gender (i.e., male vs. female), race (i.e., White vs. non-White), and service delivery approach (i.e., system of care vs. non-system of care). This relationship is depicted in Figures 11 and 12. At intake, according to the CAFAS, males in both communities were more functionally impaired than females, regardless of race. In the Stark County system of care, White males improved at a greater rate than minority males, and minority females improved at a greater rate than White females (see Figure 11). In the Mahoning County non-system of care, the reverse was seen. Minority males improved at a greater rate than White males, and White females improved at a greater rate than minority females (see Figure 12).
Analysis of change revealed initially greater rates of improvement in externalizing behaviors and emotions for children in the Stark County system of care than for children in the Mahoning County non-system of care. However, these improvement rates began to dissipate at 18 months, resulting in equivalent outcomes for children in both communities at 24 months. |
Analysis of change in CBCL Total Problems raw scores indicated there were no significant differences in rates of improvement between children in the Stark County system of care and the Mahoning County non-system of care. On average, children in both communities improved over time from intake to 24 months. However, analysis of change in CBCL Externalizing raw scores revealed significant differences between children in the Stark County system of care and the Mahoning County non-system of care (see Figure 13). At intake, caregivers in Stark County reported slightly fewer externalizing behaviors and emotions for their children than did caregivers in Mahoning County. Rates of improvement were initially greater in Stark County than Mahoning County. This trend continued from intake to 12 months, but deceleration of improvement was also greater for Stark County than Mahoning County, resulting in equivalent levels of improvement for children in both communities at 24 months.
To address the question of whether there were subgroups of children and families for whom the system of care was more effective than the non-system of care, a general growth mixture model (GGMM) was used with service delivery approach (i.e., system of care vs. non-system of care) as a predictor of different patterns of change over time. Growth mixture modeling is a relatively new analysis technique that identifies different subgroups, or classes, of individuals based on differences in patterns of change over time (Muthén & Muthén, 1998). By including the service delivery approach as a predictor of these patterns of change over time, this technique presents a useful method for determining whether intervention effects exist for subgroups of children in the comparison study sample. The first stage of the analysis identifies subgroups based on patterns of change for children across both communities combined. In the next stage, subgroups are examined for significant differences in their patterns of change over time attributable to differences in service delivery approaches.
Analysis of functional impairment was performed using the CAFAS data. Results indicated that an assumption of a single pattern of change over time for all children in the sample was more viable than assuming multiple different patterns of change.
For the CBCL Total Problems raw score, a series of models was fit to the data which proposed from one to five classes, or subgroups, of children with different patterns of change in CBCL scores over time. The best-fitting model was the two-subgroup solution. The two subgroups were named based on the nature of their change over time. Subgroup 1 was named "High-Mediums" because they had high scores at intake, and their scores dropped into the medium range by 24 months. Subgroup 2 was named "Medium-Lows" because they had scores in the medium range at intake, and their scores dropped into the low range by 24 months (see Figure 14). Significant declines in Total Problems scores occurred for both subgroups. The High-Mediums declined, on average, from an initial score of 90.25 to 63.25 by the end of the study. The Medium-Lows also declined, from a score of 53.85 to 31.04. The probabilities for subgroup membership for the entire sample were .38 for the High-Mediums and .62 for the Medium-Lows. Thus, children were relatively more likely to exhibit the Medium-Low pattern of change over time.
For the Externalizing broadband scales, a series of five growth mixture models, ranging from one to five subgroups, was also fit to the data. The four-subgroup model was best fitting for both subscales.
The subgroups were named using the same naming convention described previously based on the subgroups' scores at intake and at 24 months. The four subgroups for Externalizing were Externalizing Subgroup 1: "Low-Low," Externalizing Subgroup 2: "Medium-Medium," Externalizing Subgroup 3: "High-Medium," and Externalizing Subgroup 4: "Medium-Low." All subgroups showed a significant decline over the course of the study. The largest decline, 13.93 points, was for the Medium-Lows, followed by the High-Mediums (9.00), the Medium-Mediums (4.85), and the Low-Lows (4.22). Figure 15 graphically displays the estimated patterns of change over time for the four identified subgroups.
The next step in the analysis was to test all of the identified subgroups for significant differences in their patterns of change over time attributable to differences in service delivery approaches. In these analyses, except for those involving Externalizing raw scores,2 control variables of gender, race/ethnicity, age, and service delivery approach were entered in the GGMM models to control for initial differences between the children and families recruited from the two different communities.
Significant differences between the two service delivery approaches were found only for the CAFAS one-subgroup model. The initial status was the only significant difference between the two service delviery systems. Children served in Stark County had lower CAFAS Total scores at entry than children served in Youngstown (an average difference of 6.59 points).
In summary, CBCL measures of children's emotional and behavioral problems indicated the presence of multiple subgroups based on different patterns of change over time. The largest number of subgroups, four, was found for the CBCL Externalizing scale.
Equally as important as improvements in clinical outcomes for children are improvements in indicators of functioning in school and in the community. Important functional indicators measured in the comparison study include (a) placements in special education classes due to a behavioral or emotional problem, (b) suspensions from school due to behavioral or emotional problems, (c) the amount of prosocial peer support that children self-report, and (d) involvement with the juvenile justice system. These functional indicators were analyzed using hierarchical linear modeling to determine predictors of changes in these indicators over time.
Caregivers in both the Stark County system of care and the Mahoning County non-system of care reported on their children's special education placements for emotional or behavioral problems. As seen in Figure 16, the decrease in proportion of children in special education placements from intake to 24 months was greater in Stark County than in Mahoning County.
Boys and girls were impacted differently by service delivery approach. Girls in Mahoning County experienced an increase in special education placements over time, while girls in Stark County and boys in both Stark County and Mahoning County experienced a decrease in special education placements (see Figure 17).
The effect of service delivery approach also varied as a function of family income. In Stark County, children from families with annual incomes less than $15,000 showed a greater reduction in special education placements than children from families with incomes of $15,000 or more. In Mahoning County, the opposite was observed, with children from families with incomes of $15,000 or more showing much greater reduction in special education placements than children from families with incomes of less than $15,000 (see Figure 18).
Caregivers also reported on their children's suspensions in the past 6 months. Reductions in the proportion of children suspended from school in the previous 6 months varied as a function of gender and service delivery approach. Boys and girls in the Mahoning County non-system of care and girls in the Stark County system of care experienced similar reductions in suspensions from intake to 24 months. However, boys in Stark County experienced the least amount of improvement of all groups as a result of higher proportions of children suspended at 24 months (see Figure 19).
In addition to these educational functioning indicators, youth 11 years of age or older reported on the frequency (1 = never, 6 = always) with which they experienced peer and adult prosocial support during the 6 months prior to each interview. This Social Relationship Questionnaire measured the social support of the youth outside the home and asked questions concerning their relationships with peers and adults. Only peer social support varied as a function of service delivery. Youth in the Stark County system of care showed significantly greater improvement in peer social support from intake to 24 months than youth in the Mahoning County non-system of care (see Figure 20).
Involvement with the juvenile justice system is another important community indicator of functional impairment and behavioral problems. Offenses vary widely in severity from juvenile "status" offenses such as underage drinking to serious felony offenses such as theft and assault. Both counties' Juvenile Courts cooperated in providing data for any child in the study who had been charged with an offense between 1997 and 2000. Thus, offense data are available both before and after a child entered the study.
Of the 232 children in the Stark County sample, 91 children (39 percent) were charged with an offense at least once during the 1997-2000 period. Of the 217 children in the Mahoning County sample, 103 children (47 percent) were charged with an offense. All 4 years of the juvenile offense data are used in the following analyses, regardless of when a child was enrolled into the study. Thus, the periods for which offense data is available before and after study entry varies across children, depending upon when they were enrolled (e.g., more offense data are available after study entry for children enrolled earlier). Note, however, that children in the two communities were enrolled at similar rates. For this juvenile offense data set, the mean number of months after study entry for children in the Stark County system of care was 27.5 and for children in the Mahoning County non-system of care was 29.6 (t = 2.07, df = 192, p < .05).
Children with multiple juvenile justice charges before study entry in Mahoning County's non-system of care were more likely to have multiple charges after study entry compared to their counterparts in Stark County's system of care. |
While more children from Mahoning County were involved with the juvenile justice system during the study period, similar percentages of children in Stark County (37 percent) and Mahoning County (38 percent) committed offenses before entry into the study. However, for the period following study entry, recidivism in Mahoning County was greater than in Stark County. Specifically, children with two or more charges before study entry were more likely to have two or more charges after study entry in Mahoning County than in Stark County. Figure 21 presents a summary of the number of juvenile justice charges after study entry for those children with two or more charges prior to entering the study. While only 6 percent of children from Mahoning County did not recidivate after study entry, 30 percent of children from Stark County did not recidivate after committing two or more offenses prior to study entry. Thus, the Stark County system of care appears to have a greater impact on recidivism among the most chronic juvenile offenders compared to the Mahoning County non-system of care.
While measures of clinical symptomatology and functioning are important in understanding the impact of a system of care on children, the principles on which a system of care is founded suggest that measures of impact should extend beyond these child-focused clinical outcomes to assess impact on the family as a whole. To reflect the family focus of a system of care, the comparison study protocol included the Caregiver Strain Questionnaire (CGSQ; Brannan, Heflinger, & Bickman, 1998) as an indicator of the effectiveness of services for the family (see Appendix B). This questionnaire measures the degree of strain experienced by a caregiver as a result of his or her responsibilities related to caring for a child with serious emotional disturbance. The assumption made is that, if the child and family are adequately supported, this support will be reflected by a reduction in strain. Caregiver strain was analyzed using hierarchical linear modeling to determine predictors of changes in strain over time. Results indicated that caregivers in both communities reported similar levels of strain at entry into services, and their strain was reduced similarly over time (p > .05 for all subscales examined).
The evidence presented in previous sections of this chapter suggests that, for a subsample of children with the most severe symptoms, a system of care is more effective in changing behavioral and emotional symptoms than a non-system of care. An understanding of the relationship between changes at the level of the service delivery system and individual outcomes for children and families can only be derived by exploring direct service experience. The System of Care Practice Review (SOCPR; Hernandez, Gomez, Lipien, Greenbaum, Armstrong, et al., 2001) was included to assess the service experiences of children and families during the provision of care in system-of-care and non-system-of-care communities. Hernandez et al. (2001) provided a description of the development and application of the SOCPR within the national evaluation's longitudinal comparison study as well as results of a comparison between service delivery systems in the study. The results of these initial analyses indicated that the service experiences of families were more consistent with system-of-care principles in the CMHS-funded systems than in the non-system-of-care communities. A subsequent set of analyses was conducted in an effort to assess the extent to which care that embodies the principles of a system of care affects clinical outcomes for children being served in CMHS-funded systems of care and matched comparison communities. Because the sample for the SOCPR was small (N = 96), data were combined across all children served in the three systems of care and across all children served in the three matched comparison communities to create two service delivery groups.
Of the 96 children and families who participated in the SOCPR in the longitudinal comparison study, 75 had complete data for both CBCL Total Problems and SOCPR total scores. These 75 children and families comprised the sample for the current analyses. The number of children in the service delivery groups and their associated demographic characteristics are presented in Table 8.
The SOCPR protocol for each family consisted of multiple data sources, including document review, primary caregiver interview, child interview, formal provider interviews, and informal helper interviews. Summary scores (1 = strongly disagree to 7 = strongly agree) were generated for the following four domains and their underlying subdomains: (a) Child centered and family focused - Individualized, Full Participants, and Case Management; (b) Community based - Early Intervention, Access to Services, Level of Restrictiveness, and Integration and Coordination; (c) Cultural Competence - Sensitivity and Responsiveness, Awareness, Agency Culture, and Informal Supports; and (d) Impact - Improvement and Appropriateness of Services. Finally, a total score was also calculated.
To explore the nature of the relationship between SOCPR scores and clinical outcomes, CBCL Total Problems scores were examined in relation to SOCPR total scores. Table 9 contains descriptive statistics for the SOCPR total scores and the CBCL Total Problems raw scores and T-scores for both samples. Mean SOCPR total scores were higher for the system-of-care children than for the children in the matched comparison communities. Children in both groups had mean CBCL Total Problems T-scores within the clinical range at baseline (clinical range is defined as a T-score ³ 63). The mean for children in the matched comparison communities at the 12-month follow-up was still within the clinical range, but the mean for children in systems of care dropped below the clinical cutoff.
In order to understand the factors related to positive outcomes in children's mental health service delivery, the CBCL Total Problems raw score at 12 months was analyzed using multiple regression with the following predictors: (a) the baseline CBCL Total Problems score, (b) race, (c) service delivery approach, (d) SOCPR total score, and (e) the interaction between service delivery approach and the SOCPR total score. As expected, baseline clinical symptom scores predicted symptom scores at 12 months. More important, however, were the other significant predictors of CBCL Total Problems at 12 months: the SOCPR total score and the interaction between service delivery approach and the SOCPR total score. The relationship between SOCPR total scores and CBCL Total Problems at 12 months in each community is depicted in Figures 22 and 23, which show individual scores for CBCL Total Problems scores at 12 months as a function of SOCPR total scores. In addition, the figures include trend lines that represent the nature of the relationship between the two measures at baseline and 12 months for the group as a whole.
As seen in Figure 22, the entire sample of children from the system-of-care communities had SOCPR total scores greater than or equal to 4, which indicates that all children and families in the system-of-care communities experienced services that embodied the system-of-care principles at a consistently high level. There was a trend suggesting that as children's clinical needs increased, the perceptions that their service delivery experiences embodied system-of-care principles also increased. However, the trend was not statistically significant.
In contrast, Figure 23 shows more variability in SOCPR total scores for the sample of children and families in the non-system-of-care communities. Further, there is a tendency for the children in the non-system-of-care communities who perceived greater manifestation of system-of-care principles in their service delivery experiences to have fewer behavioral and emotional symptoms at intake. This inverse relationship between experience of the principles and clinical symptoms is even stronger at 12 months after intake into services.
In summary, children and families in systems of care reported experiencing services that embodied system-of-care principles at high levels. Their service experiences were more consistent, and their symptom severity did not vary as a function of intensity of their experiences. In contrast, children and families in matched comparison communities reported more variability in their experiences of services that embodied system-of-care principles, and their symptom severity varied inversely as a function of their experiences. A previous report of the system-of-care assessment in the longitudinal comparison study (Brannan, Baughman, Reed, & Katz-Leavy, in press) found similar results regarding the operationalization of the system-of-care principles. System scores across the systems of care were less variable than those across the non-systems of care, yet there was some movement toward the system-of-care approach in the non-systems of care. The current results underscore the importance of measuring service experiences at the practice level. The longitudinal comparison study being conducted in the Phase II comparison communities is examining further the characteristics of service experiences from the perspectives of family members and providers. There is a need to replicate these findings with a larger sample size to allow for analysis of individual variation in change over time using more sophisticated analysis strategies, like hierarchical linear modeling.
Children and families in systems of care
Children and families in matched comparison communities
|
Results from the comparison study illustrate that many children experience improvements in their behavioral and emotional symptoms after receiving mental health services and some groups of children experience greater improvements than others. One possible reason a certain child may improve more than another may be the child's service use pattern. Previously in this chapter, results from the comparison study's system-level assessment described how the Stark County system of care facilitated the involvement of core child-serving agencies, provided community-based services, and coordinated services to a greater degree than the more typical approach to service delivery used in Mahoning County. Such system-level characteristics can create a more accessible and seamless service experience for families and children. However, the actual services received play a critical role in changing behavior and reducing emotional problems.
Service utilization patterns include the types and combination of services received, the quantity of services received, the intensity of services, and the duration of services. Also important in the study of systems of care is the cost of serving children with emotional and behavior disorders. The costs of services that were effective in producing positive outcomes is important information for the future planning of children's mental health service systems.
A major component of the comparison study involved the collection and analysis of mental health services and costs data to help explain differences in the success of serving children with emotional and behavioral disorders in the two systems. Services and costs data were collected in all six comparison study communities, but results in this section will continue to focus on the Stark County and Mahoning County pair to build upon the results previously discussed in this chapter.
The following basic questions are addressed:
|
Services data were collected from the community mental health centers (CMHC) through which children were enrolled into the study in Stark and Mahoning Counties. Each CMHC has a computerized management information system (MIS) used for internal management purposes and to bill insurance companies, Medicaid organizations, families, and other parties responsible for payment. The charges for services were collected for the study to compare the costs for serving children in the system of care to the costs in the matched comparison community.
While CMHCs recorded the primary mental health services that each child received in their MIS, services for which the CMHC could not bill were not systematically recorded. Some non-billable services were recorded in the Stark County MIS, but none was recorded in the Mahoning County MIS. Thus, all non-billable services were excluded from analyses to enhance comparability between the two communities. Examples of non-billable services in Stark County included, but were not limited to, administrative support, cross-agency collaboration planning, certain evaluation services, and certain types of case management services. In addition to non-billable services, services contracted out to other agencies were not recorded in the Stark County and Mahoning County MIS's. However, staff involved in the comparison study worked with the staff of other community agencies to collect data on contracted services. For instance, any child requiring inpatient psychiatric services had to be referred out to other facilities in the community because the Stark County and Mahoning County CMHCs participating in the study did not offer inpatient services. Families also approached other community agencies independently seeking services for their child without a referral from the CMHCs, and the comparison study collected this data as well. This report focuses initially on the primary mental health services billed through the CMHCs, then briefly examines other community services that children and families received. To make this distinction clear, we refer to the CMHCs when the analyses focused on these "primary" mental health MIS data.
The primary mental health services data were collected for the entire study period from January 1997 through December 2000. Since the first child was enrolled into the study in August 1997, all mental health services that children received after entering the study were captured. Since the last child was enrolled in October 1999, at least 15 months of services data were available for all children. The analyses of services and costs data in this section include only services received in the first 18 months following enrollment into the study. Ninety-five percent of children in Mahoning County and 84 percent of children in Stark County had a full 18 months of available services data for analysis. (In Stark County, 32 percent of all children received services more than 18 months after study entry, and 36 percent did so in Mahoning County.) Of the 232 children in the Stark County sample, the participating CMHC was able to provide services data for 229. The three remaining children were missing data in CMHC records. Services data were available for all 217 children in the Mahoning County sample; however, some children and families left services shortly after intake, before the study's baseline interview could be completed. With these families removed from the study, 214 children from Stark County and 206 children from Mahoning County were included in the final services and costs data analyses presented below.
The system-of-care approach specifies a set of principles that serve to guide the organizational characteristics of a children's mental health service system and what principles providers should follow when serving children. If systems of care are distinct from non-system-of-care approaches, the system organization and principles of service delivery should be manifested in a unique service use pattern for the child and family. Children's service use patterns can be distinguished by multiple characteristics. The following characteristics are examined below: type and combination of services, quantity, duration, intensity, and continuity.
The primary types of mental health services that the Stark County and Mahoning County CMHCs recorded in their MIS's were similar, including intake and assessment services, individual counseling, family and/or group counseling, medication monitoring, and case management. The percentage of children who received a specific type of service provides an indication of its prominence in the program's service array. All children received intake and assessment services in both communities. Case management services and behavioral medication monitoring were delivered to the same percentage of children in each community. One of the differences between the Stark County and Mahoning County CMHCs was their use of different types of counseling (see Figure 24). The Mahoning County CMHC relied more heavily on individual counseling, while the Stark County CMHC provided a broader mix of individual, family, and group counseling. Ninety-five percent of children received individual counseling from the Mahoning County CMHC compared to 75 percent from the Stark County CMHC. However, the Stark County CMHC provided family or group counseling to 82 percent of children compared to 11 percent of children at the Mahoning County CMHC.
The Mahoning County CMHC offered a residential crisis stabilization unit that provided a short-term place to stay for families in need. The stabilization unit was closed in March 1999 during the middle of the study. Seven percent of the children in the study used the residential crisis stabilization unit before it closed. The Stark County CMHC offered partial hospitalization services. Five percent of children received partial hospitalization services, which typically included 4 hours of therapy and educational services per weekday.
The Stark County CMHC within the system of care delivered more case management and less medication monitoring as a proportion of the "service mix" than the CMHC in the Mahoning County non-system of care. |
Examining the percentage of children who received each type of service does not reflect the overall combination of services delivered to children. An agency's service mix describes the amount of each service delivered in relation to all other services delivered. Of all services delivered to children by the Mahoning County CMHC, the core of the service use pattern was individual counseling and case management, while the Stark County CMHC provided a more balanced mix of individual counseling, group and family counseling, and case management (see Figure 25). In addition, while similar percentages of children in the two communities received medication monitoring and case management, the Stark County CMHC actually delivered more case management and less medication monitoring than the Mahoning County CMHC delivered in relation to other services.
Children and families served by the Stark County CMHC within the system of care received more than twice as many services in the first 18 months after study entry as children and families served by the CMHC in the Mahoning County non-system of care. |
Another dimension of a child's service use pattern is the quantity of services received. A dramatic difference existed between the communities in terms of the number of services received per child. The Stark County CMHC delivered more than twice as many services (average of 65.8 per child) in the first 18 months after study entry as the Mahoning County CMHC (average of 29.5 per child; t = -5.99, df = 418, p < .001). When hours of services are examined, the Stark County CMHC and Mahoning County CMHC diverge even more. Children served by the Stark County CMHC received an average of almost three times as many hours of services (74 hours) than children served by the Mahoning County CMHC (26 hours) in the first 18 months of service delivery.
When individual services were examined, the most pronounced difference was found in the average hours of case management received per child. Although a similar percentage of children served by the Stark County and Mahoning County CMHCs received case management in the 18 months following study entry, the children served by the Stark County CMHC averaged 38 hours compared with 11 hours for children served by the Mahoning County CMHC (see Figure 26). The system-of-care approach emphasizes the importance of case management activities such as coordinating services, advocating for and supporting families, and quality monitoring. Thus, the relatively large amount of time spent on case management is a confirmation of the strength of the implementation of the system of care for families served by Stark County. In addition, the children who received assessment, individual counseling, and medication monitoring services at the Stark County CMHC received significantly more hours of each respective service compared to children served by the Mahoning County CMHC. The only service for which the Stark County and Mahoning County CMHCs provided similar hours of service per child was group or family counseling, but the Stark County CMHC provided the service to 70 percent more children (n = 175 in Stark County vs. n = 23 in Mahoning County).
There are two possible reasons children served at the Stark County CMHC received a larger quantity of services: Either (a) they were engaged in services for a longer period of time or (b) they received more intense service delivery over the same period of engagement. The duration of service use was measured by the number of months between the child's first and last service. Looking only at the first 18 months after study entry, children served by the two CMHCs did not differ significantly; both groups received services for an average of 9.5 months. Service intensity was defined as the average hours of services received per month. For the measurement of intensity, only months between the first and last date of service were included in the calculations. The intensity of services delivered to children by the Stark County CMHC (7.5 hours per month) was significantly higher than those delivered to children by the Mahoning County CMHC (3.9 hours per month; t = -4.25, df = 418, p < .001). This difference is related to the number of days children received service encounters in the two communities. During the 18-month period, children served by the Stark County CMHC had service encounters for an average of 49 days compared to 23 days for children served by the Mahoning County CMHC.
Another related method of differentiating children's service use patterns is the continuity of services received. A child's service use pattern can be interrupted because the family or mental health staff cancel appointments, the child runs away, the family moves, the child is officially discharged and re-enrolled, or a variety of other reasons. Previous analyses have suggested continuous services can have a positive impact on children's outcomes (CMHS, 1998). Since the intensity of services delivered by the Stark County CMHC within the system of care was greater, continuity of services might also be greater. Figure 27 compares service continuity within 3-month periods (six quarters). Within each quarter, the number of days between service contacts was averaged for the entire sample of children in the two communities. At the Stark County CMHC, the average break between services in the first quarter was about 4 (3.82) days, and it increased slightly over the entire 18-month period. However, the average break in services never rose above 5.45 days. At the Mahoning County CMHC, the average break between services was not only greater, but also increased dramatically over time. The average break in services at the Mahoning County CMHC during the third quarter (7-9 months after study entry) was 11.53 days, more than twice as long as at the Stark County CMHC (4.62 days), and was more than three times as long during the last quarter of services.
More intensive case management within the system of care reflects how the Stark County CMHC was collaborating with other agencies, coordinating services, and providing a continuous service experience with minimal gaps. |
Although the services data available from the Stark County and Mahoning County CMHC MIS's did not provide complete detail on the entire array of services offered to children, a distinct and significant difference still existed among the mental health services delivered through the two CMHCs for which data were available. The more intensive use of case management, in particular, highlights the strength of a system of care and reflects the implementation of multi-agency collaboration and service coordination. Relatively high levels of case management also may be related to the greater continuity of services in the Stark County CMHC. Case management often includes services such as service planning, service coordination with other providers, monitoring progress in school, and family training on daily living skills. Such wraparound service contacts can often serve the purposes of bridging the gaps between the more formal service contacts and creating more service continuity.
Aggregate comparisons of the mental health services children received at the Stark County and Mahoning County CMHCs reveal a general pattern of how service utilization characteristics such as service quantity and continuity were greater in the Stark County CMHC within the system of care, but these analyses cannot reveal individual variations in service use patterns over time. Many child and family characteristics influence a child's service use pattern, including the child's age and type of behavioral disorder (Foster, Kelsch, Kamradt, Sosna, & Yang, 2001). In the following section, a third characteristic that plays an important role in the child's service plan is examined: the severity of a child's behavioral disorder. Analyzing the change in service use patterns of children based on the severity of their functional and behavioral problems may help further distinguish the system-of-care and non-system-of-care approaches.
Changes in the type and quantity of services delivered in the Stark County CMHC within the system of care suggest a more appropriate response to the needs of the child over time than was found at the CMHC in the Mahoning County non-system of care. |
To explore the relationship between severity of functional impairment and service use patterns, children with an overall CAFAS total score between 70 and 150 were placed in a severe group and children with a score range of 0-69 were placed in a less severe group. (CAFAS 5-scale total scores range from 0-150.) Figure 28 presents the average monthly hours of service use by severity levels within Stark County and Mahoning County. During the first 18 months after study entry, the average hours of services continually decreased for children in the severe and less severe groups from both the Stark County and Mahoning County CMHCs. As might be expected, the most intensive service use pattern was during the first 6 months, and this intensity decreased as children's functioning improved and they exited services. Also evident was that children served by the Stark County CMHC received more services than children served by the Mahoning County CMHC, regardless of severity. Service utilization decreased sharply between the first and second 6-month periods and then leveled out between the second and third 6-month periods. Statistically, a significant quadratic effect was observed for service utilization over time. Regardless of the community, the severe group used significantly more hours of services over time than the less severe group (F = 5.52, df = 1/442, p < .01).
Within a community, the difference in service utilization over time between the severe group and less severe group was significant for the Stark County system of care but not for the Mahoning County non-system of care. The severe group in Stark County received significantly more hours of services than the less severe group over time (t = 3.64, df = 227, p < .001). Among children served in the Stark County system of care, children in the severe group received nearly twice as many hours of services over time than children in the less severe group. The greater disparity between the severe and less severe groups in the Stark County system of care suggests a more appropriate response to the needs of the children over time. In Mahoning County, children in the severe group received about the same hours of services over time as the less severe group (t = 0.22, df = 215, p > .05). The non-significant differences in average hours of service use between the severe and less severe groups in Mahoning County indicated that this non-system of care did not differentiate services based on a child's functional impairment.
Comparisons between communities showed that children in the Stark County system of care used more services over time than children in the Mahoning County non-system of care. This difference is accounted for largely by the severe group in Stark County. At each time point, the severe group in the Stark County system of care used significantly more hours of services than the remaining three groups. In short, the results showed that, regardless of the severity of functional impairment, children in the Stark County system of care used more hours of services over time than children in the Mahoning County non-system of care. In addition, children in the system of care who had severe functional impairment received more hours of services than children with less functional impairment, but there was no differentiation of service hours provided for children with severe and less severe impairment in the non-system of care.
When the type of services received was examined (see Figures 29 and 30), the CMHC in the Stark County system of care again showed slightly more variation for children with different levels of behavioral disorders (see Figure 29). While children in the severe and less severe groups received assessment, case management, medication monitoring, and different types of counseling, only the severe group received partial hospitalization services. One goal of a system of care is to minimize the use of restrictive services. By reserving the use of partial hospitalization for children with the most severe behavioral disorders, the Stark County CMHC minimized the restrictiveness of services for children with less severe behavioral disorders.
When changes over time were examined, both CMHCs service mixes included minor variations, but most services were provided in the same proportion over time. Both CMHCs provided assessments and evaluations, individual counseling, and medication monitoring in the same proportion over time for children in both the severe and the less severe groups. The less severe group in the Stark County system of care gradually received less group or family counseling and more case management over time. In the Mahoning County CMHC, the severe and less severe groups received more group or family counseling and less crisis residential services from intake to 6 months and from 6 to 12 months, but these figures are related to the closing of the residential crisis stabilization unit during the middle of the study in March 1999. Likewise, the decrease in percentage of residential care and the increase in case management for the severe group in Mahoning County could also be explained by the closing of the residential unit. For both severity groups in Mahoning County, there was a slight increase in percentage of medication use and a decrease in percentage of individual counseling relative to all other services.
Thus far, analyses have focused on the mental health services delivered by the two CMHCs participating in the study. However, as described previously in this chapter, many of the children enrolled into the comparison study had serious mental health disorders that typically drew the attention of multiple child-serving agencies in the community. Two of the consistently stated goals of the system-of-care grants have been (a) to serve children with multiple agency needs, and (b) to increase the coordination among these agencies in order to reduce duplication of services. Preventing the duplication of services can potentially generate a reduction in the overall community costs for serving a child and the burdens to families and providers.
The comparison study methodology incorporated a community-wide services and costs data collection effort for the purposes of identifying services received outside the CMHC and estimating the overall community costs to serving children in the study. The effort included the identification of mental health services delivered by children's psychiatric hospitals or residential treatment centers in each community. Data on these services were collected from the records of the four primary inpatient facilities in each of the two communities (two in each community). Three of these facilities delivered primarily short-term inpatient hospital services. The fourth provider was a non-hospital-based mental health agency in Mahoning County that provides residential, inpatient, and partial hospitalization. Thus, no residential treatment placements were made for children served in the Stark County system of care. In addition, placements made through special education programs, child welfare agencies, and juvenile justice agencies were included in the data set. Data were collected for foster care, group home, and residential placements made through the counties' child welfare agencies; special education placements in both counties' largest cities and some rural districts; detention placements made by the local juvenile justice agencies; and placements in community shelters.
From the services data collected from these community agencies, the Stark County system of care appeared to be slightly more effective in reducing juvenile detention services children received in the community, although none of the differences was statistically significant (see Figure 31). In the first 18 months after study entry, 7 percent fewer children were placed in juvenile detention centers in Stark County as compared to children in Mahoning County. Although the difference is not significant, it does indicate a trend. From January 1999 through July 2000, the average length of stay in the Stark County juvenile detention center was 22 days at a cost of $1,408. Using figures from this 18-month period, an estimated $22,528 less was spent in Stark County on juvenile detention costs compared to Mahoning County.
The percentage of children placed through the local child welfare agencies was slightly, but not significantly, higher in Stark County. Qualitative information describing the conditions surrounding children's placements was not available. Thus, it is difficult to draw conclusions about the implications of differential rates of child welfare placements.
An equal percentage of children in both communities was placed in special education programs at some time during the 18 months following entry into the study. However, slightly more children were placed in inpatient hospitals and residential treatment centers in Mahoning County for mental health treatment. As noted above, there were no residential treatment placements made for children served in Stark County. Thus, the Stark County placement data reflect only inpatient hospital placements. Cost data from one of the Stark County inpatient hospitals was incomplete, making it difficult to document the difference in community provider expenditures during the 18-month period.
The above description of children's service use patterns documents how the Stark County CMHC within the system of care delivered services to children in greater amounts and with greater intensity than the Mahoning County CMHC within the non-system of care. As would be expected, the costs of providing these more intensive services were also significantly greater. Overall, $4,690 was spent on the average child served by the Stark County CMHC, while $2,256 was spent on the average child served by the Mahoning County CMHC (t = -5.35, df = 418, p < .0001). As was the case with the amount of services delivered, much of the difference in costs was due to the difference in case management. For children who received case management, the Stark County CMHC spent an average of $2,773 per child for these services and the Mahoning County CMHC spent an average of $946 per child. As described earlier, both CMHCs provided case management to a similar number of children, but the Stark County CMHC dedicated many more service hours to case management per child. The more frequent use of group and family counseling at the Stark County CMHC also contributed to higher total costs. For the children who received group or family counseling at the Mahoning County CMHC, an average of $1,037 per child was spent on group or family counseling. At the Stark County CMHC, an average of $1,465 per child was spent on group or family counseling and 65 percent more children received this service. Figure 32 displays total expenditures for mental health service delivery by the two communities.
While the Stark County CMHC within the system of care spent more to deliver mental health services to children in the study, less was spent by their other child-serving providers. In particular, juvenile justice placements and associated costs were much less than in Mahoning County. |
One possible effect of the more intensive services delivered at the Stark County CMHC could be the reduction of other children's services used within the community, potentially resulting in costs savings. Figure 33 displays the dollars spent on outpatient mental health services, inpatient mental health services, and community placements with child-serving agencies. While Stark County spent $836,474 on community placements, Mahoning County spent $1,063,186 (closing the gap by $226,712). Spending on inpatient mental health services showed little difference between the communities. Overall, the Stark County CMHC spent more on mental health services for the children in the study, but the community's system of care spent less on placements in juvenile detention centers, group and foster homes, and special education programs.
The longitudinal comparison study of the national evaluation provides comprehensive data on the structure and function of a system of care and their relationship to child and family outcomes. At the system level, the data suggest that, in general, the grant initiative was effective in translating system-of-care principles into reform of a system's infrastructure and service delivery approach. At the program level, there was evidence of an expanded array of services and that system-of-care principles had penetrated into the service experiences of individual children and families in the systems of care (and to some extent in non-systems of care). At the child and family level, there was some suggestion that these changes at the system and program levels led to differences in the initial rate of improvement in externalizing problems of children, but this dissipated over time. There was more compelling evidence that intensity and continuity of services was greater in the system of care.
These findings are consistent with previous studies (Clark et al., 1994; Clark, Lee, et al., 1996). Typically, there has been symptom change across time regardless of the approach to service delivery, with subtle differences between approaches. As others have pointed out, it is difficult to isolate the differential effects of any children's mental health intervention implemented in real-world clinical practice settings (Weisz, Donenberg, Han, & Weiss, 1995; Weisz & Jensen, 1999). The few studies that have attempted to do this using a direct comparison to a no-treatment or placebo condition have found little support for the effectiveness of these interventions (Weisz & Jensen, 1999). The innovation and creativity of the system-of-care initiative also may make it extremely difficult to measure effectiveness, regardless of how effectiveness is defined (Farmer, 2000). An emphasis on dramatic improvements in clinical symptoms fails to acknowledge that children with serious emotional disturbance have a chronic disability, not an acute illness. An emphasis on measures of social functioning in school and in the community places importance on improvements in the ability to manage existing behavioral problems in these significant settings.
It is clear from the data that Stark County's system-of-care approach to serving children and families with severe emotional needs was more comprehensive than Mahoning County's non-system-of-care approach. System-of-care assessment reports indicated that Stark County had in place a system for service planning to access cross-system resources and staff to create individualized service plans for those children and families with multiple needs and at highest risk for out-of-home placement. Also, providers reported that immediate attention was given to children with severe emotional problems-where turnaround time (from intake to assessment) for service receipt was reported to be within 48 hours, as opposed to 2 weeks for children with routine needs. In addition, quality monitoring in the Stark County system of care utilized data on child functioning (CAFAS), behavior problems (CBCL), and strengths (BERS), collected as part of the national evaluation, to examine child outcomes and to promote best practices for direct service delivery. Every effort was made to incorporate the child's strengths into the service plan. This child-centered approach to service delivery may be more effective for those with the most severe needs. Conversely, in the Mahoning County non-system of care, system-level assessments indicated that, during the service planning process, individualized service plans were developed for children in the system without substantial involvement of the children themselves or their families. Also, monitoring of individualized care was noted to be infrequent. This was in part due to increased caseloads. Furthermore, regardless of the severity of their problems, children waited approximately 2 weeks from intake before services were received. Assessment reports cited that although quality monitoring data were collected, the use of the data had yet to be determined.
Although the system-of-care philosophy provides guidance on the organization of the infrastructure and service delivery system, it does not offer a prescription for the service intervention. While the Stark County system-of-care approach can be distinguished from the service delivery approach in the Mahoning County non-system of care, the service utilization patterns that children actually experience provide important details in distinguishing the service treatment intervention. Although the service utilization data available through the Stark County and Mahoning County community mental health centers (CMHC) did not provide complete detail on the entire array of services offered to children, one inescapable conclusion is that children served by the Stark County system of care received significantly more mental health services. Children served by the Stark County CMHC in the system of care received an average of more than three times as many hours of services over the same period of time, resulting in a more intensive and continuous service utilization pattern. Results also suggest case management services in the Stark County system of care were more intensive than in the Mahoning County non-system of care.
As would be expected, the greater volume and intensity of services delivered by the Stark County CMHC also cost significantly more. The increased mental health costs in the Stark County system of care are partially offset by the reduction of their costs of placements in juvenile detention centers, inpatient hospitals, and other community child-serving providers. Given that mental health services in the system of care are more expensive, the system-of-care's effect on reducing child services and placements elsewhere in the community is important for determining the full impact. Analyses of service utilization data from community providers in this report were minimal and deserve more attention. A second important question yet to be examined is the effect of service utilization on child and family outcomes. Service utilization patterns and the quality and the efficacy of the treatments and services received are an important link between the system-of-care philosophy and changes in children's clinical and functional outcomes, and this line of research will be the focus of more comprehensive analyses in the future.
1The term non-system of care is used throughout this report to refer to communities that have not received CMHS funding to establish a system of care and that have been selected as matched comparison communities for the two comparison studies.
2For Externalizing scores, the model with the complete set of control variables was unidentified and did not converge. Therefore, for Externalizing scores, the results from a model with service delivery approach as the only predictor are reported.
|
| Home | Contact Us | About Us | Awards | Accessibility | Privacy and Disclaimer Statement | Site Map |
