This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
1999 Annual Report to Congress on the Evaluation
|
|
CHAPTER SUMMARY An update on the longitudinal comparison study indicates that:
|
A significant aspect of the national evaluation is the longitudinal comparison study. The purpose of the comparison study is to answer the primary question: "Do children with behavior and functioning problems improve more when served with a system-of-care approach compared to a more traditional approach?" A second part of the question is whether improvements in children's behavior can be uniquely attributed to the system-of-care approach as opposed to other factors that can influence a child's behavior. The comparison study attempts to address these questions by making comparisons between those children and families served by CMHS-funded systems of care and those served by other types of service systems.
In the summer of 1997, selection of the three system-of-care communities from among those communities funded in 1993 and 1994 was initiated, along with selection of their three matched comparison communities. Stark County (Canton), Ohio, was matched with Mahoning County (Youngstown), Ohio, Santa Cruz County, California, was matched with Travis County (Austin), Texas, and East Baltimore, Maryland, was matched with West Baltimore, Maryland. The selection of two additional system-of-care communities for the comparison study was initiated in the fall of 1998 from among those communities funded in 1997, along with their two matched comparison communities. In each case, system-of-care communities were matched with non-system-of-care communities that used a different approach to serving children with serious emotional disturbance. All comparison study communities were selected based on the following criteria:
Service delivery approach. To conduct a credible test of the effectiveness of the system-of-care approach, communities were selected from the 22 grant communities in Phase I based on the extent of their progress in the development of their system of care. "Mature" system-of-care communities were identified based on annual site visits to assess the development of their system of care as well as input from State and local experts familiar with the communities. Communities that scored in the top quartile made up the pool of eligible "mature" systems of care. Phase II system-of-care communities were selected earlier in their funding cycle. Thus, for the Phase II communities, "maturity" was determined according to projected plans for development and assessments of progress.
In both Phase I and Phase II, eligible matching communities were identified that did not have Federal funding to support the development of a system of care. Because it was not possible to conduct a formal system-of-care assessment for site selection purposes, alternative methods were used to determine whether service delivery approaches in potential matched communities contrasted with the system-of-care approach. These alternative methods included site visits, discussions with local agency directors, and discussion with State mental health representatives.
For each matched comparison community, there were several characteristics that suggested the service delivery approach was qualitatively different from the approach supported by the grant initiative. The Youngstown non-system-of-care community was selected based primarily on direct discussions with the Youngstown agency, including a site visit conducted in the fall of 1996. Agency officials described their approach as primarily clinic based, with little emphasis on in-home services. Although referrals were received from local school systems and the juvenile justice agency, no coordinated service planning mechanisms were in place. The Austin, Texas, non-system-of-care community was selected with the assistance of statistics from the State of Texas describing the frequent use of medication used to serve children by the Austin–Travis County Mental Health and Mental Retardation department. For the fiscal year beginning in 1996, 72 percent of service assignments in the Austin–Travis County department were designated as medication related. Discussions with Austin–Travis County officials also revealed their approach to be clinic based. The West Baltimore non-system-of-care community was selected because the traditional approach to the provision of children's mental health services in the city of Baltimore was fragmented. Officials from Johns Hopkins University, the East Baltimore Mental Health Partnership, and the city of Baltimore provided descriptions of how the city's approach to serving children involved primarily independent, outpatient-based approaches within juvenile justice, social services, and city schools. In addition, the major private children's mental health agencies within Baltimore operated independently from city government agencies.
Geographic, demographic, and economic characteristics. Data from the 1990 census were used in the selection of matching non-system-of-care communities, including population size, child age distributions, racial and ethnic composition, per capita income, size of the catchment area, the percentage of people living below the poverty level, and the percentage of adults with high school educations. When possible, geographical proximity was also considered in selecting non-system-of-care communities because matching communities from the same State meant communities would be subject to the same State mental health guidelines and State health care changes (e.g., managed care). Statewide adoption of the system-of-care service delivery approach made this unfeasible for one Phase I community (e.g., Santa Cruz, California), resulting in the recruitment of a non-system-of-care community in Austin, Texas. County- and State-level survey data on the number of children in the target population utilizing juvenile justice, mental health, and special education services were also used in the selection process.
Rate of child enrollment. The number of children and families entering the system of care needed to be large enough to ensure a sample size that would yield sufficient statistical power to detect significant group differences if they existed. Communities had to be able to enroll the requisite number of children to meet the required sample size within the timeframe allocated for the study.
Child referral patterns. Depending on the referring agency, children entering mental health services may differ in terms of presenting problems, risk factors, types of disorders, and mental health needs (Walrath, Nickerson, Crowel, & Leaf, 1998). For instance, children recruited from schools differ from children referred from juvenile court and child welfare. To strengthen the selection of children with similar degrees and types of mental health problems, similarity in referral patterns was examined when selecting matching communities.
The final selection criteria for comparison study communities was willingness to participate. A few potential community participants were unwilling to participate because of other planned activities during the proposed study period. While other potential criteria for community selection exist, the designated criteria listed above represent those considered crucial for the purposes of the comparison study. Based on these criteria, the matching pairs listed in Table 10 were selected for the Phase I and Phase II comparison studies. The selection of multiple comparison pairs provides the opportunity to examine multiple replications of the comparison study design with pairs that have different geographical and demographic characteristics.
The Stark County system of care has been refining its children's mental health care delivery system over many years since an initial CASSP grant in the 1980s. The Stark County Family Council (SCFC) is the governing body that organizes the service delivery system for all children and families in need of mental health services. The SCFC structure was developed in the mid-1980s as a result of a State-mandated effort to develop a more collaborative, interagency approach to providing services. The SCFC has a long history of multi-agency collaboration and thus has made possible the infusion of system-of-care principles. With the award of Federal grant funds, the SCFC was able to implement a full-scale system of care successfully. Mental health services have expanded to include community-based case management and mental health therapy, which are provided by staff stationed in schools and child welfare offices.
In Mahoning County, children were recruited from a private agency in Youngstown. The agency is a traditional outpatient facility that provides office-based, as opposed to community-based, mental health services. Referrals come from a variety of child-serving agencies and organizations, including hospitals, homes for battered women, foster care agencies, primary care physicians, and more traditional institutions, including juvenile justice and child welfare.
The East and West Baltimore neighborhoods offered the opportunity for a comparison within an inner-city urban setting. The system-of-care community in East Baltimore is a neighborhood-based project that limits its catchment area to specific ZIP Codes. The program contracts for services with Johns Hopkins Hospital, and the program director and staff are Johns Hopkins Hospital employees. The East Baltimore program serves children from Baltimore city school districts, the juvenile justice system, child protective and placement agencies, and other mental health providers. East Baltimore has liaison staff stationed in the different child-serving agencies and schools who refer children to the program. Through the school-based component, children and families receive direct clinical services, educational support, and therapeutic after-school and summer programs. A Family Resource Coordination Unit provides families with intensive outreach services. "Neighborhood liaisons" provide individual and family therapy, emergency support services, case management, and individualized wraparound services that are provided in the home and in other community settings.
Children in West Baltimore are served by a variety of private and public agencies. No one centralized agency serves children throughout the entire West Baltimore catchment area, so the comparison study enrolled children through different agencies, including juvenile justice, child protective services, schools, and a children's mental health provider. Children are served primarily within the agency that identifies the child's behavioral problems, with little coordination across agencies. The service providers primarily serve children only in the West Baltimore neighborhoods, which minimizes potential contamination effects of the East Baltimore system-of-care initiative.
The Santa Cruz County system of care is operated by the county mental health agency. Children are referred to the system of care from a variety of community organizations: schools, the county juvenile justice system, the county child placement agency, and a private substance abuse treatment agency. Services are provided by one of several teams that have fashioned a set of services and supports to meet a particular population's needs. The Santa Cruz County system of care places special emphasis on serving children in the juvenile justice system. Two key services are the residential facility for court wards and the juvenile probation program to bring court wards home or divert placement. Besides these services, Santa Cruz offers a comprehensive array of services to children with serious emotional disturbance, including outpatient therapy, case management/care coordination, crisis intervention, special education, reunification support, wraparound, family support, and respite.
Because California was moving toward a statewide adoption of the system-of-care service delivery approach during Phase I community selection, finding a non-system-of-care community for the Santa Cruz County system of care within California that was using a different service delivery approach was not possible. Travis County, Texas, was selected as the non-system-of-care community because it had similar demographic characteristics. In particular, both counties have a large Hispanic population, which adds to the diversity of populations included in the comparison study as a whole.
The participating agency in Travis County is the county children's mental health agency. Children served in Travis County also come from a variety of sources: the county juvenile justice system, the State child protective agency, and the education system. Access to Travis County children's mental health services is through a single point of entry with a toll-free telephone number. Travis County provides services such as assessment/evaluation, medication support, case management, mobile crisis outreach, in-office young children's groups, individual and group therapy, and transportation.
A complicating factor in this comparison pair is the awarding of a CMHS system-of-care grant to Travis County, Texas, in October 1998. This may dilute the pre-existing differences in infrastructure and service delivery between Santa Cruz and Travis County across time. However, the existence of a matched non-system-of-care community in Travis County prior to CMHS funding offers the opportunity in the future to compare changes in service delivery and outcomes prior to grant initiation to the outcomes that are obtained after a system of care is fully in place.
This comparison pair offers a unique opportunity to examine the impact of services within a rural State where the system-of-care and non-system-of-care communities each span 22-county areas of approximately 15,000 square miles. The Central Nebraska Initiative for Families and Youth (CeNIFFY), housed at the Region III Behavioral Health office in Kearney, is one of nine grant communities that received CMHS funding in the fall of 1997. Referrals to the Region III office come from the departments of parole and probation, schools, the Department of Health and Human Services, self-referrals, and referrals from other child-serving agencies. A unique aspect of the project is the Professional Partnership Program (PPP). The PPP is specifically designed to serve children and youth with serious emotional disorders who also exhibit behaviors indicating risk for out-of-community placement. PPP staff function as case managers who use a "wraparound" process to assist families in building a team dedicated to flexible, nontraditional, and unconditional care. Wraparound services include respite care, transportation assistance, and recreational therapy. Additional services include outpatient, intensive inpatient, acute inpatient, day treatment, psychiatric residential treatment, and multisystemic therapy.
Region IV Behavioral Health encompasses four service delivery centers located in Sioux City, Columbus, Norfolk, and O'Neill in northeastern Nebraska. As a more traditional service delivery system, children are referred directly to the mental health department and a host of contracted private providers. Demographics of the children and families served mirror those of the system-of-care community with regard to race, household income, and the age distribution of the children served. Outpatient services include individual, couples, family, and group counseling; psychological and psychiatric evaluations and consultation; pre-admission screens; aftercare; emergency services; and medication evaluations. Substance abuse services, home-based therapy, and therapeutic foster care are also provided. The majority of referrals come from child protective services, juvenile justice or self-referral.
The Jefferson County Community Partnership is also one of nine grant communities that received CMHS funding in the fall of 1997. A range of services is available, including assessment, case management, crisis follow-up, liaison or collateral services, in-home services, supportive therapy for foster parents, respite beds, and case management for transition into adulthood. Outpatient, day treatment, inpatient, and emergency services are obtained through contractual arrangements with other community providers. Children and families enter the Partnership through school-based diagnostic and evaluation units located in two regular middle schools, two alternative middle schools, and a referral site located in Family Court. These triage units consist of mental health specialists who conduct assessments, refer to the system of care, and collect baseline data as part of the intake process. Staff members are specifically trained in the grant service array and work collaboratively with referring agencies to ensure appropriate placement.
In the non-system-of-care community, children are recruited from a similar county-based agency, the Montgomery Mental Health Authority, that serves a four-county area. Mental health services are coordinated through one main center and two satellite centers. All children who require mental health services are referred directly to the Montgomery Mental Health Authority. Outpatient mental health services are provided on-site by the outpatient therapy team. The service array provided through the center includes emergency/crisis management, therapy, case management, in-home intervention, court liaison, pre-hospitalization screening, psychiatric services, medication monitoring, and diagnostic and evaluation testing.
In both Phase I and Phase II, children participate in the comparison study for up to 2 years during and after their service experience. Children and families participating in the comparison study present for mental health services or have been referred by participating agencies such as juvenile justice, school systems, and child welfare.
In the system-of-care communities, children enrolled into services have serious emotional problems according to the CMHS grant guidelines. Thus, all children receiving services in the system-of-care communities are eligible for the study. In the non-system-of-care communities, no such formal guidelines regarding severity of problems exist. As a result, children in these communities are screened through an eligibility process to ensure only children with serious emotional problems are enrolled into the study in an effort to equate the comparison study samples. In addition to being in the 6–17.5-year-old age range at intake into services, one of the following four selection criteria for determining severity has to be met:
Although a clinical or functional assessment score above the clinical range would be enough to determine severity, multiple criteria are used because information on any one criterion is not consistently available across communities. Besides these criteria, children had to be 6–17.5 years old and the child and caregiver had to be willing to participate. All children and families who met the above criteria were selected until sample size was reached or the enrollment period had ended.
Many components of the comparison study are the same as those in the overall national evaluation. The primary component of the comparison study is the interviews conducted with children and caregivers about child and family outcomes. Once entered into the study, all children and families participate in an intake (baseline) interview and follow-up interviews at 6-month intervals for up to 2 years. Field staff at each community interview families in the convenience of their homes about child and family behavioral and functional issues. System-of-care assessments are used to evaluate the development of service delivery systems. Evaluators make site visits to each community to assess the service delivery system's infrastructure and service delivery practices compared to system-of-care principles. Management information system data are obtained from mental health agency databases to evaluate service use and service costs across time.
The Phase I comparison study also expands upon the national evaluation with smaller study components that provide additional data about families' experiences. In each community, a sample of families and their primary service providers is selected to participate in family-centered interviews to evaluate how service systems are meeting the individual needs of children and families. The perspectives of caregivers, children, and their primary providers are combined with record reviews to address this question. In selected communities, samples of families were also selected to participate in ethnographic interviews to assess cultural factors that may impact upon the provision of care.
The Phase II comparison study design includes additional modifications. To ensure the sample size is sufficient, 225 children are being recruited in each community. In addition, an intervention-level assessment component has been added to provide a better understanding of the actual interaction between service providers and their clients. Service diaries will be completed by selected families and providers to evaluate intervention-level interactions. Questions to be answered by the intervention-level assessment include: Are planned services actually received? What factors contribute to adherence to service plans? Are children's and families' experiences consistent with the system-of-care philosophy? Are providers' attitudes and practices consistent with the system-of-care philosophy? Family-centered interviews and ethnographies will not be conducted in the Phase II comparison pairs.
Selection of Phase I comparison study communities and subsequent data collection were initiated in the fall of 1997. Through July 23, 1999, a total of 925 children and caregivers have been interviewed at baseline for the comparison study ,(see Table 11). The Stark County, Youngstown, and Travis County communities completed enrollment of families into the study at the end of July 1999, having reached their enrollment goal of 200 families. The East Baltimore community completed enrollment of families into the study at the end of August 1999, having reached its enrollment goal of 150 families. West Baltimore and Santa Cruz County are continuing to enroll families into the upcoming fiscal year. The West Baltimore enrollment process has been slowed by the delayed participation of a mental health agency from the community and enrollment levels in the city's juvenile justice and social services departments that were much lower than expected. Low study enrollment in Santa Cruz County is the result of low program enrollment in the county's system of care.
A total of 560 families have been interviewed at 6 months, 340 families have been interviewed at 12 months, and 94 families have been interviewed at 18 months. Because the Stark County and Youngstown communities started
| data collection earlier than the other four communities, 62 percent of all follow-up interviews have been completed in those two communities. An additional 12 percent of all follow-up data collection has been conducted in the Travis County community. Even though the Travis County community started data collection later, the number of children served has been higher than expected. |
|
Because the Comprehensive Community Mental Health Services for Children and Their Families Program is focused on children with serious behavioral and emotional disorders, one of the goals of the comparison study is to enroll children with serious behavioral and emotional disorders in both the system-of-care and non-system-of-care communities. Another goal of the comparison study is to enroll children and families with similar demographic and behavioral characteristics in each of the matched pairs of communities. One of the challenges in comparing the effects of different service delivery approaches is minimizing the other influences that could account for differences in child and family outcomes such as demographic and behavioral characteristics. For instance, different outcomes might occur for older children who have more serious behavioral problems than younger children. In addition, the degree of similarity of children enrolled into services in the comparison study communities to children enrolled in the other CMHS-funded system-of-care communities is important to consider. A representative sample of children will allow comparison study results to be generalizable to a wider array of communities implementing the system-of-care approach. To address these issues, a description of children and families at intake is provided as well as comparisons to children and families participating in the national evaluation descriptive and outcome study samples.
All data analyses presented for the comparison study include interview data collected before May 1, 1999, at all communities.4 Because the enrollment of children into the comparison study continues, the following description of children and families should be viewed as preliminary. However, it is likely to provide an accurate representation of all children who will eventually enroll into the study. The total projected enrollment will be approximately 1,000 children and families; 827 are described in the analyses that follow.
4To allow time to prepare data, May 1999 was used as a cutoff date for data presented in this report.
| study are male, similar to the national outcome study sample as a whole. Females tended to be slightly older than males in both system-of-care and non-system-of-care communities, with an average age of 11.8 years for females compared to an average age of 11.0 years for males in the system-of-care communities and an average age of 12.3 years for females compared to an |
|
The comparison study sample is slightly younger, more likely to be male and more racially diverse than the national evaluation descriptive sample. Annual family incomes are comparable. About two-thirds of children in both studies are male, and exactly 61 percent of families in the national outcome study earn $15,000 or less per year. Some expected differences exist in the ages and racial background of children. The national outcome study enrolled more older children as a result of eight counties in California that focused on serving older teenagers. Without the children served in California, the average age of children in the national outcome study would be similar to the comparison study sample, dropping from 12.2 to 11.4. In the national outcome study, 55 percent of children were White, 15 percent were African-American, and 25 percent were Hispanic. The relatively high percentage of children of Hispanic origin in the national outcome study can also be attributed to the many California grant communities. While 10.6 percent of children served outside of California were Hispanic, 30.4 percent of children served in the eight California grant communities were of Hispanic origin. The relatively high percentage of African-American children in the comparison study sample is the result of the intentional selection of urban, inner-city communities. Ninety-six percent of children served in the East and West Baltimore comparison study communities are African-American.
Information was also collected on the stability of the living situation of the child. At some time in the 6 months prior to enrollment into the comparison study, 22.5 percent of children in systems of care were living in a residential facility outside of their parents' home such as a group home, psychiatric hospital, alcohol and drug treatment center, or juvenile detention center, compared to 16.4 percent of children in non-systems of care, indicating that the systems of care were successfully recruiting children in or at risk of out-of-home placements (χ2 = 4.92, df = 1, p < .05).
Similar to the national evaluation, the primary behavioral and functional measures for the comparison study are the CAFAS and CBCL. The CAFAS was administered to provide a broad assessment of how children functioned in different life domains and the CBCL provided parent reports of levels of emotional and behavioral symptoms.
Figure 36 shows that in the comparison study, 57 percent of all children in system-of-care communities, and 63 percent of all children in the non-system-of-care communities, were classified at the marked or severe level of impairment using the CAFAS. The marked and severe levels of functioning are the two most impaired levels as defined by the CAFAS and identify children who are in need of intensive mental health services and multiple sources of supportive care. The approximately 30 percent of children in the six comparison study communities who function at the moderate level may be assisted with outpatient care, but will likely need additional supportive services. According to CAFAS scores at intake, 90 percent of children at entry into services in the comparison study communities will likely need some type of significant, long-term services. Only 3 percent of children displayed minimal impairment at intake.
| When compared to children in the national evaluation outcome study sample, children from the six comparison study communities were more likely to exhibit marked or severe levels of functioning at intake, but children in the national evaluation outcome study sample were more likely to exhibit moderate or mild levels of functional impairment. |
|
Similar to the national evaluation outcome study, older children in the six comparison study communities tended to have more serious functional impairment according to the CAFAS (see Figure 37). The average total CAFAS score was consistently higher for older children. For the comparison study sample, when the 6–11 age group is compared to the 12–17 age group, the difference in the CAFAS scores is significant. When comparing system-of-care and non-system-of-care communities in the comparison study samples, children in the 15–17 age group in systems of care were more likely to exhibit marked or severe levels of functioning, whereas children in the 9–11 and 12–14 age groups in non-system-of-care communities were much more likely to exhibit these same levels of functional impairment. Significant differences were not found between females and males on the total CAFAS score. However, females had higher scores on the moods and self-harm subscales, and males had higher scores on the school/work, home, community, and behavior toward others subscales.5
5 School/work subscale: t = 5.365, df = 819, p < .001.
Home subscale: t = 2.357, df = 822, p < .05.
Community subscale: t = 5.085, df = 477, p < .001.
Behavior towards others subscale: t = 3.169, df = 822, p < .01.
Moods subscale: t = -2.551, df = 822, p < .01.
Self-harm subscale: t = -2.572, df = 822, p < .01.
The CBCL scores also indicated that most children in the comparison study met the clinical criteria for serious mental health disorders. Overall means were above the clinical cutoff score for the externalizing and total problems scores for children the system-of-care communities and for the internalizing, externalizing, and total problems scores for children in the non-system-of-care communities (see Figure 38). In fact, over 69 percent of children enrolled in the comparison study in the system-of-care communities and 76 percent of children in the non-system-of-care communities had total problems scores that fell within the clinical range at intake. No differences existed between older and younger children on any of the three CBCL scales. No differences existed between males and females in the system-of-care communities or in the non-system-of-care communities. The CBCL scores for all three scales were similar to the scores for children in the national evaluation outcome study. The average externalizing and total problems scale scores were both well within the clinical range.
As Table 13 shows, most children at each individual community presented with serious mental health problems. Although some variability existed across communities, at least 50 percent of the children in each community were rated within the marked/severe ranges on the CAFAS. Similarly, on the CBCL total problems and externalizing scales, at least 60 percent of the children in each community were rated within the clinically significant range. Although there were only slight differences, the three system-of-care communities all had fewer children with serious internalizing behavior problems than the non-system-of-care communities.
The comparison pair with the largest difference in percentage of children with serious mental health issues was the Santa Cruz County–Travis County pair. About 13 percent more children in Santa Cruz County were identified by the CAFAS to have marked/severe functional impairment and about 15 percent more children in Travis County were identified by both the CBCL total problems and externalizing scores to have clinically significant behavioral problems. Some of the differences as measured by the CAFAS can be attributed to the intentional focus of the Santa Cruz County program to serve older children. As presented earlier, CAFAS scores differ significantly as a function of age of the child. The average age of children served in Santa Cruz County was 14.0 years old and the average age of children served in Travis County was 10.8 years old. Another contributing factor is the focus of the Santa Cruz County program on children involved with the juvenile justice system. Although the program enrolls children into services from a variety of sources, two major referral points are the local probation department and a juvenile justice residential facility. Children involved with juvenile justice systems tend to score higher on the CAFAS because one of determinants of the CAFAS score is delinquent behavior in the community. Further evidence of Santa Cruz County's focus on children from the juvenile justice system is found by examining the percentage of children arrested or on probation in the past 6 months (see Table 14). While the average percentage of children across all communities who had been arrested or on probation in the past 6 months was about 18 percent, approximately 50 percent of children served in Santa Cruz County had been either arrested or on probation. As expected, accompanying higher rates of substance use were also found for this community.
In addition to measures of children's behavior and functioning, the comparison study protocol also includes indicators of family functioning. The Caregiver Strain Questionnaire (Brannan et al., 1998), which was administered to all caregivers, measures the degree of strain experienced by a caregiver as a result of his or her parental responsibilities related to the child with behavioral problems. For instance, did the caregiver have to miss work as a result of the child's problems? Did the caregiver feel angry toward the child as a result of the child's behavioral or emotional problems? Did the caregiver worry about the child's future?
Caregivers with children who have more serious behavioral and emotional problems experience more strain in their lives. When children are compared by CAFAS scores, caregivers dealing with children with higher CAFAS scores experienced more overall strain in their lives than caregivers with children who had lower CAFAS scores (t = -6.615, df = 457, p < .001 – moderate and marked categories; t = -3.921, df = 471, p < .001 – marked and severe categories). Caregivers with children who scored high on the CBCL also experienced more overall strain than other caregivers. Caregivers with children who had CBCL scores above the clinical range at intake experienced significantly more strain than caregivers with children who scored below the clinical range on the CBCL (t = -13.438, df = 794, p < .001). The other factor that affected caregiver strain was the age of the child. Caregivers with older children reported more strain in their family life than caregivers with younger children.
While data are currently available for 443 children and families who have completed 6-month interviews for the comparison study, data collection and coding have been completed for less than half of the anticipated final sample that will have been successfully followed at 6 months by the end of the study. Therefore, it is deemed premature to perform statistical tests to evaluate changes across time or changes between groups at this point in time. Any differences that are found may be spurious and due to unstable patterns that may not be substantiated in the final data set.
To address the central question of this evaluation, "Are greater improvements in children's behavior and functioning associated with a system-of-care approach?" it is critical to know the degree of system development in the system-of-care as well as the non-system-of-care communities. The system-of-care assessment has been designed to assess the extent to which the comparison study communities embody system-of-care principles. The assessment was conducted using the instrument developed in Phase II, which was organized in terms of generic components of any service system (e.g., governance, entry into the system, service planning) to allow comparable assessments of non-system-of-care as well as system-of-care communities. Development of the Phase II system-of-care assessment instrument is described in more detail in chapter II.
Data for the assessment were obtained through semi-structured interviews with key stakeholders during a 3-day site visit to each of the six communities, and through a review of background information provided by the communities in the form of five data tables and other background materials.
The assessment yielded two products for each community: (a) a narrative report that describes the community's approaches to accomplishing the system-of-care principles and challenges encountered, and (b) summary scores on a scale of 1 to 4 (with 1 being a lesser degree of development) for each principle, which indicate the community's level of development for selected system-of-care principles at the infrastructure and service delivery levels.
Site visits to the Santa Cruz County, California, and Travis County (Austin), Texas, communities were conducted in late January and mid-February, respectively; site visits to Stark County (Canton) and Mahoning County (Youngstown), Ohio, were conducted in late April and mid-June, and site visits to East and West Baltimore were conducted in early April and mid-May.
|
Following is a summary of comparisons between the paired communities for infrastructure and service delivery development across eight system-of-care principles: family focused, culturally competent, interagency, community based, accessible, collaborative/coordinated, individualized, and least restrictive. Components in the infrastructure domain include governance, management and operations, service array, and quality monitoring. Components in the service delivery domain include entry into service system, service planning, service provision, and care monitoring and review. Because the unit of |
|
analysis was the system, the sample size (i.e., three systems of care, three comparison communities) was too small to test the statistical significance of the differences across systems.
Stark County (the system-of-care community) scored higher than Youngstown for five of the eight principles assessed in the infrastructure domain (see Figure 39). The considerably higher level of development in both interagency involvement and the ensuring of community-based services in Stark County reflects the system's emphasis on both attributes as core system values. A key focus of the grant has been to foster strong commitment across agencies to develop a multi-agency infrastructure through mechanisms such as pooled funding and an active, broad-based governance structure (the Stark County Family Council). Mechanisms that were put into place to maximize the use of community-based options included expanding community-based therapy options and developing intermediate service options such as partial hospitalization within the community.
Youngstown's higher level of development in the area of accessibility reflects efforts to restructure the intake process to address accessibility issues that previously had been identified. Also contributing to Youngstown's strength in this area were the multiple mechanisms in place to minimize financial barriers to receiving services, including sliding-scale fees, flexible funding, and scholarships. Both communities put efforts in place to minimize the inappropriate use of restrictive service options, but lacked the full range of intermediate service options to be fully effective in this area. Youngstown's slightly higher score in this area can be attributed to its efforts to monitor the use of restrictive settings, which were not as strong in Stark County.
Average scores for the service delivery components in Stark County indicated higher levels of development along seven of the eight system-of-care attributes when compared to those of Youngstown (see Figure 40). The greatest differences were for the interagency, community-based, and coordinated care attributes. Stark County's strengths can be attributed to that community's efforts to involve the multiple child-serving agencies at all levels through several mechanisms, including using shared intake forms, outstationing staff, and involving other agencies in service planning as needed. Stark County's interagency approach to service planning was instrumental in helping to avert out-of-community placements. Youngstown's somewhat higher score for least restrictive mirrors its higher score for this same attribute in the infrastructure domain and reflects the strength of its case review process, which gives attention to this area.
Santa Cruz County (the system-of-care community) showed greater levels of infrastructure development for six of the eight attributes (see Figure 41). The greatest differences between the two communities were for the community-based and individualized attributes. Although there was room for improvement in Santa Cruz County, there were several mechanisms in place to support the program's philosophy of providing individualized care. These mechanisms included using a multi-agency, multidisciplinary approach to planning and providing services and incorporating family members as paraprofessionals in the program's staffing. In contrast, Travis County (Austin) had very few mechanisms in place to individualize available services. Austin's low score of 1 for the community-based attribute reflects the many gaps in the service array available in that community, including a lack of substance abuse treatment, therapeutic foster care, and day treatment. Further, no efforts were in place to monitor children in out-of-community placements and plan for their return. While the family-focused score for Austin was higher than for Santa Cruz, neither community did well in this area. The low score for Santa Cruz resulted from the lack of family representation on the system of care's governing body (i.e., Steering Committee). Although there were two official slots for family representatives, no family member had attended the meetings held during the assessment year. While family representation existed on the Family Advisory Council, this was not the body that governed the system of care. In addition, families were involved only minimally in quality monitoring efforts. In contrast, Austin had one family representative on the governing body (out of 13 members). However, there was a sense that not all members were receptive to the input of this family representative. Austin's relatively high scores for the interagency, accessible, and collaborative/coordinated attributes reflected Travis County's long-standing efforts to develop a multi-agency approach to developing its system. These efforts were enhanced shortly prior to the assessment as a result of a State-funded pilot program, and also in order to apply for a CMHS system-of-care grant, which was awarded in the fall of 1998.
More differences between Santa Cruz County and Travis County (Austin) were evident in the service delivery domain; Santa Cruz County was more developed across all eight system-of-care attributes (see Figure 42). The greatest differences were seen between the sites' community-based, least restrictive, and culturally competent service delivery attributes. These findings are consistent with those in the infrastructure domain in these areas and illustrate strong links between mechanisms put into place at the infrastructure level in Santa Cruz County and their effectiveness at the service delivery level. Note that, while Austin scored higher than Santa Cruz on the family-focused attribute in the infrastructure domain, Santa Cruz scored higher than Austin on this attribute in the service delivery domain. In Austin, there were very few services for families in the service array. The services that were present had limited capacity and were very hard for families to access. In general, service systems in Austin did not identify families' needs and plan services to address them as well as those in Santa Cruz. In Austin, service systems also did not identify families' strengths in the service planning process to the extent that this was done in Santa Cruz.
Differences in the development of system-of-care attributes at the infrastructure level were more evident between East and West Baltimore than in the other two paired communities. East Baltimore (the system-of-care community) had higher scores for all eight attributes, and was substantially more developed than West Baltimore in six areas (see Figure 43). The attributes on which East Baltimore showed the greatest strength in the infrastructure domain were culturally competent, interagency, community based, and accessible. The program's success in these areas can be attributed to the incorporation of these specific system-of-care principles as central operating values as well as its success in actively involving multiple agencies in governance and management and operations functions. In addition, other efforts such as the hiring and training of community members as staff, and the emphasis placed on providing home- and school-based services significantly contributed to East Baltimore's strength at the infrastructure level. In West Baltimore, the lack of an interagency structure was evident and influenced other attributes. However, the efforts to provide oversight to behavioral managed care through an umbrella organization were reflected in the community based and collaborative/coordinated attributes.
System-of-care attribute scores for the service delivery domain follow similar patterns to the scores in the infrastructure domain; East Baltimore was stronger across all eight attributes and received substantially higher scores in four areas: family focused, least restrictive, interagency, and community based (see Figure 44). West Baltimore's relatively strong scores for least restrictive, accessible, and collaborative/coordinated can probably be attributed to the mechanisms put into place by the behavioral managed care umbrella group. In addition, strong commitment on behalf of the school system to deliver school-based mental health services with an emphasis on keeping children in school settings contributed to both East and West Baltimore's strengths in these area, as did the presence of a national school-based mental health technical assistance center at the University of Maryland that is funded by the Maternal and Child Health Bureau of the Health Resources and Services Administration.
Comparisons across all of the system-of-care and non-system-of-care communities revealed the following trends:
On a more quantitative level, 50 percent of scores were at a rating level of 3 or above within the infrastructure domain, and 88 percent of scores were 3 or above within the service delivery domain for the three system-of-care communities. For the three non-system-of-care communities, only 25 percent of scores within the infrastructure and service delivery domains were at a rating level of 3 or above. This indicates that system-of-care principles were being implemented at a higher level within the three system-of-care communities.
Previous evaluations of mental health services for children and families have focused on examining specific outcomes and the services and costs associated with obtaining these outcomes (Bickman et al., 1995; Bickman, Summerfelt, & Noser, 1997; Lambert, Brannan, Breda, Heflinger, & Bickman, 1998). Minimal attention has been devoted to practice-level parameters and cultural factors that may influence the impact of systems of care. The Phase I comparison study was designed as a multi-level evaluation that would assess ancillary factors in service delivery systems and communities that may shape the overall impact of systems of care. A clear interpretation of the development of system-of-care components and associated outcomes can only be obtained by understanding other contextual factors at the practice and community levels that can influence change.
To assess contextual factors, additional evaluation components were included in the Phase I comparison study in both system-of-care and non-system-of-care communities. These evaluation components were designed to (a) evaluate services experiences at the family and provider levels, and (b) determine how cultural factors influence the provision of care in system-of-care and non-system-of-care communities.
Assessment of service experiences at the practice level was included to determine whether system-of-care principles were being directly expressed in practice-level interactions among service delivery personnel and families. System-of-care principles may be fully implemented at the overall community and participating agency levels, but incomplete infusion of these principles into practice interactions will limit tests of the overall effectiveness of the approach. For example, in system-of-care communities, if individual providers deliver services utilizing an approach that is not consistent with system-of-care principles, then outcomes can only be attributed to the specific characteristics of the individual provider's approach rather than to the system-of-care model. Conversely, in non-system-of-care communities, system-of-care principles may be adopted by individual providers and may directly influence the services that are provided, but may not be characteristics driving development at the overall community and specific agency levels. An assessment of service experiences at the individual child and family level allows the testing of potential differences between system-of-care and non-system-of-care communities and the examination of how service experiences overall may influence the relationships between service delivery systems and outcomes.
The family-centered interview (FCI) portion of the comparison study was conducted under subcontract to the Department of Child and Family Studies at the Louis de la Parte Florida Mental Health Institute, an active collaborator and partner in the national evaluation since its inception in 1994. Case studies were used as the approach to evaluate service experiences due to their unique applicability for investigating social phenomena in real-world settings (Yin, 1994). The case study method had been used previously in several Phase I communities to provide qualitative information regarding families' service experiences and the level of expression of system-of-care concepts at the service delivery level. The specific case study protocol for this study (Hernandez & Gomez, 1998) provided both qualitative and quantitative information about families' experiences at the interface between services provided and the needs and strengths of the children and families.
Samples of families were selected from system-of-care and non-system-of-care communities for participation in family-centered interviews. The protocol for each family participating in this part of the Phase I comparison study consisted of multiple data collection components, including document review, primary caregiver interview, child interview, formal provider interviews, and informal helper interviews. Interviews and document reviews were conducted by data collection teams who typically conducted an extensive 1- to 2-week data collection site visit to obtain information for all participating families. Information was combined across the document review and interview data using a sequential analysis process that included coding, sorting, rating, and examining the information collected by the data collection team. The four overall objectives and their underlying measurement areas are listed in Table 15. Based upon an assessment of the qualitative information collected in the interviews and document reviews, trained case reviewers assigned a summative rating to each measurement area using a 7-point rating scale (1 = strongly disagree and 7 = strongly agree). For the impact objective, the measurement areas of improvement and appropriateness of services were assigned a rating based in part on the respondents' ratings of four items using a 5-point rating scale (1 = not at all and 5 = very much) that assessed how much services had helped the child, how much they had helped the family, how sufficient they were in meeting the child's needs, and whether the child and family could better deal with problems.
Family-centered interviews were conducted in the Stark County system-of-care and Youngstown non-system-of-care communities during September 1998. Twenty-one families participated in Stark County, and 22 families participated in Youngstown. Figure 45 displays the results for these two communities for the four major objectives of the family-centered interview protocol. For each objective, significant differences were obtained with the results indicating higher levels of implementation of key system-of-care principles and impact at the service experiences level in the Stark County system-of-care community. Within the objectives, the most significant differences were obtained for individualization and case management; access and integration; and awareness, responsiveness, and informal supports. In each case, the differences supported greater expression of each measurement area in the Stark County community.
Family-centered interviews were conducted in the Santa Cruz, California, system-of-care community and the Austin, Texas, non-system-of-care community during May and June 1999. Fourteen families participated in Santa Cruz, and 16 families participated in Austin. Figure 46 displays the results for these two communities across the four major objectives of the FCI. For each objective except impact, differences were obtained with the results indicating higher levels of implementation of key system-of-care principles at the service experiences level at the Santa Cruz system-of-care community. The lack of a difference between communities for the impact objective may reflect the fact that, at the time of the study, Austin had recently received funding from CMHS to develop a system of care. Some aspects of service delivery in Austin mirrored the approach in a system of care. Families were beginning to participate as partners in service planning and delivery, services were being offered in the community, transportation to services was being provided, and providers were easily contacted by families. Within the individual measurement areas, the greatest differences were obtained for individualization, integration, and awareness and informal supports.
These results indicate that the service experiences of families are more consistent with system-of-care objectives at the two system-of-care communities than at their matched non-system-of-care communities. In combination with the previously presented information that indicated differences in systems development across the communities, this suggests that systems of care may be having a significant impact at the level of service provision. Family-centered interviews will be completed at the East and West Baltimore communities during September and October 1999, and these additional results will provide a full picture of the expression of system-of-care principles at the service experiences or practice level within the Phase I comparison study.
An ethnography was conducted in the East and West Baltimore comparison study communities during January 1999. The purpose of the ethnography was to develop an understanding of how the urban, inner-city culture contextualizes help seeking and the provision of care, how families construct and define the challenges of raising a child with a serious emotional disturbance, and how the service delivery process is perceived and utilized from the families' perspectives.
An ethnographic team lead by an anthropologist from the Louis de la Parte Florida Mental Health Institute and comprised of locally recruited family members conducted the ethnography. Twenty caretakers from families who were divided almost equally between the system-of-care and non-system-of-care communities participated. Caretakers participated in multiple individual and group discussions with the ethnographic team to obtain the qualitative data for this portion of the study. A smaller group of service providers also participated in discussions to provide information for the ethnographies. The data were systematically coded and analyzed using a software program and interpretations were gleaned from a participatory process that involved the ethnographic team, site personnel, and the participants.
Analyses of the results by the ethnographic team indicated the following:
The implications of the results of the ethnography are that cultural factors associated with raising a child with serious emotional disturbance in an inner-city environment may have a significant overriding impact on interactions with any services, including those services offered by systems of care. The perception that a series of temporary solutions would be pursued until children reach adulthood attests to the investment that parents continue to make in utilizing the services that are available. Preventing negative outcomes of losing their children to the risks associated with "running the street" was a high priority, according to the group of caretakers participating in the ethnography. Successfully acquiring needed services appears to be associated with finding individuals who follow system-of-care principles; long-term follow-up and continued service provision at the level of individual need are important to these caretakers' longer term concerns. These latter factors are also consistent with a system-of-care approach.
Selection of the Phase II comparison communities was completed in the spring of 1999. Since then, the majority of work has focused on establishing relationships and buy-in for participation in the evaluation, particularly in the non-system-of-care communities, in preparation for data collection. Four field staff were hired and participated in
|
comprehensive training. Since their hire, most of the field staff's work has focused on setting up local field office operations. Each community has been involved in working in conjunction with key personnel from community mental health agencies to establish strategies for data collection. Data collection began in August 1999. Updates on the progress of data collection will be provided in the next Annual Report to Congress. |
|
The Phase I comparison study began data collection in August 1997. As of August 1999, data had been collected for 2 years. During that time, study enrollment was completed in four of the six communities. Additional study enrollment in two communities and follow-up data collection in all communities will continue through FY 2000 and into early FY 2001, at which time final data analyses will begin. Phase II comparison studies just began in FY 1999 and will continue into FY 2002.
The diverse nature of the Phase I comparison study sample will help make results generalizable to other system-of-care communities funded by CMHS. Younger and older children, males and females, and African-American, White, and Hispanic children are well represented in the national outcome study as well as the comparison study. Also important to generalizing findings is similarity in the severity of children's mental health disorders. In the comparison study and national outcome study samples, children's behavioral and emotional symptoms as measured by the CBCL are, on average, above the clinical range. While children's functional levels as measured by the CAFAS were higher in the comparison study, over 48 percent of children in both samples display marked or severe functional impairment. Some differences in child demographics and severity of disorder currently exist between system-of-care and non-system-of-care communities in the comparison study. If these differences persist in the final sample, future analyses will need to take into account these differences. However, a complete analysis of child and family changes over time will be appropriate only when more data are collected.
The system-level and service experiences assessments of the matching pairs of Phase I comparison study communities have shown that different service delivery approaches are being used at the system-of-care and non-system-of-care communities. System-of-care communities have implemented system-of-care principles more broadly than non-system-of-care communities. Service experiences reflect more culturally competent, community-based, and family-focused services, according to families and providers. An ethnography conducted in Baltimore, however, indicated that interactions with service delivery personnel who embody system-of-care principles may be a key ingredient facilitating change regardless of the presence of CMHS funding.
As data collection concludes in the next year, a more complete analysis, including full "triangulation" of qualitative and quantitative results, will become available.
|
| Home | Contact Us | About Us | Awards | Accessibility | Privacy and Disclaimer Statement | Site Map |
