This Web site is a component of the SAMHSA Health Information Network

Enter Keywords Choose Website This Site Mental Health at HHS All Mental Health

Online Publications Order Publications National Library of Medicine National Academies Press Publications Homepage

printer friendly page e-mail this page bookmark this page shopping cart  current or new account

This Web site is a component of the SAMHSA Health Information Network.

Longitudinal Comparison Study

The core evaluation provides information on the number and type of children served and the amount and type of services delivered. We also are able to compare the child's behavior and functioning at different time points during treatment and explore the relationship between changes in behavior and functioning with changes in child and family outcomes. However, the core evaluation does not address two critical questions:

• How much of the observed changes in the children's mental health can be attributed to the system-of-care approach beyond changes that may have occurred under an alternative service delivery approach?
• Do the observed changes in mental health outcomes persist after children and families have exited from mental health services, and do they persist longer in system-of-care sites.

The comparison study is designed to address these questions by 1) comparing changes in children's behavior and functioning in system-of-care sites to changes in children's mental health in sites which use alternative service delivery approaches; and 2) systematic data collection at set time intervals for a two-year period irrespective of whether or not children and families are still receiving services.

Site Selection

Four system-of-care sites were selected based on the following criteria:

• "Maturity." Preliminary data indicate that not all sites had fully implemented the system-of-care approach in their community that would permit a reasonable test of the system-of-care model. To conduct a credible test of the effectiveness of the system-of-care approach, we selected grantee sites that were identified as having a mature system of care. Sites were identified as mature based on system site visits and input from experts familiar with the sites.
• Feasibility of finding an appropriate comparison site match. Unique characteristics regarding the number of children served, the nature of the service delivery system, and the type of target population of some of the mature grantee sites made it unlikely to find an appropriate comparison site. For example, the state of Vermont was originally identified as one of the eligible "mature" system-of-care sites. However, Vermont implemented a statewide system of care with an emphasis on crisis intervention. Identifying a similar state as a comparison site was not considered feasible. Therefore, we limited our choices of grantee sites to those sites that had a reasonable chance to be matched with comparison communities.
• Rate of child enrollment. The number of children and families entering the system of care needed to be large enough to ensure a sample size that would yield sufficient statistical power to detect significant group differences if they existed. Thus, sites had to be able to enroll the requisite number of children to meet the required sample size within the first year of the study.

Based on these criteria, the following four system-of-care sites were selected: Stark County, Ohio; East Baltimore, Maryland; Santa Cruz County, California; and Milwaukee, Wisconsin.

To obtain meaningful results, the comparison sites had to match the system-of-care sites as closely as possible on several criteria with the exception of using a system-of-care service approach. In selecting the comparisons sites the following criteria were used:

• Geographic, demographic, and economic characteristics. Using data from the 1990 Census, population size, child age distributions, racial/ethnic proportions, per capita income, size of catchment area, the percentage of people below the poverty level, and the percentage of adults with at least high school educations, were considered in the selection of the matching comparison sites.
• Child referral patterns. Depending on the referring agency, children entering mental health services may differ greatly in terms of presenting problems, risk factors, types of disorders, and mental health needs. For instance, children recruited from schools are expected to be different from children referred from juvenile court, and child welfare. To ensure comparable recruitment patterns, the ability to recruit children in the comparison site using referral proportions similar to those in the system-of-care sites was considered.

Based on these criteria the following four comparison sites were selected: Mahoning County in Ohio to be paired with Stark County; West Baltimore as a comparison site for East Baltimore; Austin, Texas as a match for Santa Cruz; and Milwaukee where children in the juvenile justice system not served by the system-of-care initiative will be compared with children served by the "Wraparound Milwaukee II."

The selection strategy was an attempt to limit site-level differences that could masquerade as "treatment" effects. Differences between sites could limit our ability to estimate the effects of the system-of-care approach as compared to other service delivery approaches by confounding site differences with treatment effects. However, individual level data are collected to be able to detect and control for any additional differences that may exist between sites.

Site Characteristics

Sites with different geographic, demographic, and system characteristics can be affected differently by a system-of-care approach; thus, selecting a variety of sites will help us examine such differences. Each of the system-of-care sites is characterized by unique traits, and together with their matched sites, we will be able to examine issues related to the diversity in populations served, contextual/ environmental characteristics, regional representation, and catchment area size.

The Stark County (Canton)-Mahoning County (Youngstown), Ohio pair are sites that provide county-wide mental health services through a specific provider organization. In the case of Stark County, Child and Adolescent Services in Canton has been refining their health care delivery system over many years and has emerged as a nationally known example of the system-of-care philosophy. In Mahoning County, children will be recruited from a similar agency (D & E Counseling Center) in Youngstown. Both communities serve a predominantly Caucasian and African American population.

The Baltimore sites offer the opportunity for an in-city comparison study. The system-of-care site, East Baltimore, is a neighborhood-basedproject that limits its catchment area to a specific neighborhood. West Baltimore mirrors the demographics of East Baltimore with both neighborhoods having a mostly urban African-American population. However, children in West Baltimore attend schools in their own neighborhood and receive mental health services from a different mental health agency than children in East Baltimore which guards against potential contamination effects of the East Baltimore system-of-care initiative. Both sites will have to respond to a recently implemented city-wide managed care initiative. However, since managed care will affect mental health services in East and West Baltimore at the same time, the effects on children and families should be similar which will reduce the possibility of confounding managed care effects with service delivery system effects. East Baltimore has liaison staff stationed in the different child serving agencies and the schools. These liaisons identify and refer children for services to the East Baltimore Partnership. This recruitment approach will be mirrored in the West Baltimore site.

Santa Cruz County, California and Travis County, Texas is the only between-states comparison. Because California is moving toward a statewide adoption of the system of care, it was not possible to find an appropriate match within the state. We selected Travis County in Texas as the comparison site because it has similar demographic characteristics; in particular, both sites serve a large percentage of Mexican Americans adding to the diversity of populations included in the comparison study as a whole. At both sites children will be recruited through the county mental health agency.

Milwaukee, Wisconsin offers yet another unique population and comparison study. Participants will include exclusively children who have been involved with the juvenile justice system. They will be recruited into the study after they have completed services with Wraparound Milwaukee II, the system-of-care initiative, or after they have completed regular probation, probation with therapeutic intervention, or residential treatment. This strategy will provide the opportunity to study the long-term effects of different court ordered juvenile justice interventions.

Child and Family Sample Selection

Sample Size

The comparison study will include a total sample of 1,250 children and families. A sample size of up to 200 children and their families will be recruited in each of the sites with the exception of Milwaukee where 75 children will be recruited from the system of care and another 75 children will be recruited from the non-system-of-care intervention.

Selection Criteria

Children between the ages of 6 and 17.5, with a new episode of treatment, and an Axis I, DSM IV diagnosis will be eligible for inclusion into the study. In addition, children must have scores within the clinical range as measured by the CBCL and/or the CAFAS. Other screening criteria that will be used for sites that do not use the CBCL and CAFAS as part of their screening and intake process include: multiple-agency involvement, history of out-of-home placement or being at risk for out-of-home placement, and participation in an educational program for children with serious emotional disturbance. At baseline, however, all children and their primary caregiver will be administered the CBCL and the CAFAS irrespective of whether they already were administered during intake or assessment. If the resulting scores are not within the clinical ranges, these children will be dropped from the sample, and new children will be systematically included in the baseline data collection until the sample quota is filled.

Including only children who are entering a new episode of treatment is considered important so that changes in behavior and functioning, that usually occur after treatment is initiated, can be fully captured. Children younger than 6 years of age are not included in the study because for this age group it is difficult to distinguish developmental from psychological issues, and many of the assessment instruments are not suitable for very young children.

In the system-of-care sites, all children and families who meet the above criteria will be selected for possible inclusion until sample size is reached. In the comparison sites, a systematic quota sampling approach will be used to reach sample size. Using the demographic profile of the respective system-of-care sample as a base, comparison site children will be systematically selected to proportionally represent the demographic distributions (e.g., gender, age, ethnicity) of the paired system-of-care site. The demographic characteristics in each site's sample will be constantly monitored to ensure that the matching samples do not start to diverge.

Longitudinal Data Collection Methodology

Sources of Data

Data will be collected from two sources: 1) face-to-face interviews with children and their primary caregiver, and 2) agency management information systems (MIS). Data collected from caregivers and children include child and family descriptive information, child behavioral and functional assessment information from the CBCL and the CAFAS, previous living situations, and satisfaction with mental health services. This information is similar to the information obtained in the core evaluation. Additional questions will be asked of children regarding substance use and relationships with family members and friends. Youth who are at least 16 years old will also be asked about their employment experiences and, if appropriate, about independent living skills.

The agency data collection will include service use and service cost data from child-serving agencies, juvenile justice system involvement data, and academic performance, attendance, and behavioral data from schools.

Methods

Interviews will take place in the home or any other place that is convenient to the family. Interviews will be conducted by data collectors who are trained in administering structured interviews. This is a departure from the core study where interviews are predominantly conducted by clinicians. The use of designated data collectors will enable the interviews to be scheduled independent of clinic visits, or of clinicians conducting home visits. This strategy will guarantee that all information can be collected in a similar way at set time intervals.

Data will be collected at intake into treatment, and at six, twelve, eighteen, and twenty-four months after intake. Interviews will be completed within a two-month window of their official due date. If children and their families move from the geographical area and it is no longer feasible to conduct face-to-face interviews, we will continue data collection using telephone interviews in an effort to reduce data attrition. Agency data will be collected once or twice a year depending on the timeliness of the data. For instance, it may be more cost effective to collect data from schools after each semester but only extract information from juvenile court once a year.

Field Staff and Macro Study Team Responsibilities

Unlike the core evaluation, the field staff in the comparison sites are Macro employees; they are directly supervised by the Macro evaluation staff in Atlanta. In each study site, staff includes a field coordinator to manage all data collection activities at the site, an administrative assistant to assist with office management and data collection, and a staff of interviewers.

Current Progress in Longitudinal Comparison Study Sites

Much of the recent work has involved setting up the sites to prepare for data collection. Key staff have been hired and training of field staff has been completed in three of the four comparison pairs (Stark County-Mahoning, Santa Cruz-Austin, and East-West Baltimore). All six of these sites have begun to collect data, and a computerized tracking system that electronically sends information back and forth between the sites and Macro staff in Atlanta has been successfully implemented. Preliminary results are anticipated by late summer of 1998 and will be reported in the next Annual Report to Congress. Efforts are still underway to determine the feasibility of conducting a comparison study in Milwaukee.

Table of Contents | Previous | Next