This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
2001 Annual Report to Congress on the Evaluation
|
|
CHAPTER SUMMARY In summary, comparison data from two system-of-care communities funded in 1997 and their matched communities indicated that
|
National evaluation data collected to examine systems implementation and service utilization and associated service costs have yielded greater understanding of the types and patterns of services children and families receive within service systems. Yet, as grant communities develop their infrastructure and implement services according to system-of-care principles (as described in Chapter II), these principles may not be infused into the practices of individual providers and the service experiences of children and families. The rich services and costs data obtained from the Ohio communities in the Phase I comparison study (as described in Chapter III) have provided a detailed perspective on services delivered in two different service systems, and the System of Care Practice Review substudy provided an initial examination of selected families' service experiences at the practice level. To gain a broader understanding of the extent to which providers deliver services and interact with children and families in their practices according to the system-of-care principles, a new component was added to the evaluation of communities receiving grants in 1997. The second comparison study (Phase II), conducted with system-of-care communities funded in FY 1997, began in spring 1999. To better understand factors that influence outcomes for children and families, this second comparison study was designed to include examination of the service experiences of children and families at the practice level as well as the perspectives of child-serving providers on the ways in which they deliver services. As in the Phase I comparison study, the central focus of the Phase II comparison study is to examine whether outcomes for children and families receiving services within systems of care differ from those of children and families receiving services in more traditional service delivery systems.
By examining service experience and delivery at the practice level on a broader scale than in the Phase I comparison study, this second comparison study offers the opportunity to examine the extent to which service experiences are consistent with system-of-care principles and program implementation. Further examination of service experience will provide important information about the similarities and differences in system-of-care and non-system-of-care communities.13 An overview of children enrolled in this comparison study and preliminary information about the assessment of service experience at the practice level are presented here.
The longitudinal design of this study is the same as that of the Phase I comparison study: Children and families are assessed every 6 months for up to36 months after intake into the study (whether they remain in services or not). As in the Phase I comparison study and the core evaluation of all funded communities, the Phase II comparison study includes interviews with all caregivers and with youth aged 11 years or older about child and family outcomes, system-of-care assessments of each community's service system, and examination of service use and associated costs from data extracted from management information systems (MIS's) in each community.
In this study, the topics of service assessment and experiences are added to child and caregiver interviews at each follow-up data collection point. Caregivers are asked additional questions about which services they received as well as service locations, frequency, intensity, and duration during the 6-month period prior to each follow-up interview. Caregiver reports of services received were added, in part, to supplement data available from local MIS's, and to assure that information about services would be available for each child in the study. This new data collection effort includes the administration of two new service experience measures with each follow-up interview. One instrument assesses experience within the system as a whole and is administered to caregivers, and the other assesses experiences with individual providers and is administered to both caregivers and youth aged 11 years or older. An intensive examination of service experiences at the practice level is being conducted with a subset of children enrolled in the comparison study, which includes both caregivers' and providers' perceptions of received services. In addition, another substudy assesses provider attitudes and practices in the system-of-care and non-system-of-care communities through community-wide survey data.
The same criteria used to select communities for the Phase I comparison study (see Chapter III) were used for this Phase II comparison study (service delivery approach; geographic, demographic, and economic characteristics; rate of child enrollment; child referral patterns; and willingness to participate). In addition, the capacity of local mental health agencies to enroll children into services was also taken into account in the selection process.
Two communities among those funded in 1997 as well as two matching communities that do not have systems of care in place were selected for the Phase II comparison study in fall 1998. The two grant communities selected represent different regions of the country with differing socioeconomic characteristics. Human Services Region III, based in Kearney, Nebraska, one of the system-of-care communities, was paired with the non-system-of-care community, Human Services Region IV, based in Norfolk, Nebraska. Jefferson County (Birmingham), Alabama, was selected as the other system-of-care community and was matched with Montgomery County (Montgomery), Alabama. Consistent with goals of the comparison study, the system-of-care and non-system-of-care communities differ in their approaches to serving children and families.
The comparison pair in Nebraska offers a unique opportunity to examine the impact of services within a rural State where the system-of-care and non-system-of-care communities each span 22-county areas of approximately 15,000 square miles. The grant recipient, the Central Nebraska Initiative for Families and Youth (CeNIFFY), housed at the Human Services Region III Behavioral Health office in Kearney, changed its name to Nebraska Family Central during FY 2001 (hereafter called Region III). Referrals to the Region III office come from the departments of parole and probation, schools, the Department of Health and Human Services, self-referrals, and referrals from other child-serving agencies. A unique aspect of the project is the Professional Partnership Program (PPP). Although this is a statewide program specifically designed to serve children and youth with serious emotional disturbance who are at risk for out-of-community placement, it has been greatly enhanced by the system-of-care grant program. PPP staff serve as case managers and use a wraparound process to assist families in building a team dedicated to flexible, nontraditional, and unconditional care. Available wraparound services include respite, transportation assistance, and recreational therapy. Additional services include outpatient, intensive inpatient, acute inpatient, day treatment, psychiatric residential treatment, and multisystemic therapy.
Human Services Region IV, the non-system-of-care community, encompasses four service delivery centers located in Sioux City, Columbus, Norfolk, and O'Neill in northeastern Nebraska (hereafter called Region IV). In this traditional service delivery system, children are referred directly to the mental health department and a large number of contracted private providers throughout the 22-county service area. The majority of referrals to mental health services come from child protective services, juvenile justice agencies, and self-referrals. Available outpatient services include individual, couples, family, and group counseling; psychological and psychiatric evaluations and consultation; pre-admission screens; aftercare; emergency services; and medication evaluations. Substance abuse services, home-based therapy, and therapeutic foster care also are provided. For children with serious emotional disturbance, the region has limited infrastructure and resources for providing services locally; consequently, many children with serious emotional disturbance become wards of the State and are sent to residential facilities outside the region. Children and families have been recruited into the study from two county-based agencies located in different cities and from several private counseling offices located throughout the region. During FY 2001, recruitment was expanded to include children receiving services through the PPP in Region IV in order to reach those children with greater symptom severity. As in Region III, the PPP serves children at risk of out-of-region placement; however, in Region IV, the PPP is much smaller, with a limited number of counselors and case managers and without the array of services available to children in Region III. Not all children served through PPP in Region IV receive wraparound services. Study recruitment from PPP has been limited to children who are not receiving these services. Consequently, it was determined that recruitment from the PPP would not conflict with study goals. In general, agencies in Region IV operate independently, and limited efforts are made to coordinate among agencies.
In Jefferson County, the system-of-care community, the Jefferson-Blount-St. Clair Mental Health and Mental Retardation Authority is the grant recipient. The Jefferson County Community Partnership (JCCP), the governing body of the grant program, brings together the grant recipient with other community agencies in the COPE (Children Overcoming Problems Everyday) program. A range of services is available through JCCP, including assessment, case management, crisis follow-up, liaison or collateral services, in-home services, supportive therapy for foster parents, respite beds, and case management for transition into adulthood. Outpatient, day treatment, inpatient, and emergency services are obtained through contractual arrangements with other community providers. Children and families enter JCCP services through school-based diagnostic and evaluation units located in two regular middle schools, two alternative middle schools, Family Court, and the Department of Human Resources. Diagnostic and evaluation specialists conduct intake and assessment and refer children to appropriate COPE teams stationed in these locations. Each COPE team includes a combination of mental health liaisons, diagnostic and evaluation specialists, case managers, outpatient therapists, substance abuse counselors, domestic violence counselors, psychiatric consultants, in-home services teams, and parent advocates. These teams work collaboratively with partnering agencies to ensure that children and families receive needed services.
In Montgomery County, the non-system-of-care community, children are recruited from a similar county-based agency, the Montgomery Mental Health Authority, that serves a four-county area. Mental health services are coordinated through one main center and two satellite centers. All children requiring mental health services are referred directly to the Montgomery Mental Health Authority. Outpatient mental health services are provided on-site by the outpatient therapy team. The service array provided through the center includes emergency/crisis management, therapy, case management, in-home intervention, in-home therapy, court liaisons, pre-hospitalization screening, psychiatric services, medication monitoring, and diagnostic and evaluation testing. School-based intake, assessment, and counseling services were initiated in 2000 with counselors stationed in three alternative schools, and crisis services were provided by an additional counselor to four regular schools in Montgomery County; however, children are not directly referred from school programs to the comparison study. This community differs from the system-of-care model in its mental health agency structure and limited case management. Although there is an array of services provided by the county mental health agency, there is limited integration of these services within the agency and with services offered by other agencies throughout the community. In addition, there is only one case manager for the mental health agency, and not all families have access to case management. These limitations may reduce awareness of available resources and accessibility to other community-based services needed by families.
In August 1999, Phase II comparison study enrollment began. A goal was set to enroll 225 children in each of the four communities. This goal was established based on power estimates appropriate for the longitudinal study design. It was calculated as the minimum number of children needed to detect changes over time in the outcomes of children and families served within each community over the course of the study. In the system-of-care communities, all children who are between the ages of 6 and 17.5 at the time they enter services are eligible for participation in the comparison study.
In the non-system-of-care communities, it is necessary to locate providers serving children with behavioral and emotional problems of a severity similar to those receiving services in the system-of-care programs in order to equate the samples. In these communities, a screening process to determine eligibility for study participation is implemented to ensure that only children with serious emotional problems are enrolled into the study. As in the Phase I comparison study, children are selected according to clinical status, service history, risk or history of out-of-home placement, or participation in special education for serious emotional disturbance. Enrollment across communities is closely monitored to maintain similarity of sample characteristics for each matched pair. As sample differences occur, adjustments in recruitment strategies (such as those described above for Region IV) are made.
As of August 24, 2001, intake interviews have been completed for 542 children enrolled in the Phase II comparison study across all communities. In addition, 297 6-month, 176 12-month, and 81 18-month interviews have been completed (see Table 13). Currently, more children have been enrolled into the study in Region III, Nebraska, than in the other three communities. In Region III, enrollment began in early August 1999 and was accelerated because protocols for service intake were well established at that time. In the other communities, recruitment began somewhat later and progressed more slowly at first. In addition to improvements made in the service intake process in these communities, changes to study enrollment locations and protocols have helped recruitment rates increase in the other three communities. Baseline enrollment is scheduled to end in September 2002, and it is anticipated that enrollment goals will be met in each community.
Maintaining families over the course of the comparison study is essential in order to generalize results to the larger target population. Overall, completion rates, calculated only on those families that have reached each follow-up data collection point at this early stage of the study, are high for 6-month (86 percent), 12-month (90 percent), and 18-month (94 percent) follow-up interviews. These rates compare favorably to those of similar studies (e.g., Bickman, Lambert, Andrade & Peņaloza, 2000; see Chapter III, page 49, for information on completion rates in comparable studies). The first study participants reached their 24-month interviews in August 2001. Follow-up data collection is expected to continue through September 2003.
Analyses presented in this section include baseline interview data for 442 children and 6-month data for 212 children. As displayed in Table 14, children and families initially enrolled in paired communities (Region III and Region IV, Nebraska; Jefferson County and Montgomery County, Alabama) share some similar characteristics; however, some demographic differences are seen. In Nebraska, Region III, the system-of-care community, has enrolled significantly more males (70.3 percent) than Region IV (52.9 percent). Children in the two Alabama communities differ slightly in age: Children recruited early into the study in Jefferson County, the system-of-care community, are slightly older (13.5 years) than those in Montgomery County (11.9 years), the non-system-of-care community. This difference may be explained by the focus on service provision in middle schools in Jefferson County, whereas children of any age receive services through the mental health agency in Montgomery County. Differences of this type, seen early in the study, reflect less than half of the total expected sample. Enrollment is monitored closely and will continue to be monitored throughout the study to assure that samples drawn from system-of-care and non-system-of-care communities are closely matched. Demographic differences such as the ethnicity of children served in the Alabama pairs and the Nebraska pairs reflect the service populations of these communities. Children are matched across each pair and not across all four communities included in this study.
In all four communities, most children initially enrolled in the study are in the custody of their biological mothers, with the second largest group in the custody of two parents (either both biological parents or one biological parent and one stepparent; see Table 15). The most notable difference in custody arrangements among communities is the high percentage of children in Region IV who are wards of the State (27.6 percent). This percentage is consistent with information presented in the above description of the Region IV service system (see Community Selection, pp. 147-8), where it is explained that children in Region IV (the non-system-of-care community) with the most severe behavioral and emotional problems often are made wards of the State for placement in residential treatment facilities out of the region. The percentage of families with annual incomes above and below $15,000 is similar in the two Nebraska communities. In the two Alabama communities, a greater percentage of families have incomes below $15,000 in Montgomery County than in Jefferson County. This family income is below the annual income of $17,650 for a family of four established as the Federal poverty level for the year 2001 (DHHS, 2001a).
Children enrolled in the comparison study at this early stage of the study differ somewhat in their service intake characteristics (see Table 16). In the Nebraska pair, more referrals come from schools (27.3 percent) and other mental health agencies (37 percent) in the system of care (Region III) than in the non-system-of-care community (Region IV; 6.9 percent and 9.2 percent, respectively), whereas more referrals are made from social services (43.7 percent) or are self-referrals (23 percent) in Region IV than in Region III (10.3 percent and 6.7 percent, respectively). Specific child and family risk factors are largely similar across all communities; however, more children enrolled in Region IV have a history of drug use (22.4 percent) than in Region III (15.6 percent), and more children in Region III have a biological family member who has had a mental illness (47.4 percent) than those in Region IV (32.5 percent). The average number of child and family risk factors do not differ in each pair; however, they are slightly higher in the Nebraska pair (Region III: 4.0; Region IV: 3.6) than the Alabama pair (Jefferson County: 2.6; Montgomery County: 2.8).
At this early stage of study enrollment, diagnoses assigned to children differ to some extent in each pair of communities. In Montgomery County's non-system-of-care service system, more children have received diagnoses of oppositional defiant disorder (55.9 percent) and ADHD (55.9 percent) than in Jefferson County's system of care (8 percent and 36 percent, respectively). In Nebraska, more children have been diagnosed with ADHD in Region III's system of care (42.7 percent) than in Region IV's non-system-of-care service system (12.3 percent), and more children have been diagnosed with conduct-related disorders in Region IV (9.9 percent) than in Region III (2.1 percent).
Approximately one-third of all children entering services had not received any previous services in the two Alabama communities (Jefferson County: 32.9 percent; Montgomery County: 30.3 percent). In Nebraska, only 7.5 percent of children in Region III's system of care had not previously received services, while 18.4 percent of children in Region IV (the non-system-of-care community) had not had any services in the past. Among those children who had received at least one service, some differences were seen between matched communities in the types of services received (see Figures 64 and 65). In Nebraska, more children in Region III had received school-based (64 percent) and day treatment services (14 percent) than those enrolled in the study in Region IV (45 percent and 5 percent, respectively). However, more children enrolled in Region IV previously had received some type of therapy for substance abuse (14 percent) than those in Region III (7 percent). In Alabama, more children in Montgomery County (the non-system-of-care community) had received outpatient services (61 percent) prior to study entry than in Jefferson County's system of care (45 percent).
In the Phase II comparison study, as in the Phase I comparison study, children's emotional and behavioral problems are assessed by administering the Child Behavior Checklist (CBCL; Achenbach, 1991a) to caregivers at each data collection point. To obtain youth perceptions of their emotional and behavioral problems, the Youth Self-Report (YSR; Achenbach, 1991b) is administered to youth aged 11 years or older. Impairment in functioning, an important indicator of the extent to which children's behavioral and emotional problems impact functioning in the home, school, and community, is assessed using the Child and Adolescent Functional Assessment Scale (CAFAS; Hodges, 1990b). In the Phase II comparison study, CAFAS total scores are calculated using all eight CAFAS subscales (Home role, School/Work role, Community role, Behavior Toward Others, Moods/Emotions, Self-Harmful Behaviors, Substance Use, and Thinking). Children's behavioral and emotional strengths are assessed using the Behavioral and Emotional Rating Scale (BERS; Epstein & Sharma, 1998). In addition to the demographic characteristics described above, intake scores on each of these measures are used to monitor children's levels of behavioral and emotional problems as well as functional impairment in each community's study sample, and serve as an important source of information for determining adjustments to recruitment strategies. More information about each of these measures can be found in Appendix B. Mean total scores on the CAFAS, CBCL, YSR, and BERS are provided for each community in Table 17.
Children's CBCL Total Problems T-scores do not differ greatly across all four communities, but they do show significant, albeit small, differences between matched pairs. Although CBCL scores are slightly higher for children enrolled in the study in Montgomery County (the non-system-of-care community) and in Region III (the system-of-care community), all Total Problems T-scores are above 63, indicating that children across all communities have significant emotional and behavioral problems on average. YSR Total Problems T-scores are typically lower than CBCL Total Problems scores, as is also the case in this study. A slight difference in YSR mean Total Problems T-scores is observed between children enrolled in Nebraska's matched pair communities.
Average total CAFAS scores among children enrolled in the study in the system-of-care communities are somewhat higher than in the non-system-of-care communities, indicating that these children may be exhibiting slightly greater impairment in functioning. In each community, however, the average total CAFAS scores are above 90, indicating that the average level of impairment in functioning is in the marked or severe range. In the Alabama communities, this difference may be due to age differences and referral sources of children enrolled in the study: Children enrolled in Jefferson County are somewhat older, and a majority are referred by juvenile justice agencies. As a result, they may be exhibiting greater impairment in functioning in several CAFAS domains.
Efforts to modify study recruitment in Montgomery County were initiated during FY 2001 so that the ages of children enrolled in the study could be matched more closely. Greater equivalence in children's ages and levels of functional impairment also may be obtained in the future as recruitment and sample monitoring continues.
Similarly, children's CAFAS scores are higher in the system-of-care community (Region III) than in the non-system-of care community (Region IV) in Nebraska. This may be explained by the high percentage of children (27 percent) who are wards of the State in Region IV. Many of these children were in residential treatment facilities prior to, or at the time of, study enrollment. Because these environments are far more controlled, children living in these settings cannot exhibit the same levels of functional impairment in their home, school, and community role performance as children living at home and attending school in the community. In addition, higher CAFAS scores generally are found in male populations, and Region III's system-of-care includes a significantly greater proportion of males.
Finally, children's strengths, as measured by the BERS, are largely comparable across communities, although children's strengths are slightly higher among children enrolled in Region IV (the non-system-of-care community) than in Region III.
At this early stage of the Phase II comparison study, 6-month data are available on only a smaller subsample of children. Consequently, it is not possible to examine changes in children's clinical characteristics within each community. To provide preliminary information about progress among children enrolled in the study, data from the system-of-care communities (Jefferson County and Nebraska Region III) and the non-system-of-care communities (Montgomery County and Nebraska Region IV) were combined. Among children for whom 6-month interviews were completed, average total scores on the CAFAS, CBCL, and YSR were lower at 6 months than at study intake, indicating declines in functional impairment and behavioral and emotional problems. BERS scores were somewhat higher, indicating improvement in children's strengths. These scores are presented in Table 18 for system-of-care and non-system-of-care communities.
Information about services received from intake to 6 months was collected using the Multi-Sector Service Contacts form (MSSC; see Appendix B), which documents caregivers' recollections of the specific services their children and families received. According to caregiver reports, the 138 children in the system-of-care communities had received on average 5.2 (range 1-12) services in their first 6 months of service delivery, while the 58 children who were served in the non-system-of-care communities had received on average 4.0 (range 1-10) services (t = -3.195, df = 194, p = .002).
For analysis purposes, the services included in the MSSC were grouped into eight categories: assessment, case management, outpatient therapy (individual, group, or family therapy), medication treatment and monitoring, restrictive services (inpatient hospitalization, residential treatment center, therapeutic or wilderness camp), day treatment, therapeutic group or foster home, recreation, and respite or family support. The percentage of children receiving these services in system-of-care and non-system-of-care communities during the first 6 months in the study are presented in Figure 66.
Children in system-of-care communities were significantly more likely to receive case management; medication treatment and monitoring; and day treatment, foster, or group home services than those in the non-system-of-care communities. Children in the non-system-of-care communities were more likely to receive individual, family, or group therapy. These preliminary differences between the two types of communities are consistent with implementation of systems of care in which intensive case management is central to service provision. That children in system-of-care communities received more services through therapeutic group or foster homes and day treatment is also in keeping with the implementation of systems of care designed to keep children in least restrictive settings available in their communities.
The greater percentage of children receiving medication treatment and monitoring services in systems of care may be an indication that children in these communities are obtaining more appropriate treatment with medication and that their medication use is better monitored. Preliminary findings indicate that this may be the case. In the system-of-care communities, a greater percentage (22 percent) of children not previously taking medication for behavioral or emotional problems had received medication-related services during the first 6 months than in the non-system-of-care communities (6 percent). In addition, of children who had taken medication for their behavioral or emotional problems at some point prior to intake, a higher percentage (77 percent) of children in system-of-care communities received medication treatment or monitoring services during their first 6 months in services than in the non-system-of-care communities (46 percent). This finding suggests that for children previously on medication closer attention was given to medication usage in system-of-care communities. In the Phase I comparison study, children in Stark County, the system-of-care community, received a greater amount of medication monitoring services, but these services were not received by a greater percentage of children than in Youngstown, the comparison community.
CBCL and CAFAS scores of children who received a particular type of service and those who did not were compared within system-of-care and non-system-of-care communities. In the non-system-of-care communities, there were no differences in the total CAFAS scores or CBCL Internalizing or Externalizing syndrome scores of children who had or had not received services for seven of the eight types of services. Children who received inpatient or residential treatment services did have significantly higher total CAFAS scores (128.6) than those who did not (84.0; t = -2.239, df = 57, p = .03). These children also had higher CBCL Internalizing (71.7) and Externalizing (74.4) syndrome scores than those who did not (62.0 and 66.2, respectively) receive this service (Internalizing: t = -2.21, df = 56, p = .03; Externalizing: t = -2.02, df = 56, p = .049).
In the system-of-care communities, greater differences in clinical scores were seen among children receiving and not receiving these various services. Differences in CBCL Internalizing and Externalizing syndrome scores can be seen in Figures 67 and 68. Children who received medication treatment and monitoring services had higher Internalizing CBCL scores, as did those children who received day treatment/group or foster home, residential or inpatient, or outpatient therapy services. Children who received day treatment/group or foster home, residential or inpatient, or outpatient therapy services also had higher CBCL Externalizing scores, as did those children whose families received respite or family support services. Similarly, children who received medication services also had higher total CAFAS scores (127.3 vs. 107.5; t = -3.25, df = 134, p = .001), as did those who received day treatment/group or foster home, or residential services (Day treatment: 143.8 vs. 113.7; t = -3.91, df = 133, p < .001; Residential: 146.3 vs. 114.6; t = -3.74, df = 133, p < .001). That differences are seen in the level of severity among children who do and do not receive specific services in system-of-care communities that are not seen in non-system-of-care communities suggests that greater attention is given to the level of need of a particular child when making service-related decisions in system-of-care communities. In addition, these differences reflect the goals of systems of care to enable children to be served while remaining in the home or community and out of residential settings, except when these settings are most appropriate to children's needs.
These differences in the clinical severity of children receiving services suggest that in system-of-care communities services are provided more appropriately to those children most in need of the service. Mean CBCL and CAFAS scores of children receiving day treatment or group home or residential inpatient services were similar, suggesting that less restrictive day treatment or group home services are being provided as alternatives to residential placement so that children can remain in their communities. Further examination of differences between communities will be possible as more data are collected.
While examining the array of services received in these communities is important to describing system-of-care implementation, there is a increasing need to better understand what occurs at the practice level and to assess how families actually experience the delivery of services they receive.
Significant evidence has accumulated over the past decade indicating that the system-of-care approach has had a positive impact on how services are delivered to children and families with mental health needs (Chamberlain & Reid, 1991; Green, Johnson, & Rogers, 1998). Although there is substantial information about how systems develop over time, there continues to be a lack of convincing evidence about service effectiveness at the individual level (Farmer, 2000; Pumariega & Glover, 1998). The SGR emphasizes the importance of improving our understanding of family's experiences with services, specifically so that information can be fed back to the system to have a positive impact on available services, service delivery systems, and policy decisions. Understanding how families actually experience the services they receive is critical to understanding how system-of-care principles are applied and experienced by families at the service delivery level.
To enhance understanding of whether services are received as planned, recent studies have examined treatment fidelity in individualized, community-based interventions targeting children with serious emotional disturbance and their families. Fidelity to service delivery approaches has received empirical attention, including approaches used in system-of-care communities such as wraparound, a coordinated approach to planning the provision of formal and informal services within the community (Burns & Goldman, 1999; Clark et al., 1998); Multi-Systemic Therapy (MST), an intensive and time-limited, home- and family-focused treatment for children referred from the juvenile justice system (Henggeler, 1999); and Family-Centered Intensive Case Management (FCICM), an individualized care framework designed to support the family in caring for their child at home (Evans, Armstrong, Kuppinger, Huz, & Johnson, 1998). Yet, limited evidence is available to demonstrate that services are being delivered in a way that reflects the principles of the system of care. The concern is that if services are not being delivered in ways that are consistent with the system-of-care principles, it may not be realistic to expect strong positive outcomes (Bryant & Maxwell, 1997).
In order for service provision to be effective, children and families must first receive these services as intended. If families encounter obstacles in their service experiences, they may be more likely to discontinue services. For example, families may discontinue services for reasons ranging from difficulties in accessing services they need to receive, to frustrations they experience because of a lack of understanding by providers of their cultures or a lack of respect for their opinions. System-of-care programs are intended to provide culturally competent services according to individual child and family needs, make services accessible within the community, provide services in the least restrictive settings to meet the child's needs, and promote coordination among service providers and agencies. By delivering services in this manner, systems of care should be able to provide the most appropriate and needed services, enhance engagement with services, and provide more positive service experiences for children and families. Yet, although system change to implement service delivery according to system-of-care principles may occur in communities, this change may not be reflected in the practices of individual providers in their interactions with children and families.
In the Phase II comparison study, three areas of assessment were included to examine service experience, fidelity to a treatment plan, and providers' perspectives of service delivery. In addition to obtaining information about the services received by children and families and their satisfaction with these services, two measures were added to assess whether service experiences reflected system-of-care principles. A substudy of a limited number of families in each community and their providers was included to examine fidelity to a service plan and service experiences that may affect implementation of that plan. Finally, a survey of providers in the system-of-care and non-system-of-care communities also was administered to yield new information about similarities and differences in providers' attitudes and practices in these settings.
To understand whether families' service experiences as a whole and their experiences with individual providers are in accordance with system-of-care principles, two new measures were developed: the Experience with Service System Questionnaire (ESSQ) and the Service Experience Questionnaire (SEQ). These instruments were developed following standard instrument development practices that included a review of the literature to guide construct and item development; several levels of expert review to obtain recommendations regarding methodology, constructs, item clarity, instrument format, and cultural competence; and revisions based on these recommendations. Reviewers included individuals familiar with systems of care and the service experiences of children with serious emotional disturbance and their families.
Data collection using these measures began in spring 2000, as communities began to administer 6-month interviews. Initial administration of instruments led to slight changes in item wording for ease of interviewing and to address minor issues related to the interpretation of items within specific communities.
This 26-item instrument is administered at each follow-up assessment to caregivers of children who have received services related to behavioral or emotional problems during the previous 6 months. The ESSQ assesses the extent to which caregivers experienced the overall service system as embodying key principles of the system-of-care model. Caregivers are asked to rate their experiences with the service system as a whole. For example, when the respondent is posed questions about the effectiveness of service coordination (a key system-of-care principle), the coordination and linkages across the various child-serving sectors (e.g., mental health schools, juvenile justice, child welfare) are rated. Items address how well the system of care has provided services that are family focused, individualized, culturally competent, accessible, and coordinated, in keeping with the system-of-care principles (see Chapter II for an in-depth description of these principles). At 6 months, 189 caregivers completed the ESSQ. Examination of measurement qualities indicates good internal reliability of the measure (Cronbach's alpha = .98). Additional 6-month interviews need to be completed in each community before comparisons of service experiences can be made.
The SEQ assesses the extent to which the experiences that caregivers and youth have with individual providers are consistent with system-of-care principles. The SEQ assesses the extent to which the services received match the individual needs of the child; focus on the family; encourage involvement in service planning and decisionmaking; and are culturally competent, accessible, and coordinated with other services. The SEQ is administered to caregivers of all children and to youth aged 11 or older at each follow-up data collection point if the child has received services during the previous 6 months.
The SEQ has two parts. Because case management is vital to systems of care, caregivers and youth rate their experiences with their case manager in the first part. In non-system-of-care communities, case management services often are not directly provided; therefore, caregivers and youth in these communities rate experiences with their primary service provider in this part. In the second part, caregivers and youth rate experiences with the service provider from whom the child received the most services, in addition to the case manager or primary provider, during the 6-month period. The caregiver version of the SEQ has 39 items, and the youth version has 31 items. Administration protocols include methods to assure that caregivers and youth report about their experiences with the same provider in each section. Data for 159 caregivers and 87 youth are available for analysis at 6 months. Initial examination of these measures has also yielded strong internal reliability, with Cronbach's alphas of .98 and .97 for the two parts of the caregiver measure and .96 for each part of the youth measure.
Without understanding how well an individual client's actual treatment adheres to what was planned and intended, it may be difficult to explain evaluation findings. A system of care may have key components (e.g., case management, multidisciplinary treatment teams, wraparound services) in place and may make a wide array of service options available, but clients may not receive those services as intended. That is, if the study does not find the desired outcomes, such as improvements in child functioning, it is difficult to know whether the system-of-care theory failed or whether the theory was not truly implemented as designed. To some extent, this was the case with the Fort Bragg Evaluation Project (Bickman et al., 1995). The evaluation established that the service demonstration was developed in accordance with the program's theory and that it was, as a whole, implemented well (Heflinger, 1996). However, little was learned about whether individual children and families received individualized treatment regimens recommended by a multidisciplinary treatment team.
The service experience and treatment adherence substudy was designed to investigate the congruence between service or treatment plans and what is actually experienced by children and families. In addition, it examines the factors that may impact adherence to a service plan. The perspectives of both caregivers and providers on service experiences are obtained to examine the extent to which each finds that shared experiences reflect system-of-care principles and values.
Whether or not families adhere to service plans may occur for a variety of reasons, and it is important to identify the contributing factors. These factors may be present at multiple levels, including the child, family, service, and system levels, and may help explain why certain outcomes occurred (e.g., improved child functioning, level of client satisfaction). This information can also provide clues about how to improve adherence to care plans. Also of interest are the "informal" supports used by families in the care of their children, how those supports interact with and affect adherence to service plans, and the relationship of these variables to outcomes.
Families are recruited into the substudy at the time of their baseline interview, shortly after entry into services. After soliciting caregiver participation and obtaining approval to contact the provider about study participation, separate initial interviews to review the treatment planning process are conducted with each caregiver and provider. Caregivers then keep a biweekly diary of their service encounters, and providers keep a similar log that also includes recording the occurrence of coordinating activities conducted for the child or family. Caregivers and providers are then interviewed by telephone every 2 weeks for the next 16 weeks to report on their service experiences, including (a) which services and supports were provided, (b) who received the services, (c) appraisals of service provision according to system-of-care principles, and (d) attributions about why or why not a service occurred. A case record review is conducted once at the end of the substudy to assess established service plans, attendees at service planning meetings, services received, documentation of service coordination, explanations for nonparticipation, and any changes in service plans.
In total, 200 families (50 in each community) will be recruited into the substudy. After development of protocols and data collection tools, one caregiver and one provider in each of three communities were enrolled into a pilot of the substudy in winter 2001. Due to staff transition in the fourth community, it was not possible to pilot the study in that community; however, the first family enrolled in that community is serving to pilot-test the study there. While study implementation proceeded smoothly during the pilot phase, new information was gained and issues were raised about aspects of the study's methodology. Based on feedback from staff implementing the study in each community, slight changes were made to enrollment protocols, item wording, and questionnaire formats to reflect conditions encountered in the communities more realistically. Record extraction procedures were also pilot-tested to determine the extent to which the desired information could be extracted from children's files, despite the fact that significant variations were found in the file content and format across communities. The first study participants were enrolled in late spring 2001 in all four communities. At this writing, 21 families have been enrolled into the substudy. Future annual reports to Congress will provide data analyses from these families and additional participants.
In addition to understanding whether families experience services in accordance with system-of-care principles, it is important to understand the extent to which service providers understand these principles and are able to apply them in their practices. Providers attitudes about system-of-care principles may influence their choices in implementing these principles in their practices; however, other obstacles also may impact providers' implementation. For example, providers may profess that it is best to provide services at times and locations that are convenient to families, but may not be able to do so because of imposed limits on their practice hours, restrictions on their travel to families' homes, or limited availability of suitable service delivery locations. Measuring service providers' attitudes and practices will provide additional information about the extent to which system-of-care principles are implemented by individual professionals across various service agencies within a community. The extent to which these practices are related to children's outcomes in these communities also may be examined.
The Provider Attitudes and Practices Survey (PAPS) is designed to assess the attitudes and practices of a broad variety of service providers who work with children and families. The purpose of the survey is to determine the extent to which system-of-care principles are reflected in the attitudes of individual providers and whether they implement services according to these principles. In addition, providers will be asked questions about their training, the type of services they provide, and their familiarity with and use of evidence-based practices. The Provider Attitudes and Practices survey will be mailed in early 2002 to a sample of providers representing the major child-serving agencies at each of the four Phase II comparison study communities. Differences in provider attitudes and practices in system-of-care and non-system-of-care communities will be examined to assess the extent to which these differ. The survey will be sent to providers a second time during the following year to appraise possible changes in attitudes and practices.
The Phase II comparison study, begun in spring 1999 in two system-of-care communities funded in 1997 and two matched non-system-of-care communities, has enrolled 542 families and will continue to enroll families during the next year. Follow-up rates are high, with most families continuing in the study. Although some differences in demographic and clinical characteristics of children and families are seen at this stage of enrollment, these differences are monitored closely, and adaptations to study enrollment procedures are made to improve comparability in matched pairs of communities. At this early stage, limited preliminary information is available about outcomes of children and families in each community, although children are showing improvements to 6 months. Children and families have received a variety of services in each community, with those in the system-of-care communities having received more services in the first 6 months. There is also evidence to suggest that children served in the systems of care are receiving services more appropriately matched to their levels of impairment in functioning and behavioral and emotional problems. This preliminary finding is similar to findings in the Phase I comparison study (see Chapter III) that indicate a better match of services to service needs among children in the system-of-care than among those in the non-system of care.
Important components of the Phase II comparison study include the assessment of service experience and service provision at the practice level. These components of the comparison study have multiple methods intended to examine both caregivers' and children's experiences and the perspectives of providers. First, new measures administered to all families at follow-up have been developed to assess overall service experiences as well as specific experiences with individual providers. Second, a treatment adherence and service experience substudy was launched to provide information about the extent to which treatment plans are followed and about the conditions that may impact treatment provision. Finally, a survey of providers is planned for the coming year to assess attitudes and practices across various child-serving sectors within each community. Each of these assessments of service experience will expand upon the examination of service experience using the System of Care Practice Review (SOCPR) protocol employed in the Phase I comparison study.
|
| Home | Contact Us | About Us | Awards | Accessibility | Privacy and Disclaimer Statement | Site Map |
