This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
2000 Annual Report to Congress on the Evaluation
|
|
In summary, system-of-care assessments of 17 grant communities funded in 1997 and 1998 indicate that:
|
A central question that this evaluation addresses is whether greater improvements in children's behavior and functioning are associated with a system-of-care approach. The underlying assumption depicted in the system-of-care theory model (see Figure 2) is that a fully implemented system of care will lead to more effective service delivery, improved clinical and functional outcomes, and greater satisfaction with services. To answer the question, it is critical to know the extent to which grant communities have implemented true systems of care. The system-of-care assessment has been designed to ascertain the degree of system implementation and to document how systems of care develop over time. It is inevitable that the real-world implementation of community-based service programs will differ at least in some respects from the ideal design. The study of implementation fidelity examines the degree to which a program's design was followed. Without this knowledge, it is difficult to know whether to ascribe program outcomes to the program as originally planned or to a hybrid implementation of the program design (Bickman, 1987; Chen, 1990; Lipsey, 1990). The implementation study in this evaluation, the system-of-care assessment, examines whether each grant community has established formal structures and organizational arrangements that support service delivery practices guided by system-of-care principles. The assessment was particularly challenging because the system-of-care philosophy and approach can take many different shapes and forms when implemented in different communities.
A system-of-care assessment has been part of the national evaluation since its inception. As part of Phase I, 31 sites within 22 grant communities funded in 1993 and 1994 were assessed at four points in time during their 5-year funding periods. The original protocol assessed "attributes" expected to be found in the infrastructure as well as in the service delivery of systems of care. Data were collected through stakeholder interviews, document review, and case record abstraction. Qualitative data from these assessments have improved the understanding of how systems of care develop and operate over time. Findings from the Phase I system-of-care assessment have been reported in previous Reports to Congress (Center for Mental Health Services [CMHS], 1996, 1997, 1998, 1999) and elsewhere (Vinson, Brannan, Baughman, Wilce, & Gawron, in press).
The Phase I system-of-care assessment indicated that despite early implementation challenges and gaps in services, grant communities dramatically expanded the diversity and range of services available to children and families. Grant communities varied, however, in how input from diverse child-serving agencies, families, and community members was used to manage and set policies for their systems of care. Often these relationships were challenged by differing priorities and conflicting agency mandates regarding service delivery. While families were actively involved in planning their own services, they were generally less involved in managing the system that delivered those services. The assessment also found that the practice of outstationing case managers in partner agencies was an effective strategy for spreading the system-of-care approach throughout the community. Ultimately, if the communities shared their resources, strengthened their partnerships, and firmly established system-of-care values by the end of the grant period, then service delivery practices that reflect those values were likely to continue beyond the Federal funding.
Building on what was learned in Phase I, a new system-of-care assessment tool was developed for the national evaluations of grant communities funded in 1997-98 (Phase II) and 1999-2000 (Phase III). This chapter describes the new approach that is being used to assess system-of-care development in Phase II and Phase III grant communities and then presents preliminary findings from 18 sites funded in 1997 and 1998. These 18 sites representing 17 grant communities were assessed in the winter and spring of 2000 using the new assessment tool. At this early stage, the findings reported below should be considered preliminary, but they are helpful descriptions of the status of a subset of grant communities in their second and third years of CMHS funding.
Information in this chapter was collected in 18 sites representing 17 grant communities, including two distinct sites within one grant community. The assessment framework is discussed below, followed by a description of the assessment measure and data collection procedures.
Development of the Phase II assessment tool began with a revised conceptual framework. In essence, this framework describes the basic generic components of any service delivery system, and it rates each component on how well it has realized key system-of-care principles. Following the literature, the Phase I system-of-care assessment tool, and other work done in the field, the framework was divided into two domains: the system infrastructure and the service delivery process. The system infrastructure domain refers to the organizational arrangements and procedural framework that support and facilitate service delivery. The service delivery domain involves the activities and processes undertaken to provide services to children and families that address and relieve emotional and behavioral challenges experienced by the child.
Eight generic service system components were identified. The infrastructure domain had four components that addressed the areas of governance, management and operations, service array, and evaluation and quality monitoring. The four system components in the service delivery domain were entry into services, service planning, service provision, and case review. The components were identified by examining the structures and processes that are common across major child-serving systems (child welfare, education, juvenile justice, and mental health) and those that were in place among the Phase I grant communities. Among the components that were identified, it was determined whether they were related to policies, procedures, and arrangements across agencies (i.e., infrastructure level) or whether they were related to direct service provision to children and families (i.e., service delivery level; see Table 2). According to the program theory, systems of care are distinguished from more traditional service systems by how those components are organized and how program elements are carried out. In systems of care, these structures and activities should be directed by and demonstrate system-of-care values and principles. Hence, this measure assesses the extent to which the eight components across the two domains (four for infrastructure and four for service delivery) are implemented in accordance with system-of-care principles. For this assessment, the following eight system-of-care principles were selected: family focused, individualized, culturally competent, interagency, collaborative/coordinated, accessible, community based, and least restrictive.
The revised framework is organized into a matrix that has system components as the columns and the system-of-care principles as the rows (see Figure 4). The framework cells, where the component columns and the principle rows intersect, contain indicators of system performance. For example, at the intersection of governance and family focus, three distinct indicators address the general question, "To what extent is system governance conducted in a family-focused way?" Across the eight components, 118 indicators were generated to assess the eight system-of-care principles.
The system-of-care assessment was developed based on this revised framework. In the development process, several steps were taken to maximize measurement quality. First, the framework was reviewed by experts in the field whose suggestions were incorporated into a revised measure. Second, interview protocols were developed that closely followed the revised framework. Third, the interview protocols were pilot-tested in four Phase I grant communities, and revisions were made based on those experiences. Fourth, near-final versions of interview protocols were reviewed by field experts. The final version of the measure is being used to assess system-of-care development in Phase II and Phase III grant communities.
Two trained interviewer-raters collect most of the data for the system-of-care assessment during a 3-day site visit. The system-of-care assessment relies on three primary sources of data, including
Before the visit, grant communities provide documentation of information specifically requested for the system-of-care assessment, including summary information about their governing body, staff composition, training events, fiscal status, and array of service options. While on site, interviewers use a standard protocol to obtain information from individual client records and also conduct the interviews. The information the interviewer-raters collect in their semistructured interviews constitute most of the data reported in this chapter.
Interviewers use 13 semistructured interview protocols to collect information from a variety of individuals involved with system development and service delivery (see Table 3). Whenever possible, they ask more than one informant about efforts to address the same indicator. Interviewing a wide range of respondents ensures that multiple perspectives are represented. The interviews vary in length from 30 minutes to about 2 hours.
Some of the items in the interviews are used to gather contextual or descriptive information, while others are linked to indicators in the framework. For items that map to framework indicators, interviewers use responses from individual informants to rate the system on a 5-point scale (with 1 being the lowest and 5 the highest) using established criteria for that item. That is, the qualitative data collected in the semistructured interviews are used to produce ratings for each of the 118 indicators. Responses from the various informants are rated separately. This approach, which bases numerical ratings on qualitative data, uses comparisons within and between grant communities and generates quantitative scores as well as rich descriptive information.
The following example illustrates the scoring process. There are three indicators in the cell at the intersection of the management and operations component and the family focus principle. One of those indicators is:
The staffing structure includes laypersons and paraprofessionals, such as family members, to support families in the care of their children.
To collect information for this indicator, interviewers ask questions of the project director and family representatives who are involved in system-of-care development. They then generate a separate score for each respondent based on that individual's answers. For example, a score of 1 indicates that the respondent reported no efforts to include laypersons or paraprofessionals in the staffing structure. A score of 2 indicates that there was an effort, but that it was minimally effective in providing enough staff positions to meet the demand for services. A score of 3 indicates that the effort to supply an adequate number of paraprofessional staff was somewhat effective in meeting the need, but that considerably more effort was needed for the goal to be fully achieved. A score of 4 indicates that the efforts were moderately effective, but that some additional effort was needed to meet the needs of all children and family served by the program. A score of 5 indicates that the efforts were very effective and fully sufficient to meet the larger goal. In this example, a 5 means that staffing capacity is sufficient to meet the need of all or almost all families who need the services provided by paraprofessional staff.
Because the data collected through the system-of-care assessment are used to track system development over time and to make comparisons across systems, it is imperative that the ratings are reliable. A key means of assuring reliability is through interrater reliability, that is, the extent to which different interviewers rating the same system at the same time report similar results. To reduce variation across raters, rating criteria for each item are as explicit as possible, and site visitors are trained to apply those criteria in a standard fashion. In addition, each site visitor is required to achieve 85 percent agreement with the correct ratings for 44 hypothetical interview scripts during training. Furthermore, when scores are submitted, a senior researcher reviews the responses to identify inconsistencies or errors in scores; any questionable ratings are reviewed with the site visitor and resolved. Subsequent to training and before data were collected, site visitors did achieve the required 85 percent agreement.
Because scores are calculated for individual items from each respondent, combining scores can be accomplished in a variety of ways depending on the question of interest and the analysis selected. For this report, composite scores were calculated for each of the principles in the framework within each domain (infrastructure and service delivery). In addition, simple mean scores were computed for each item of each indicator in individual cells (see below).
A content analysis of the qualitative data provided information on the types of strategies and approaches used by the grant communities for selected principles. Qualitative data analyzed for this report primarily came from the semistructured interviews conducted with system-of-care stakeholders. After scores were reviewed and data entered, responses from interview questions were entered into a qualitative data management software program (EZ-Text). The software program aided the analysis by allowing the generation of lists of answers to specific questions across all grant communities and all respondents within a grant community. For example, to learn what types of services were offered by paraprofessionals within and across grant communities, a report was created that listed the grant community code, respondent code, and a list of paraprofessional services. This information was then used to illustrate the bases of ratings by describing exactly how grant communities operationalized system-of-care principles. In addition, grant community efforts that scored high in specific areas were compared and contrasted with grant community efforts that did not score high in those areas.
Eighteen sites (17 grant communities) were assessed in the winter and spring of 2000 (see Figure 5). Ten of these sites, representing nine grant communities funded in 1997, were in their third year of funding and received their second assessment. Eight grant communities, funded in 1998, were in their second year of funding and received their first assessment. All 17 grant communities assessed in these site visits had a strong base of existing services and resources from which to draw. In fact, in some of these grant communities, systems were being built upon structures and service arrays that had existed for over a decade before the infusion of CMHS grant funds, or before legislation had instituted system-of-care principles and values statewide. For example, Kentucky (1985), Texas (1988), Michigan (1990), and Missouri (1990) received Child and Adolescent Service System Program (CASSP) grants from the National Institute of Mental Health that formed the basis for system-of-care development within their State systems. The North Carolina and Maine programs grew out of CMHS Phase I programs. Vermont (1988), Kentucky (1990), Nebraska (1995), Missouri, and Wisconsin have passed legislation mandating interagency collaboration, intensive case management, and mental health services aimed at maintaining children in their home communities. Similarly, Michigan, Nebraska, North Dakota, Oregon, Washington, and Wyoming have instituted statewide reform of public agencies from the State level to the local level to increase interagency collaboration and to enhance the provision of individualized and family-focused services. In Michigan, North Dakota, and Wyoming, the reforms have included American Indian communities located within their States. Alabama is guided by the provisions of a judicial consent decree from the early 1990s to provide comprehensive, coordinated, and individualized services to children in its child welfare and juvenile justice systems.
Although most of the 17 grant communities (18 sites) included in this assessment had strong foundations upon which systems of care could be further developed, there are important differences among them. Community-specific environmental factors such as geography, sociopolitical structures, cultural influences, and economic resources have significant effects on system development (National Assembly of National Voluntary Health and Social Welfare Organizations, 1991). Grant communities visited for this assessment included four American Indian communities, an Appalachian Mountain community, sites that served largely Latino and/or African-American communities, and others that served fairly homogenous Caucasian communities. Recognizing the variation in system-of-care approaches implemented by each grant community is critical to understanding how systems of care develop. The grant communities studied in this evaluation report vary in several key characteristics that affect their implementation and operation.
These community-level differences provide an important context for understanding how grant communities focus system development, how priorities are set, and how choices are made in system-of-care development. The Comprehensive Community Mental Health Services for Children and Their Families Program established a standard set of requirements for grant communities that includes mandatory family involvement in system development and a minimum array of service options, but it allows for relatively broad flexibility in how those requirements can be met. Grant communities with different underlying characteristics and varying resources may individualize the development of their systems of care according to local community context. The next section of this report presents qualitative information that describes this variation in implementation approaches within the context of quantitative scores.
The information presented below illustrates the extent to which systems of care have developed since receiving CMHS funding. These preliminary findings are based on ratings derived from interviews with stakeholders and do not include data from ratings of documents or client records. The information collected with this assessment tool provides a description of how systems of care have provided services and have applied system-of-care principles within their structures.
Scores for each of the eight system-of-care principles applied to each domain (infrastructure and service delivery) were averaged across the 18 sites. Simple mean scores were computed for each item of each indicator in the 64-cell matrix formed by the eight system components and the eight system-of-care principles (see Figure 4). Findings for both the infrastructure and service delivery domains are first presented globally, highlighting overall patterns across principles.
Figure 6 shows the score averages for the eight principles applied to each domain across sites. On average, sites tended to score higher in the service delivery domain than in the infrastructure domain. Site approaches to these principles at the infrastructure and service delivery levels are described below.
As noted earlier, systems of care developed in these grant communities have diverse catchment areas, target populations, and cultural characteristics. These differences have implications for how the systems of care are implemented, and the pace and nature of their development over time. The following section discusses to what extent the grant communities achieved system-of-care principles. When scores for a given principle varied widely across sites (with standard deviations above 0.70), individual indicator scores are provided for more detailed explanation of the variation. Also, because service planning and service provision activities are closely connected, the two components are discussed together within each principle. Finally, because the American Indian grant communities have tribal government councils instead of service system governing bodies, information on governance is not reported for those four sites.
Mean scores across all sites for the principle of family focused within the domains of infrastructure and service delivery were similar (M = 3.73, SD = 0.46 and M = 3.97, SD = 0.29, respectively). This represents the highest rating within the infrastructure domain. Because the CMHS Guidance for Applicants specifies that grant communities must demonstrate that family members are involved in system development and governance, there is support and encouragement for the early formation of partnerships with families in sites. This, coupled with the momentum and leadership that the family movement has experienced in the past few years, may have impacted the sites' abilities to enhance family involvement opportunities in their systems of care to a level approaching "moderate effectiveness."
Family Focused — The recognition that (a) the ecological context of the family is central to the care of all children; (b) families are important contributors to, and equal partners in, any effort to serve children; and (c) all system and service processes should be planned to maximize family involvement. |
Family-Focused Grant Governance. Three indicators are averaged to compute the rating for family-focused governance:
Because the CMHS Guidance for Applicants specifies that grant communities must demonstrate that family members are involved in system development and governance, there is support and encouragement for the early formation of partnerships with families in sites. |
In general, most sites were rated as moderate to very effective in this area (cell M = 4.42, SD = 0.37). They all reported that family members or representatives from family organizations actively participated on governing bodies. Types of governance functions in which families typically participated included strategic planning, service array development, and formation of interagency partnerships. Sites reported that they sought and respected family input. They also supported families' attendance at governing board meetings through stipends, travel reimbursement, and child care provision, and they scheduled meetings for times and locations that are convenient for families.
Family-Focused Program Management and Operations. Two indicators are averaged to compute the rating for family-focused management and operations:
On average, sites scored slightly above the mid-range in this area, but there was some variation (cell M = 3.58, SD = 0.75).
A majority of the 18 sites (95 percent) utilized family members and other paraprofessionals to provide direct services to children and families; however, the number of services provided by family members and paraprofessionals varied considerably. The highest scoring sites reported that family members or paraprofessionals provided an average of eight different types of services. Services most commonly provided included mentoring, facilitating family support groups, advocacy, therapeutic foster care, transportation, tutoring, parent aide, respite care, child care, and family liaison. In the lowest scoring sites, families or paraprofessionals provided two or fewer direct services.
Eighty-three percent of the 18 sites also reported that family members actively participated in program operations, but again, the nature and depth of that involvement varied across sites. Family members performed an average of five functions in sites that scored highest. These sites reported that family members participated in grant operations by helping to interview and recruit staff, training professionals and family members, collecting data for evaluation or quality monitoring purposes, attending management meetings, and serving on program committees. The lowest scoring sites reported only one of these family-focused functions on average.
Family-Focused Quality Monitoring. Three indicators are averaged to compute the rating for family-focused quality monitoring:
Although scores in this area varied across sites, on average they tended to be low (cell M = 2.33, SD = 0.78), indicating that considerable effort is needed to attain this goal. Nonetheless, the highest scoring sites scored above the mid-range, indicating that they were including families in some capacity and were beginning to examine family outcomes and service experience data.
Overall, 16 of the 18 sites visited (89 percent) reported involving family members in their program evaluation and quality monitoring processes in some way. Nine sites (50 percent) reported that family members collected data for national or local evaluation. Only seven sites (39 percent) included family members as participants on evaluation or quality monitoring committees. In the highest scoring sites, family members served in at least four different capacities, including participating on quality monitoring committees or evaluation teams, collecting evaluation data, aiding in the interpretation of findings, and helping to present data and findings to multiple audiences. In the lowest scoring sites, families had limited, if any, involvement in quality monitoring activities.
Eventually, all grant communities are expected to collect information on family outcomes and service experiences as part of the national or local evaluation. At the time of their system-of-care assessment, three sites had begun to examine or analyze those data, and two sites had used collected information to make system improvements. Examples of system improvements included reducing the time families wait for service planning meetings, enhancing the role of service coordinators to improve care monitoring, and adding special services in rural areas.
Family-Focused Entry into the Service System. Family-focused service entry includes only one indicator:
This means that the entry process is not complicated, and that family members feel comfortable and respected during the process. Across most sites, caregivers reported that entry into the system was not complicated or difficult (cell M = 4.78, SD = 0.40), with only one site scoring below 4. Approaches used by sites to facilitate entry into the service system are discussed in more detail below in the section addressing the principle of accessibility.
Family-Focused Service Planning and Provision. Three indicators are averaged to compute the rating for family-focused service planning and provision:
Sites scored high on average for family-focused service planning, and there was not much variability across sites (cell M = 3.97, SD = 0.47), with all sites scoring above the mid-range. Scores for family-focused service provision were similarly high, with little variability (cell M = 3.93, SD = 0.40).
Examples of Services Provided to Caregivers, Siblings, and Other Family Members
|
Sixteen (89 percent) sites reported that family members were present for service planning meetings at least 90 percent of the time. All sites reported that when family members were present, they routinely participated in service planning by identifying and prioritizing issues to be addressed, developing goals and objectives of the plan, and selecting planning team members, types of services, and service providers. Fifteen (83 percent) sites reported that they assessed family needs for at least 90 percent of all families, and 14 (82 percent) provided examples of at least three types of services that focused on family needs that were included in individual service plans. However, only 11 sites (61 percent) reported that at least 90 percent of families received all the services that service planning teams developed for them.
All sites reported some degree of family participation in service provision. Approaches for achieving family focus in service provision included
All sites reported that they assessed strengths for at least 75 percent of families. Sixty-six percent of these sites provided at least three examples of how they incorporated family strengths in planning services.
Examples of the Incorporation of Family Strengths into Service Provision “For parents who are willingly involved, (I invite) them into family therapy and try to use them to engage those parents less involved.”— Outpatient Mental Health Clinician “A mother who was very experienced with our therapeutic case management program now serves as a facilitator for family/child team meetings and serves on the parent advisory board.”— Care Coordinator |
A primary tenet of the system-of-care approach is providing individualized care to children with severe emotional disturbance and their families. Overall, sites scored above the mid-range on the individualized principle in the infrastructure domain (M = 3.24, SD = 0.47) and somewhat higher in the service delivery domain (M = 3.81, SD = 0.34). These findings suggest that, on average, sites were making strides toward individualizing services for children, but that additional formal structures and procedures may be needed to advance these efforts further.
Individualized — The provision of care that is expressly child centered, that addresses the child’s specific needs, and that recognizes and incorporates the child’s strengths. |
Individualized Care and Management and Operations. Individualized care and management and operations include only one indicator:
On average, sites scored above the mid-range (cell M = 3.41, SD = 0.68), with some variability across sites. Approaches for achieving individualized care at the management and operations level included
Although most sites reported using these and other mechanisms, they differed in the extent to which the efforts were formalized. Respondents also had differing perspectives on the effectiveness of those efforts.
Individualized Care and Service Array. Individualized care and service array includes only one indicator related to the broadness of the service array:
Examples of Additional Services Provided through Grant Programs
|
The average score across the sites was relatively high and reflected broad variability across scores (cell M = 3.99, SD = 0.86). Indeed, 12 sites (67 percent) reported providing all the services required in the Guidance for Applicants (see Table 4). All sites also offered other service options that were not required. This is in keeping with earlier findings from this initiative (Vinson et al., in press) that grant communities tended to be successful at offering a wide range of services with few key service gaps.
Individualized Care and Quality Monitoring. Two indicators are averaged to compute the rating for individualized care and quality monitoring:
As with the principle of family focus, quality monitoring efforts related to providing individualized care were not yet fully implemented at most sites; however, there was variability across sites (cell M = 2.15, SD = 0.71). While most sites had not yet examined the data they had collected, two sites funded in 1997 reported that system improvements had been made in response to findings.
Individualized Care and Service Planning and Provision. The indicators that are averaged to compute the ratings for individualized care in service planning and provision include the following:
Sites tended to score above the mid-range for individualized service planning (cell M = 3.73, SD = 0.52) and slightly higher in individualized service provision (cell M = 4.03, SD = 0.34).
Fifteen sites (83 percent) reported that they developed individualized service plans for at least 90 percent of the children whom they served. Seventy-two percent of caregivers reported that service plans matched their child's needs very well or extremely well, and all others reported that service plans matched their child's needs moderately well. However, only 50 percent of sites reported that 90 percent of the children served received all services that were planned for them. Explanations offered for this included insufficient capacity, waiting lists, and families changing their minds about the services.
Eleven sites (61 percent) reported that their efforts to monitor the care provided to children were at least moderately effective, and almost all sites admitted that improvements could be made in the monitoring of service provision. Examples of barriers to successful monitoring included lack of cooperation by some agencies, the existence of turf issues between agencies, lack of response to requests for release of information, grant staff having difficulty in establishing contact with service providers, large caseloads, and issues related to family circumstances (e.g., lack of telephones, not providing contact information, or not granting permission to release records).
Examples of efforts to create a culturally diverse staff included posting employment information about available positions at minority universities, having current minority staff contact their alma maters, conducting recruitment efforts at minority universities, and utilizing interns from historically Black colleges and universities. |
In general, sites scored higher on indicators assessing the use of a strengths-based approach for children. Fifteen sites (83 percent) reported assessing strengths for all children served, but sites continued to experience some difficulty integrating children's strengths into the planning and provision of services. Relatively few sites (n = 7) provided at least four case examples of how their service plans incorporated children's strengths, and only six sites provided at least four examples of how their provision of services incorporated children's strengths.
Cultural Competence
Similar to their colleagues funded in earlier cohorts, these sites continue to struggle with achieving a culturally competent system of care at both the infrastructure and service delivery levels (M = 2.53, SD = 0.81 and M = 2.78, SD = 0.93, respectively; see Figure 6; Vinson et al., in press). There was, however, a broad distribution of overall mean scores across sites, ranging from 1.41, demonstrating almost no efforts in this area, to 4.16, suggesting moderate to high levels of effectiveness.
Culturally Competent — Sensitivity and responsiveness to, and acknowledgment of, the inherent value of differences related to race, religion, language, national origin, gender, socioeconomic background, and community-specific characteristics. |
Cultural Competence in Management and Operations. Two indicators are averaged to compute the rating for cultural competence in management and operations:
Sites scored in the mid-range overall but with considerable variability across sites (cell M = 3.13, SD = 1.21). For example, three sites received scores of 5.0 for the principle of cultural competence within management and operations. Examples of efforts to create a culturally diverse staff included posting available employment positions at minority universities, having current minority staff contact their alma mater, conducting recruitment efforts at minority universities, and utilizing interns from historically Black colleges and universities. Other examples of efforts included actively supporting academic advancement and training of American Indian staff, employing American Indian tribal members as paraprofessionals and social workers, and seeking new employees within American Indian communities prior to posting positions externally. In general, sites scoring low in this area used fewer methods to develop a culturally diverse staff, and they employed methods that were often informal and less structured.
Cultural Competence in Service Array. Cultural competence in the service array focuses on one indicator:
Sites tended to score lower in this area, but scores were broadly distributed across sites, ranging from 1.00 to 4.33 (cell M = 2.41, SD = 1.09).
Examples reported by higher scoring sites included having cultural consultants available to providers, employing cultural mentors for families who wanted to learn more about their culture, and contracting with local cultural organizations to provide services. American Indian sites offered several culturally relevant services such as sweat lodge ceremonies, drumming groups, and storytelling with elders. In contrast, sites scoring low in this area reported few or no examples of adding to or modifying the service array to attend to the cultural needs of the population.
Cultural Competence in Quality Monitoring. Two indicators are averaged to compute the rating for cultural competence in quality monitoring:
Sites scored particularly low in this area, with wide variation (cell M = 2.11, SD = 0.92). At the time of assessment, six sites (33 percent) reported that they were collecting data related to culturally competent care. Two sites reported that they had begun to analyze that data, but they had yet to make improvements to the service system as a result of the process. The remaining sites indicated that data had not yet been collected in this area.
Ten sites (55 percent) reported that their communities engaged in somewhat effective efforts to ensure that the quality monitoring process was culturally competent. Efforts reported as being helpful included having instruments translated into appropriate languages, using bilingual interviewers, and having individuals of color participate on quality monitoring and evaluation committees.
Cultural Competence and Entry Into Service System. Two indicators are averaged to compute the rating for cultural competence in service system entry:
On average, sites scored below the mid-range in this cell (cell M = 2.36, SD = 0.86), but three sites scored considerably higher. In these sites, outreach efforts to specific cultural groups included brochures, a youth outreach initiative, American Indian pow-wows, presentations at community organization meetings, and the publication of grant-related information in tribal newspapers. In contrast, five sites reported minimal or no efforts to ensure that the entry process was culturally competent.
Cultural Competence and Service Planning and Provision. Three indicators are averaged to compute the rating for cultural competence in service planning and provision:
In general, sites struggled in their efforts to make service planning (cell M = 2.87, SD = 0.98) and provision (cell M = 2.92, SD = 1.8) culturally competent. Those sites scoring higher included American Indian communities that reported culturally competent strategies such as combining Western medicines and traditional healing techniques, linking families with spiritual leaders, and involving children and families in culturally specific activities such as American Indian youth groups, sweat lodge ceremonies, and drumming. Similarly, a site that served a large Latino population reported using extended family structures as vital resources. Sites scoring low in this area reported minimal or no efforts to incorporate culture into their service planning processes or in service provision. In addition, where sites served largely homogenous catchment areas or when staff and families shared race or ethnicity, discussion around cultural values often did not take place.
Agencies most commonly represented on governing bodies are mental health, education, juvenile justice, and child welfare. |
The system-of-care approach calls for the active involvement of public agencies in all child-serving sectors, including mental health, child welfare, education, juvenile justice, and public health (Stroul & Friedman, 1986). Real and substantial barriers, both philosophical and structural, must be overcome to realize this goal. At this stage in development, sites rated their own efforts to maximize interagency involvement in both infrastructure (M = 2.95, SD = 0.52) and service delivery (M = 3.27, SD = 0.56) as somewhat successful overall, but scores were lower than for all other principles except cultural competence (see Figure 6). Almost all sites described participation of some agencies in the public sector, but rarely did all five sectors participate fully in the system of care.
Interagency — The involvement and partnership of core agencies in multiple child-serving sectors, including child welfare, health, juvenile justice, education, and mental health. |
Interagency Involvement in System Governance. Two indicators are averaged to compute the rating for interagency involvement in system governance:
Sites typically involved multiple agencies in their system governance, with little variance across sites (cell M = 4.01, SD = 0.35). Agencies most commonly represented on governing bodies were mental health, education, juvenile justice, and child welfare. Public health was rarely represented. Occasionally, other human service providers and programs such as drug and alcohol treatment providers and income assistance were reported as partners in governance.
Interagency Involvement in Management and Operations. Four indicators are averaged to compute the rating for interagency involvement in management and operations:
Overall, the sites rated their capacity to incorporate interagency involvement into their management and operations slightly below the mid-range, with little variability (cell M = 2.48, SD = 0.42).
Sites reported an increase in available resources as a result of their participation in the system-of-care grant (44 percent), and they also reported that agency philosophies became more family focused (39 percent), thus significantly impacting routine agency operations. Other ways routine agency practices were altered as result of the grant included increased collaboration among child serving agencies (28 percent), changes in treatment approaches (28 percent), increased knowledge of existing resources (22 percent), and increased training opportunities (22 percent).
Interagency Involvement in Quality Monitoring. Two indicators are averaged to compute the rating for interagency involvement in quality monitoring:
Overall, sites scored low on these indicators (cell M = 2.18, SD = 1.10), with a wide distribution of scores ranging from a low of 1.00 to a high mean score of 4.00.
Sites that scored high reported that public health, education, mental health, juvenile justice, and child welfare were actively involved in the quality monitoring processes of instrument development, data collection, data review, and data dissemination. They typically facilitated these efforts by involving representatives from the various agencies on quality monitoring and evaluation committees. Other sites reported that only the grant-receiving agency was involved in quality monitoring efforts. Although several sites have implemented data collection efforts, none reported system change resulting from monitoring or evaluation efforts.
At this stage in development, sites rated their own efforts to maximize interagency involvement in both infrastructure and service delivery as somewhat successful overall, but scores were lower than for all other principles except cultural competence |
Interagency Involvement in Service System Entry. Two indicators are averaged to compute the rating for interagency involvement in service system entry:
In general, sites scored around the mid-range, with a relatively wide range in scores (cell M = 3.07, SD = 0.76). Several sites (28 percent) were fairly successful in this area and reported a broad distribution of referral sources, including mental health, education, juvenile justice, and child welfare. In contrast, those sites receiving low scores reported a limited distribution of referral sources.
Interagency Involvement in Service Planning. Interagency involvement in service planning is assessed by one indicator:
Overall, sites scored above the mid-range, with scores widely ranging from 1.17 to 4.42 (cell M = 3.36, SD = 0.93).
Sites scoring high in this area reported that all agencies involved with particular children and their families routinely participated in service planning processes for those individual children and families. Sites scoring low reported that multiple agencies did not participate even when they were directly involved with the children, or that participation was not routine. While each of the sites with high scores reported specific agencies that were difficult to engage in the service planning process, several of the remaining sites stated that participation varied according to individual staff and could not be generalized to agency behavior or practice. Two sites reported that other agencies were not a part of their system's service planning process and therefore were not expected to participate on a routine basis.
One guiding principle for system-of-care development is that children with emotional disturbance and their families should receive services that are integrated, with linkages between child-serving agencies and mechanisms for planning, developing, and coordinating these services (Stroul & Friedman, 1986). For this assessment of system-of-care development, indicators measured processes that facilitate information sharing and the coordination of service delivery. A total of 11 indicators are used to operationalize this principle. However, eight indicators contribute to the overall mean score, because three of these indicators are not scored (two assessing the case review process and one assessing service array). Sites were slightly more effective at incorporating this principle at the service delivery level (M = 3.49, SD = 0.39) than at the infrastructure level (M = 3.14, SD = 0.42).
Collaborative and Coordinated — Professionals working together in a complimentary manner to avoid duplication of services, eliminate gaps in care, and facilitate the child and family’s movement through the service system. |
Collaborative-Coordinated Management and Operations. Two indicators are used to assess the collaboration and coordination of system management and operations:
In general, sites scored within the mid-range in this area, with little variability (cell M = 3.31, SD = 0.45). Scores in this area ranged from 2.14 to 3.81, with fewer than 11 percent of the sites scoring below 3.0.
Seventy-two percent of sites routinely held interagency team meetings to facilitate the coordination of services. Other examples of mechanisms to facilitate the service coordination included interagency case management meetings (33 percent), interagency case review meetings (17 percent), child and family team meetings (17 percent), and intake and placement team meetings (17 percent). Most sites reported using several of these methods.
Collaborative-Coordinated Quality Monitoring. Scores for collaborative and coordinated quality monitoring represent one indicator:
Overall, sites scored quite low in this area, with over 88 percent receiving a score of 2 or lower. There was, however, considerable variability (cell M = 1.56, SD = 0.73). Two sites indicated that they had collected and begun to analyze data, but the remaining sites either had not collected data in this area or had not analyzed or used the information for service improvement.
A common vehicle for quality monitoring related to the coordination of services was the administration of consumer satisfaction surveys. These surveys were typically designed to obtain feedback from families and providers describing their experiences within their system of care. In those sites where consumer satisfaction surveys had been conducted, results from these surveys indicated that consumers were satisfied with the services they had received. Examples of additional efforts to monitor service coordination included routine reviews of individual service plans, case record reviews, and the collection and analysis of services and cost data.
Collaborative-Coordinated Entry into Service System. Collaborative and coordinated service entry is assessed by one indicator:
Overall, sites were rated high in ensuring that entry into the service system was collaborative and coordinated (cell M = 3.65, SD = 0.88). The distribution of scores within this area was relatively broad, with scores ranging from 2.50 to 5.00, but nearly 39 percent of all sites received a mean score of 4 or higher.
Sites that received high scores were able to provide concrete examples of their outreach efforts. They stated that they provided in-service training to the staff of various public agencies, conducted presentations to various community groups and organizations, published and distributed newsletters and brochures, met with community leaders, and hosted several training events. Through these activities, sites were able to broaden the reach of their services by informing the wider community about the grant program, whom it served, and what it offered.
Collaborative-Coordinated Service Planning and Provision. Two indicators are averaged to compute the rating for collaborative and coordinated service planning:
Although 50 percent of sites reported having a unified service planning process, the level of participation by child-serving agencies varied considerably. Only 33 percent of sites reported that all involved providers and child and family organizations routinely participated in the service planning process. Scores in this area were also within the mid-range, with little variability (cell M = 3.23, SD = 0.57).
Two indicators are averaged to compute the rating for collaborative and coordinated service provision:
The overall mean score for sites for collaborative, coordinated service provision was in the upper mid-range, with little variability across sites (cell M = 3.70, SD = 0.53).
When examining the coordination of services among child and family-serving agencies and providers, sites reported most often that schools worked well to coordinate services (61 percent). Other agencies that they reported as working well to coordinate services included child welfare (56 percent), juvenile justice (50 percent), and mental health (44 percent). Sites referenced a broad range of perceived barriers and obstacles inhibiting coordinated service provision. Forty-five percent of sites identified issues concerning large caseloads as an obstacle to coordinated service provision. Sites also identified issues concerning confidentiality (22 percent) and territorialism (17 percent) as significant service coordination barriers.
The principle of accessibility in systems of care asserts that services should be accessible to families and children, meaning they must be financially affordable, have an uncomplicated and timely application and approval process, be provided at times and locations that are convenient for family members, and have enough capacity to meet needs without long waiting lists. In the service delivery domain, sites scored higher in this principle than in any other principle, with little variability across sites (M = 3.99, SD = 0.28).
Accessible — The minimizing of barriers to services in terms of physical location, convenience of scheduling, and financial constraints. |
Access and Management and Operations. Accessibility in system management and operations is assessed by one indicator:
On average, sites scored high, but with some degree of variability (cell M = 4.33, SD = 0.75). Sites reported that the cost of services was not a barrier to families receiving them because families rarely pay directly for them from their own resources. Grant funds or various health plans (e.g., Medicaid, CHIP, or Indian Health Services programs) covered most of the costs of services for many children. Other families had private insurance to cover treatment costs. For those families and children not covered by public or private health plans, services were paid either by families through sliding-scale fees, by local county tax levy funds, or by flexible funds provided by the grant.
Access and Service Array. Two indicators are averaged to compute the rating for accessibility of the service array:
Sites scored at the mid-range in this cell, with some variability across sites (cell M = 3.01, SD = 0.63). Many sites (67 percent) reported having the full array of grant-required services available in their programs. However, several programs, especially those funded in 1998, reported that they currently lacked sufficient capacity to meet the demand for all service needs. Services reported to have insufficient capacity due to few providers and long waiting lists included diagnostic and evaluation services, medication management and other psychiatric services, respite care, intensive day treatment, therapeutic foster care, residential treatment, and inpatient hospitalization.
Ten sites that scored above the mid-range in this cell maximized accessibility of the service array by offering services at times and places convenient for children and families. Examples included providing services in the home, after-hours, and on weekends; having 24-hour coverage; providing transportation to services; and being generally responsive to family needs.
Access and Quality Monitoring. One indicator is used to assess access and quality monitoring measures:
Mean scores across sites were low but varied dramatically from 1.00 to 5.00 (cell M = 2.00, SD = 1.32). The varied scores reflect the unequal pace at which sites develop quality monitoring mechanisms. Nevertheless, one high-scoring site reported that review of their data revealed a long lag time between the point of referral to their program and the interagency services planning meeting. They were able to determine the cause for the delay and corrected it through administrative changes to the referral process.
Access and Entry into the Service System. Three indicators are averaged to compute the rating for accessibility of service entry:
Sites scored relatively high, with some variability (cell M = 3.95, SD = 0.45). A majority of sites (55 percent) scored above the mid-range, indicating somewhat effective mechanisms. The highest scoring sites used multiple outreach approaches that included newsletters and brochures, parent-to-parent contact, information booths at school enrollment times and student orientations, community presentations, and various training events. Most sites (83 percent) reported that families entered into their programs easily and quickly.
Access and Service Planning and Provision. Three indicators are averaged to compute the rating for accessibility in service planning and provision:
Sites scored relatively high in service planning, but with a great deal of variability (cell M = 3.76, SD = 1.04). In the area of service provision, sites scored even higher and with much less variability across sites (cell M = 4.06, SD = 0.27). As indicated above, sites reported that they routinely provided services at times convenient to children and families (including after-hours and weekends) and they provided services in homes, schools, and other community locations preferred by individual families.
The principle of community-based systems of care asserts that services will be provided within close proximity to the community in need. Characteristics of a community-based service system include offering a full array of services within the community and minimizing the need for children and families to leave the community for services. Overall, sites scored at the mid-range, with little variability across sites (M = 3.5, SD = 0.43). In the infrastructure domain, sites scored somewhat lower, with greater variability across sites (M = 3.0, SD = 0.71), and in the service delivery domain, scores were above the mid-range, with less variability across sites (M = 3.8, SD = 0.58).
Community based — The provision of services within close geographic proximity to the targeted community. |
Community Based and Service Array. Two indicators are averaged to compute the rating for community-based service array:
Overall, sites scored in the mid-range, with some variability (cell M = 3.31, SD = 0.86). Sites that scored high in this cell reported that they accomplished community-based service delivery by (a) providing home-based wraparound services that included family support, mentoring, and respite; (b) reducing and preventing the need for out-of-community placements through earlier and more intense intervention; (c) providing transportation and home-based services to families; and (d) having a resource-rich urban community.
Although most sites (67 percent) reported having a full array of services in their communities, they still placed children needing residential treatment or hospitalization out of the community. Residential facilities were not available in every community. Most programs offered intensive intervention to prevent the need for such placements. When placements did occur, grant programs strived to return children home as soon as possible by providing intensive wraparound interventions. The Central Nebraska Initiative for Families and Youth and the Clackamas Partnership in Clackamas County, Oregon, are two sites that have focused their efforts on reversing a statewide tradition of court-ordered out-of-home placements (Nebraska) and on bringing home children who are already in residential or State hospital treatment facilities (Clackamas County). Both programs described extensive efforts in place at both the program and policy levels, and they envision success in reaching their goal to have all children receive community-based care in least restrictive environments.
Community Based and Quality Monitoring. Community based and quality monitoring is assessed by two indicators:
Sites scored very low in this area but with wide variability (cell M = 1.76, SD = 1.20). Only three sites scored above the mid-range. These low scores likely reflect the sites' early stages of data collection and utilization. Sites scoring highest in this area reported having collected and examined data on the use of out-of-community services. They interpreted decreasing trends in these placements as early positive system outcomes. Other sites with relatively high scores reported that data have been collected but not yet examined.
Community Based and Service Provision. Community-based service provision is assessed by one indicator:
Sites scored above the mid-range, with some variability across sites (cell M = 3.8, SD = 0.57). As indicated above, most sites reported having a full array of services available within their catchment areas. However, in some of the more rural areas, not all services were available in every family's own community or neighborhood. Therefore, some families had to travel out of their home community to receive some services. A strategy used by many sites to address this situation was to provide services in family homes.
The principle of least restrictive services in a system of care means that services should be delivered in settings that maximize freedom of choice and movement and present opportunities to interact in normative environments. Overall, sites scored at the mid-range, with some variability (M = 3.34, SD = 0.51). Scores were lower and had less variability in the infrastructure domain (M = 3.08, SD = 0.62) than in the service delivery domain (M = 3.66, SD = 0.79).
Least Restrictive — The priority that services should be delivered in settings that maximize freedom of choice. |
Least Restrictive and Management and Operations. Least restrictive and management and operations is assessed by one indicator:
Overall, sites scored at the mid-range but had some degree of variability (cell M = 3.29, SD = 0.75). Those sites with the highest scores reported that they reduced the use of restrictive placements through the provision of intensive in-home or in-school services by well-organized and effective teams of service providers.
Least Restrictive and Quality Monitoring. Least restrictive and quality monitoring is assessed by two indicators:
Sites scored relatively low in this area but with wide variability (cell M = 2.18, SD = 1.29). Only three sites scored above the mid-range. These scores likely reflect the sites' early stages of data collection and utilization. Nevertheless, two sites that scored 5.0 reported that data had been collected and examined on the use of overly restrictive care. They interpreted the decreasing trends in the use of restrictive care as early positive system outcomes. Other sites either had not collected data or had not examined or used the information for service improvements.
Least Restrictive and Service Provision. Least restrictive service provision is assessed by one indicator:
Overall, sites scored above the mid-range, with some degree of variability across communities (cell M = 3.7, SD = 0.87). Sites reported that while they strove to provide services to all children in least restrictive environments, lack of service options, facilities, or capacity sometimes hindered the quick and efficient movement of children into less restrictive settings.
This chapter described the status of 18 sites (17 grant communities) funded in 1997 and 1998 that participated in a system assessment process during the winter and spring of 2000. In general, sites scored moderate to high on the system assessment (considering their stage of development), with overall mean ratings for system-of-care principles varying around the mid-point. Scores in this range indicate that sites are making efforts to achieve the goals of most system-of-care principles and are experiencing some success. Sites scored highest in the family-focused principle. All sites reported that family members or representatives from family organizations actively participated on governing boards, and a majority of sites used family members or paraprofessionals to provide direct services to children and families. Sites scored lowest in the cultural competence principle, indicating that creating a culturally competent system of care presents the greatest challenge. There was, however, wide variability across sites, with some sites using effective recruitment methods to hire culturally diverse staff, and others using creative outreach methods to reach special populations. Sites had more difficulty developing service array options that meet the unique needs and traditions of specific cultural groups. Notably, the American Indian sites were overall more effective in addressing the cultural needs of the children and families served. Across all principles, sites appeared to be in the early stages of collecting and utilizing data to monitor their systems' quality of care. Sites will continue to participate in longitudinal system assessments in order to track how their systems develop and evolve over time.
|
| Home | Contact Us | About Us | Awards | Accessibility | Privacy and Disclaimer Statement | Site Map |
