This Web site is a component of the SAMHSA Health Information Network

Enter Keywords Choose Website This Site Mental Health at HHS All Mental Health

Online Publications Order Publications National Library of Medicine National Academies Press Publications Homepage

printer friendly page e-mail this page bookmark this page shopping cart  current or new account

This Web site is a component of the SAMHSA Health Information Network.

2001 Annual Report to Congress on the Evaluation
of the Comprehensive Community Mental Health
Services for Children and Their Families Program

Home | Summary | Table of Contents | Figures | Tables | I | II | III | IV | V | VI | VII | VIII | Appendix | Appendix B

APPENDIX A

System-of-Care Assessment Methods

TOOLS

A standard and reliable assessment tool is required to assess systems of care and document their development and implementation. At the time this evaluation began, no tool existed; thus, an initial assessment tool, including both a conceptual model and a set of measures, was developed for the national evaluation in 1994 and utilized to assess the first 22 grant communities funded in 1993-94. Using the system-of-care monograph developed by Stroul and Friedman (1986) and the Federal grant guidance, and through consultation with experts in the field of children's mental health, the conceptual model was created to guide the development of the tool. The tool was designed to assess a children's mental health service delivery system in terms of the organizational arrangements and procedural framework developed and used to support and facilitate service delivery and the activities and processes undertaken to provide services.

Revisions to the initial assessment tool were made in1996, and the revised tool has been utilized for the evaluation of the grant communities funded since 1997. The revised tool retains the conceptual approach of the original tool but further specifies generic components that make up the organizational arrangements, activities, and processes of a system. These components include governance, management and operations, service array, quality monitoring, entry into the system, service planning, service provision, and case monitoring and review. Each of the generic components is rated on the degree to which it adheres to the eight system-of-care principles: family focused, individualized, culturally competent, interagency, collaborative/coordinated, accessible, community based, and least restrictive.

The intersection of these organizational aspects and system-of-care principles form the assessment framework (see Figure A.1). Each cell in the framework contains indicators or measures of system performance that are linked to a series of questions asked of respondents during semistructured interviews described below. For example, for the cell in which governance and family focused intersect, questions are asked about three distinct indicators to address the general question, "To what extent is system governance conducted in a family-focused way?" Vinson and colleagues (2001) provide more detailed information about the development of this tool in their article outlining the successes and challenges faced by earlier CMHS-funded grant communities.

PROCEDURES

The assessment design entails conducting multiple data collection site visits to each grant community over the course of its 5-year funding cycle. Grant communities funded in 1997 are assessed four times on a 12-month cycle, while those funded in 1998, 1999, and 2000 are assessed three times on an 18-month cycle. The first assessment point is 12-18 months after initial funding, the second assessment point is 24-36 months after funding, the third assessment point is 36-54 months after funding, and the fourth, for those communities receiving four assessments, is 48-60 months after funding. Not all grant communities have data for all assessment points. Two grant communities funded in 1997 were not sufficiently developed to determine ratings at the first assessment point. However, descriptive reports were prepared for these communities. In addition, in a number of cases the grant was distributed to multiple communities within a State. In three of these cases, the grant communities have elected to have assessments on a rotating schedule such that some communities will have data for assessment points 1 and 3 and others will have data for assessment points 2 and 4. In others, the grant communities have elected to have assessments in only one community.

As of July 2001, data from the first assessment point were available for 30 grant communities, data for the second assessment point were available for 11 communities, and data for the third assessment point were available for 10 communities. Currently, grant communities have not yet had their fourth assessment. A total of five grant communities had data available for all three assessment points. By this time next year, cross-sectional data will be available for 39 grant communities at assessment point 1, 27 communities at assessment point 2, 11 communities at assessment point 3, and nine communities at assessment point 4. By this same time, six communities will have had three assessment points and five communities will have had four assessment points.

The assessments are conducted during a 3- to 4-day site visit by a pair of trained site visitors who conduct approximately 25 semi-structured interviews with a variety of stakeholders. In addition, site visitors review 6-10 randomly selected case records and gather information regarding membership of the governing body and case review structures, system-of-care staffing structures, training events provided by the system during the assessment period, funding information, and services available in the service array. This information is used by the site visitors to rate each indicator for each cell in the framework. After using this tool for the first assessment of grant communities funded in 1997, the ratings were changed from a 4-point to a 5-point scale to increase the sensitivity of the measure. Analyses of data collected using the 5-point scale showed that the data were evenly distributed across the five points. In addition, there were greater differences between the overall mean scores and the highest score for the 5-point scale than for the 4-point scale.1

Because the data collected through the system-of-care assessment are used to track system development over time and to make comparisons across systems, it is imperative that the ratings are reliable. In order to ensure interrater reliability (i.e., reduce variation across raters), rating criteria for each item were made as explicit as possible. Site visitors participated in a 3-day training session to learn how to apply criteria in a standard fashion. At the training, each site visitor was required to achieve 85 percent agreement with accurate ratings for 25 hypothetical scripts. Additionally, reliability testing and refresher training sessions are conducted twice a year to ensure continued reliability among site visitors.

DATA ANALYSIS METHODS

Some of the qualitative data collected during the assessments provide descriptive information about the grant-funded program and the grant community. Other qualitative data link to each of the indicators in the framework, which are rated by the site visitors and aggregated into a cell score. Besides ratings, the site visitors develop a comprehensive report that is both descriptive and evaluative in content. The grant communities have an opportunity to review the reports and correct any errors of fact before they are finalized.

Because quantitative scores for the initial visits for the communities funded in 1997 were rated on a 4-point scale and quantitative scores for all other community site visits were rated on a 5-point scale, an approach was constructed to equate ratings across all site visits. This involved developing range scores to describe how the ratings were distributed within and across grant communities. For this analysis, five equivalent ranges were identified separately for the 4- and 5-point rating scales that essentially represented five categories in which mean scores on principles could fall (see Table 4). Based on the scoring system that was employed, these five categories range from the lowest (range 1) to the highest (range 5) possible expression of system-of-care principles within grant-funded communities. As noted in Table A.1, the lowest possible range (1) indicates that no or almost no efforts have been made, while the highest possible range (5) indicates that efforts have been made, these efforts have been effective, and intended goals have been accomplished.

1Scores within the infrastructure and service delivery domains were negatively skewed, with skewness values ranging from -.110 to -1.419 with a mean of -.523 for infrastructure and from -.875 to .769 with a mean of -.210 within the service delivery domain. No skewness values across either domain were found to be greater than 2.0. Kurtosis values for the infrastructure domain ranged from -1.185 to 2.324 with a mean of .107. Within the service delivery domain, kurtosis values ranged from -1.303 to 1.131 with a mean of -.348. A kurtosis value greater than 2.0 was found within the infrastructure domain for the community-based principle.

Table of Contents | Previous

Back to Top