Measuring Improvement -- Services and Cost

Introduction

Differences in system-of-care service outcomes are influenced by two primary factors--client case mix and service mix. Client case mix refers to the preexisting characteristics among the children and families served by the system of care. As described earlier in this report, the children and families enrolled in the CMHS evaluation are diverse. Families represent many races, ethnicities, and cultures; children vary in age, presenting problems, diagnoses, treatment histories, and risk factors known to be associated with mental health outcomes. Children also differ in the severity of their mental health disorders. Service mix refers to characteristics of services received, including array (i.e., type of services), intensity, duration, and family participation. The experience of system-of-care services is likely to be different from site to site, and the experiences of families within sites are likely to be different as well. It is anticipated that clinical, functional and consumer satisfaction outcomes differ depending on level of mental health challenges, demographic characteristics, and service use patterns. This section of the report provides an overview of an outcome model that examines these relationships. Preliminary clinical/symptomatology data from the CMHS system-of-care evaluation are used to illustrate the associations between child characteristics, service mix, and outcomes over time.

Measuring Outcomes: Child and Family Characteristics and Services Received

Figure 27 illustrates our conceptualization of how child characteristics (i.e., client case mix) and services received (i.e., client service mix) influence outcomes. Aggregating data across all children and families obscures the fact that some subgroups make more dramatic gains while others may experience less favorable outcomes. Subgroups of children also experience the service system differently as evidenced in their utilization patterns. By disaggregating our sample, we can investigate more rigorously which groups of children experience the most favorable outcomes, and describe their utilization patterns. This study of services, costs, and outcomes will provide both aggregate and subgroup (disaggregated) descriptive profiles of service use patterns for the children participating in the evaluation.

Figure 28 illustrates the relationship between client case mix and service mix. Preliminary analyses on client case mix and service mix support the assumption regarding differential outcomes for various groups of children. In this model, client case mix is measured in terms of demographic characteristics, clinical status, and pre-treatment history. Service mix is assessed in terms of service duration, number of services received, and service utilization patterns. Please note that we are not currently collecting data on the natural course of the disorder or data addressing informal network (e.g., family, community, etc.) service supports.

Illustrating the Model: A Site-based Example

A single county-wide system of care comprising a wide array of child-serving agencies and families, was chosen for this illustration. Children are referred into the system of care through several entities including the county Department of Mental Health (DMH), Child Protective Services (CPS), the Department of Health Care Services (DHCS), the Department of Probation (DOP), service providers and organizations in the community, and several school districts. Service information for these analyses came from a county-wide Management Information System (MIS) operated by the DMH.

In this example, analyses were shaped by type and amount of data available from the site. For example, this MIS provided the amount of time children spend in each service making it possible to measure amount of services in hours. This database, however, only recorded formal mental health services and did not include services that might have been available through other sources such as juvenile justice or social services. Only charges for services were provided, thus the cost analyses do not reflect actual costs. The number of children for whom intake and six-month outcome data had been collected was sufficient for these types of analyses. Other MIS and service databases may not have these features, therefore calling for alternative analyses that may be more appropriate. Finally, in order to capture the services received by children before entry into the evaluation, the service data include services received in the four to six weeks prior to intake data collection. These were largely assessment services.

Child Characteristics

The relationship between the child's intake level of social functioning and outcomes is an important consideration. Children with different levels of challenges would be expected to make differential gains. When we aggregate across all children in the system of care, independent of the level of challenge, we lose the ability to identify children who make marked improvement from those who experience subtle change. For this example, children were grouped on the level of intake severity using the total score of the CAFAS. Children were divided into four functioning levels based on CAFAS cut off scores for serious, marked, moderate, and mild/minimal impairment.

Figure 29 illustrates the differential improvements made by each of the groups. Children with higher CAFAS score at intake, indicating more severe challenges, made the most dramatic improvement.

Service Utilization Variables

Service utilization can be operationalized in several ways (Hansson, & Sandlund, 1992). For the purposes of this illustration, we considered services in terms of duration (i.e., hours received), service intensity (i.e., number of different services received), and service mix (i.e., types and amount of services received). In the current study, the multitude of services reported in the MIS were collapsed into broad service categories (see Table 6).⁽¹⁾ All services received four to six weeks prior to the evaluation intake assessment, and those services received until the six-month follow-up data collection were included in these preliminary illustrative analyses. Both the number of service hours and the types of services received were taken directly from the MIS data file. A cluster analysis was used to identify service mix groups--those services most likely to be grouped in a treatment plan. It is important to note that the service mix clusters are specific to this system and the children and families receiving services. It would be necessary to reconsider service mix clusters as the service array and/or system capacity changes, and for each individual system of care, as those differences would result in different clustering. Since the sample size was relatively small (N=97), we limited the clusters to four for these analyses.

Number of Hours Used

Children and families were grouped according to the number of service hours they received. The number of service hours were arranged into quartiles (i.e., less than 30 hours, 30 through 45 hours, 46 through 80 hours, and more than 80 hours of service). Only those services received four to six weeks prior to enrollment into the system of care and up until the six-month data collection were included in this analysis.

Figure 30 provides the results of this analysis. The intake and six-month mean CAFAS scores for the children in each service use group were plotted indicating change in social functioning over time. The children who received the most hours of service (i.e., more than 80 hours) during this interval had markedly higher intake CAFAS scores (i.e., a mean CAFAS Total Score of 92, indicating serious impairment), than did the children in the other groups. Accordingly, the children who received fewer service hours of service during this interval had substantively lower intake CAFAS scores. More service hours were provided to children with serious impairment, and fewer service hours were provided to children with moderate impairment as measured by the CAFAS. These findings indicate that in this system, children experienced service utilization patterns that reflect the differences in their levels of clinical need. It is important to note that all children made improvement between intake and six-month data collection periods, however the children scoring in the serious range experienced greater reductions in severity over time than did the other groups.

Number of Different Types of Services Used

One principle of systems of care is to offer access to a wide variety of services to address children's specific needs. It is assumed that children and families with higher levels of challenge will likely need a broader array of services to address their complex needs. The extent to which this occurred was examined in this example by grouping children based on the number of different types of services they used in the four to six weeks prior to collection of intake data through the six-month data collection period.

Figure 31 illustrates that the children receiving eight or more services also had the highest intake CAFAS scores, a mean score of 91, indicating a serious impairment. Those children receiving four or fewer services had an average CAFAS score of 60, indicating moderate impairment. Again, these findings provide some evidence that in this system of care children use services in accordance with their need. In addition, the children in all three groups showed improvement in level of functioning between intake and six months.

Service Mix

Although the number of hours of services received provides a measure of the amount of services, it indicates nothing about the kinds of services that are represented by those hours. For example, an hour of outpatient services is quite different from an hour of transportation. Similarly, the number of different types of services used provides an indication of the variety of services, but does not include information about variations in the hours received in individual service categories.

To get a clearer picture of the intensity of services children received and the type of services that were received, the information on service hours and service type was integrated into service mix using a cluster analysis (Anderberg, 1973). This cluster analysis yielded the four service mix categories depicted in Figure 32. The service mix categories represent varying degrees of service intensity. For example, the children in Mix 1 used only case management, assessment and outpatient therapy, and they received fewer of those services than the children in the other service mix categories. The children in Mix 4 used a wide variety of services, including wraparound, and used more hours of each service type than the children in the other service mix categories.

The children who used the most intense service mix (i.e., service mix 4) entered the system of care with the greatest challenges evidenced by CAFAS scores in the serious impairment range (i.e., mean score 95). These children made the most dramatic improvement between intake and six month data collection. It is important to note that children who received service mix 3 experienced the greatest number of crisis services in addition to traditional services such as case management and assessment services. While these children did not appreciably improve, they did remain stable, which is a positive outcome for children and families in crisis.

As with other service utilization variables, children who experienced different service mixes also demonstrated differential change in CAFAS scores over time. Again, the children who entered the service system with the greatest challenges and who received the most intensive mix of services, also demonstrated the greatest improvement in functioning six month after intake. Other than the children in service mix 3, each service mix group made improvements in mean CAFAS total scores during this six-month interval.

Cost of Services

To study cost and case mix, children were grouped by level of functioning as measured by the CAFAS. The average cost of serving each functional group (i.e., mild, moderate, marked, severe) was calculated. As mentioned before, this site's MIS recorded only charges for services, not actual costs. Since hospital and residential services were not included in this MIS, the state's reimbursement rate is used as an estimate for comparison purposes.

The analysis of costs revealed the same pattern found with other service utilization variables (see Figure 33). In general, the findings support the expectation that children with the greatest need receive more services than the children with the least need. The lowest average cost of services were associated with the children in the mild impairment group (i.e., $4070). Average costs of services for moderate (i.e., $7,905) and marked (i.e., $12,171) disorders were higher. During the study period, services provided to children with functioning scores in the serious impairment range (i.e., $13,106) at intake cost more, on average, than did the services to children in the other case mix groups. Had those children been served in residential treatment centers, the cost of their care for roughly the same period would have nearly doubled (i.e., 26,100).

Summary

The preliminary findings reported here support the use of a case mix/service mix framework for studying services, costs and child outcomes. Results from one system-of-care site indicated that service mix and case mix are related in the expected ways, that is, children with greater challenges experience utilization patterns characterized by greater involvement in the system. These early findings also provide some evidence that the system of care that was used to illustrate the case/service mix approach was responsive to the needs of children, providing fewer and less intense services to the children with milder problems and more intensive services to children with greater challenges. Moreover, the services provided through the system of care appear to have supported improvements in clinical and functional outcomes for the children in each of the service utilization groups.

1. Note that no information on inpatient hospitalization or other residential treatment services were recorded in this MIS at the time of these analyses.

Table of Contents | Previous | Next