This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
2000 Annual Report to Congress on the Evaluation
|
|
CHAPTER SUMMARY In summary, the services and costs study from the national evaluation indicates that:
|
The primary purpose of the services and costs study is to describe the types of services used by children and families, their utilization patterns, and the associated costs. This information can be used in the aggregate to track system variables over time such as shifts in system expenditures, service use patterns (e.g., reductions in use of residential services, increase in family support service use), and proportion of dollars from various sources that contribute to the system. With multi-agency services and costs information, aggregate data can be used to identify cost-shifting across service sectors (e.g., from education to mental health). Information on the types of services children and families receive also helps to describe, understand, and explain differential improvement among children and families served in systems of care. Without information on the type, length, or intensity of services, one cannot examine the impact of services on outcomes. For example, a system may see clear improvements in symptoms and functioning in some children across time. However, if services data are not available, much remains unknown about what types, combinations, or intensity of services contributed to those improvements. Some questions that can be addressed with services and costs data are:
A consistently found predictor of children's mental health service use is the presence of psychological disorders (e.g., Burns, Gwaltney, & Bishop, 1995; Offord et al., 1987; Zahner, Pawelkiewicz, DeFrancesco, & Adnopoz, 1992); however, the majority of mental health services are consumed by a small percentage of children (Burns, Costello, et al., 1995). Although the evidence suggests that the presence of psychological problems is a powerful predictor of any service use, disorder type and symptom severity do not appear to strongly influence how services are used (Friedman & Street, 1985; Hundert, Cassie, & Johnston, 1988). Burns (1991), for example, observed few differences in levels of care across diagnostic groups with the one exception being more children with affective disorders were treated in hospitals. Similarly, Dore, Wilkinson, and Sonis (1992) found that the severity of problems did not determine the restrictiveness of services except for children receiving outpatient services. One exception is the preliminary evidence that children with externalizing problems are more likely to enter mental health treatment than children with internalizing problems (Wu et al., 1999).
Some demographic variables have been found to be related to service utilization. In general, boys use more mental health services than girls (LaClave & Campbell, 1986; Padgett, Patrick, Burns, Schlesinger, & Cohen, 1993; Roghmann, Babigian, Goldberg, & Zastowny, 1982), especially services provided in residential settings (Burns, 1991). Some authors speculate that the predominance of conduct disorder among boys compared to girls may account, in part, for this discrepancy (Garralda & Bailey, 1988). Externalizing problems are much more likely to come to the attention of parents, teachers, and law enforcement than internalizing problems. There appears to be no evidence that gender is related to treatment dropout (Armbruster & Fallon, 1994; Beers, 1992; Bui & Takeuchi, 1992; Kaminer, Tarter, Bukstein, & Kabene, 1992).
Evidence suggests that mental health service use increases with age until middle age and then decreases. This relationship has been referred to as the inverted "U" in contrast to the upright "U" exhibited for physical health services (Scheffler & Miller, 1991). Hence, small children tend to be less likely to use mental health services and adolescents tend to use more (Padgett et al., 1993; Realmutto, Bernstein, Maglothin, & Pandey, 1992). Once in treatment, however, adolescents tend to drop out of services earlier and have shorter lengths of stay than younger children (Bui & Takeuchi, 1992).
The literature on the impact of race and ethnicity on children's mental health service use is not as developed, but some research does exist from the last two decades. Using hospital survey data on 1986 admissions, Burns (1991) found that minority children (aged 10 to 18) used outpatient and inpatient services in proportions similar to their representation in the general population but were over-represented in partial hospitalization programs. Minority children in a sample of users of publicly funded outpatient services in Minnesota also used a proportional amount of services (Realmutto et al., 1992). With the effects of age, gender, and poverty controlled, researchers found that African-American adolescents received more episodes of care in an outpatient clinic (Bui & Takeuchi, 1992). Another clinic study, however, found no link between race/ethnicity and amount of outpatient service usage (Tolan, Ryan, & Jaffe, 1988). In a sample of insured children, White children had a higher probability of outpatient service use than their African-American and Hispanic counterparts (Padgett et al., 1993). Race and ethnicity were not found be significant predictors of dropout from outpatient treatment in the studies reviewed here (Bui & Takeuchi, 1992; Gould, Shaffer, & Kaplan, 1985; Tolan, Ryan, & Jaffe, 1988); however, a recent meta-analysis of 125 studies found that minority status appeared to be associated with early withdrawal from services (Wierzbicki & Pekarik, 1993). Other studies found that the effect of race on attrition from therapy ceased to be significant when the effect of socioeconomic status was controlled (Armbruster & Fallon, 1994; Viale-Val, Rosenthal, Curtiss, & Marohn, 1984). It is likely that barriers such as language and culture may influence whether and how children use mental health services.
Study of the role of the family in the treatment of mental health problems has become critical as inpatient and residential treatment settings increasingly are recognized as overly restrictive for many children. The emphasis on community-based alternatives places greater responsibility on families in the treatment and management of mental health care. As children seldom seek mental health services for themselves, families play an important role in bringing children to treatment (Kazdin, 1989). Caregivers are subsequently important agents in making and following through on treatment decisions and in supporting gains outside the therapeutic milieu (Friesen & Koroloff, 1990). There is evidence that parents' level of education affects children's mental health service use. One epidemiological study found that children whose mothers had at least high school educations were approximately twice as likely to have used mental health services as children whose mothers had not finished high school (John, Offord, Boyle, & Racine, 1995). Among families with similar insurance coverage, children whose parents had college degrees were more likely to use outpatient mental health services (Padgett et al., 1993).
In studies of clinic samples, children from disadvantaged homes appeared more likely to receive mental health services (Burns, Costello, et al., 1995) and to be referred to child psychiatrists (Garralda & Bailey, 1988) than other children. Conversely, several studies found no relationship between children's premature termination from treatment and their parents' education (Kaminer et al., 1992) or socioeconomic status (Beers, 1992; Dover, Leahy, & Foreman, 1994; Gilbert, Fine, & Haley, 1994; Gould et al., 1985; Kaminer et al., 1992). One study found that children from families in the upper socioeconomic groups were more likely to complete outpatient treatment (Armbruster & Fallon, 1994), and another found that children from "non-poor" families remained in services longer than adolescents from poor families (Bui & Takeuchi, 1992). Viale-Val and colleagues (1984) found that social class had varying impact at different stages of the therapeutic process.
No relationship was found between maternal mental health status and the amount of health services (both physical and mental) used by children, when other factors were controlled (Riley et al., 1993). Although some have found a relationship between mothers' psychological impairment and their children's default from treatment (e.g., Dover et al., 1984; Gould et al., 1985, among others), others found no such relationship (Beers, 1992).
Previous service utilization studies have found that service delivery systems (i.e., clinically managed care vs. traditional fee-for-service) can have profound effects on how children use mental health services (Summerfelt, Foster, & Saunders, 1996), and that client and family predictors of hospitalization may differ across delivery systems (Bickman, Foster, & Lambert, 1996). In addition, there is evidence that client (e.g., symptomatology, functioning, age) and caregiver and family factors (e.g., caregiver strain, family functioning) predict the combination of outpatient, intermediate, and residential services that children use in a continuum of care (Foster, Summerfelt, & Saunders, 1996) and in traditional community settings (Farmer, Burns, Angold, & Costello, 1997). When children are grouped by their service utilization patterns, it appears that child demographic (e.g., age) and clinical (e.g., symptomatology, functioning) variables as well as caregiver and family variables (e.g., caregiver strain, material resources, family functioning) can predict membership in those groups (Brannan, 1996; Lambert, Brannan, Breda, Heflinger, & Bickman, 1998). In contrast, the study of the relationship between children's mental health service use and child outcomes is scant and contradictory (Angold et al., 2000; Hoagwood, 2000), even when the analyses are conducted on data from the same sample (compare, for example, Andrade, Lambert, & Bickman, 2000; Foster, 2000).
There are additional issues that need to be addressed to more completely understand mental health service use for children and their families and the costs associated with these services. The impact of including multiple community demonstration sites to evaluate socially complex services raises a number of important issues (Wolff, 2000) that are now being addressed within the adult mental health services literature (Clark et al., 1998). Capturing the services components of different delivery systems is a complex task that becomes even more complex when incomplete and inconsistent data across study sites are available from electronic management information systems (MIS). For example, the impact of variability across MIS's for sites funded by the Comprehensive Community Mental Health Services for Children and Their Families Program has been highlighted as an important barrier that needs to be overcome to improve the capacity to conduct services and costs analyses effectively (Foster, Kelsch, Kamradt, Sosna, & Yang, in press). Benefit-cost analyses have rarely been applied in analyzing the impact of different models of children's mental health treatment and the coordination of services for children and their families within community settings. This type of analysis requires an expanded conceptual framework that includes the consideration of other societal costs and benefits, including other government programs beyond public mental health services, the private sector, and personal/family expenditures (Foster & Holden, in press). Expanded resources and data collection methodologies will need to be developed to conduct meaningful benefit-cost analyses within community settings that employ the system-of-care approach. Creative strategies for developing and sustaining cross-agency information sharing will also be needed in the future to improve the yield from studies assessing the services and costs associated with children's mental health services programs (Woodbridge & Huang, 2000).
Clearly, much remains to be understood about what characteristics predispose children to use mental health services, what factors predict utilization patterns, and the relationship between service use and child and family outcomes. The national evaluation seeks to add to this knowledge base within the confines of the data available in the grant communities. This chapter describes the ideal services and costs database, identifies potential difficulties inherent in using existing data sources, and provides some findings from grant communities funded in 1993-94. Additional data sources have been pursued in the Phase I comparison study and, as that evaluation component comes to closure in the next year, additional information on these issues will become available.
To readily address questions related to mental health service use and costs, stakeholders need access to useful information on the amount, type, and costs of services children and families receive. |
To readily address questions related to mental health service use and costs, stakeholders need access to useful information on the amount, type, and costs of services children and families receive. Because services and costs data are complex, it is imperative that data be maintained in a management information system (MIS) capable of categorizing data into useful components. An MIS is the technical infrastructure for recording information that can be used to manage services and systems and to answer key conceptual and policy questions. An effective MIS allows flexibility in manipulation and analysis of the data. In addition, the ideal MIS should have the following characteristics:
First, a system-of-care MIS should include all services received by children and families, including traditional and nontraditional services. The latter may include family support services, recreational activities, independent living services, or instrumental support (e.g., payments for rent, utilities, or temporary living expenses) paid with flexible funds. Nontraditional services may represent only a small fraction of program expenditures, but because they are a distinguishing feature of systems of care and have a considerable impact on child and family outcomes, they are essential to evaluation. Historically, many nontraditional or innovative services were funded through government service grants and were not reimbursed by third-party payers. In these service grant programs, there was no fiscal incentive to record individual services received by children and families. Instead, it was sufficient for accountability purposes to record aggregate numbers of services rendered or numbers of children served without tracking what services were provided to whom. These service recording practices have implications for the sufficiency and comprehensiveness of systems' MIS's.
MIS data should also include mental health services not funded through grant program activities such as residential services. While systems of care are committed to providing services in the least restrictive environment, residential services may be a necessary component of a complete continuum of mental health services and should be available in the rare event that a child needs them. Residential services are also very expensive, and no costs study is complete without them. These services need to be included in the MIS data in order to address important system-level indicators (e.g., whether there has been a reduction in the use of residential services) as well as child-level outcomes (e.g., number of re-admissions).
MIS data should also include services provided by or through multiple service sectors, including mental health, child welfare, juvenile justice, education, and physical health. In some instances, obtaining the necessary information from other public agencies is quite challenging because expenditure data are seldom linked to specific services or even stored in an electronic MIS. These data are critical, however, for analyses of cost-shifting and multi-agency involvement in systems of care.
A second desirable feature of MIS data is that the services included are grouped in meaningful categories. Those categories should not be aggregated to the extent that meaningful differences across services are obscured. For example, nontraditional services (e.g., certain family support services) may be billed as more traditional services (e.g., case management) because no appropriate code has been created or to maximize reimbursement of services. Conversely, an MIS may provide excessive detail about service type. For example, the MIS might differentiate among different types of outpatient therapy (e.g., art therapy, psychotherapy, play therapy). This specificity might have few implications for expenditures but may be of interest in the study of the relationship between services received and outcomes. In any case, categorization of service variables should be meaningful and automated to ensure effective data utilization.
Essential Components of MIS Systems
|
A third desirable feature of an MIS is that it include individual identifiers that allow the researcher to characterize services received by a given individual. This information could allow one to describe service utilization patterns or episodes of care. Individual identifiers can also be used to link services data to information on child and family outcomes. Such linkages can help to identify the amount and sequencing of services contributing to positive outcomes for children with specific characteristics.
A fourth desirable feature of an MIS is that it include information on expenditures linked to specific services. Such information is critical for calculating and understanding differences in expenditures across individuals. It can also be used to monitor system performance and efficiency. For example, services and costs data can be used to show that reductions in the use of costly residential services allow the system to provide more services to more children and families.
Finally, the data provided in the MIS must be complete and accurate. This includes accuracy of the information on service type and units as well as individual identifiers. If the latter are inaccurate, then problems with matching the services to specific individuals will occur. For expensive services, even seemingly minor omissions or errors can have profound effects on findings. Hence, it is imperative that a routine and systematic procedure be in place at the agency to monitor the quality of the data in the MIS. Incomplete or inaccurate data can be supplemented by coding information from clinical charts that may detail specific services that have been provided to individual children and their families. This is an extremely time- and cost-intensive data collection strategy that reiles upon the assumption that clinical records are accurate and complete, an assumption that is rarely met in the field.
Sample Service Use Variables
|
Mental health service utilization patterns are defined as successive service events that describe the history or course of mental health care (e.g., Haas-Wilson, Cheadle, & Sheffler, 1994; Hornbrook, Hurtado, & Johnson, 1985). Beyond this broad definition, service use patterns can be described in a variety of ways (e.g., Hansson & Sandlund, 1992; Steinwachs, 1990), and different approaches can be used for different analyses. Because services and costs MIS's are seldom developed for research purposes, most researchers must shape their analyses to fit the type of information held in the MIS and not vice versa.
Nonetheless, to describe service use patterns and to examine the associated costs, the following fundamental data elements are important:
|
|
|
|
Even when these data elements are present, it is important to be aware of changes that have occurred in the recording of data over time. As systems develop, it is not unusual for administrators to refine service encounter forms and recording procedures. For example, imagine that a grant community targeted service accessibility as the focus for quality improvement efforts. There had always been a field on the service encounter form for clinicians to record the location of the service, but staff completed that field only sporadically. Upon launching the quality monitoring effort, clinicians were instructed to record location on every encounter form. Analysis of the data may indicate that clinicians were providing more services in families' homes than before, but only because there had been a change in recording procedure, not in service provision.
Finally, it is important to consider the primary purposes for which the MIS is used. MIS's typically are developed for system management, fiscal, and accountability purposes. Therefore, there are often incentives and disincentives for recording or not recording information. For example, if the MIS is used as a billing system, there will be considerable incentive to record every billable service, but little incentive to record services that cannot be reimbursed. This is particularly true in many community mental health services today where direct service providers are expected to bill a certain number of hours to justify their positions. In general, all MIS's come with some unevenness, and it is critical that the nature of, and reasons for, those inconsistencies be explicitly investigated and documented.
Figure 68 visually summarizes the final status of services and costs data obtained from 31 separate sites within the 22 grant communities funded in 1993-94. Twenty-five of the sites reported that they had some form of electronic MIS system. Seventeen of the 31 sites provided some type of services or costs data. This included services data from 16 sites and corresponding costs data from 15 of these sites. MIS data from 10 of these sites contained services data from other partner agencies. However, data from 7 of the 31 sites could not be used for a variety of reasons. Some of these sites extracted service information from individual case records to summarize data for a small subset of the children served. Other sites attempted to "estimate" the services children and families may have received by reviewing individualized service plans. Yet other sites provided aggregate summaries of grant expenditures by service category. Of the sites that provided no services and costs information, one was developing an extensive MIS that would eventually integrate demographic, clinical, outcomes, and services information, but that system was not fully operational for the majority of their funding period. Another site did have local services information, but did not have access to that data. Other sites tried to access State Medicaid data, but were unsuccessful.
Ultimately, the MIS's from 10 sites within 9 grant communities were considered adequate for use in the national evaluation, but these, too, presented challenges. Almost every data set received was extracted from the fiscal MIS (i.e., billing system) of a community mental health center. While a proportion of children served through the grant program can be expected to be involved with multiple agencies, services provided by or funded through other agencies were not typically included. For example, services such as family preservation services would more likely be provided through the child welfare agency and therefore would not appear in the billing system of a community mental health system. In addition, most of the non-billable services provided in grant communities (e.g., recreational services) typically were not included in their MIS's. Other services children received that were not provided by community mental health centers, most notably residential services, also were seldom recorded. This was also true of the data provided from one State public mental health MIS because in that State, residential services were paid primarily through the education sector. Indeed, only one of the 10 electronic MIS's included residential services such as psychiatric hospitals and residential treatment centers.
This section briefly summarizes findings from Phase I services and costs study analyses that have been presented in previous Annual Reports to Congress (CMHS, 1997, 1998, 1999), research conferences, and journal articles (Foster et al., in press; Hodges, Doucette-Gates, & Kim, 2000; Hodges & Kim, 2000). More in-depth information regarding these analyses can be obtained from the above citations. Across these various sources, services and/or costs findings have been reported for 10 sites within 9 grant communities funded in 1993-94.1 Because the MIS's from these sites differed significantly, analyses conducted differ across sites. In general, however, the findings summarized below can be organized by the following areas: (a) description of child mental health service utilization patterns; (b) examination of which baseline clinical, functioning, and demographic characteristics contribute to or predict service use; and (c) exploration of the relationship between service use and mental health outcomes. The number of sites differs across analyses depending upon the data that were available and whether those analyses were reported previously.
Utilization patterns can be described in a variety of ways. Length of service, amount of services received, continuity of care, and types of services received are discussed. It is important to note that programmatic differences across sites and in the information kept in their MIS's (e.g., types of services included, definitions of categories) make direct comparisons across sites difficult. For example, the Milwaukee MIS included a fairly broad range of outpatient and residential services, while all others included primarily reimbursable outpatient services only.
The mean average length of service across all 10 sites was 17 months. There was considerable variation, however, with average lengths of service ranging from 11 months in the Baltimore, Maryland, site to 23 months in the Philadelphia and the Vermont sites. |
Length of stay in service was computed as the time between a child's first service encounter and their last, as recorded in the site's MIS. The mean average length of service across all 10 sites was 17 months. There was considerable variation, however, with average lengths of service ranging from 11 months in the Baltimore, Maryland, site to 23 months in the Philadelphia and the Vermont sites (CMHS, 1998, 1999). The variation in lengths of service was due, in part, to programmatic factors. For example, the Milwaukee program was designed to move children from residential treatment settings to the community with 1 year of aftercare to facilitate successful transition. Hence, the average length of service in Milwaukee was 12 months with a relatively even distribution across children served (CMHS, 1999). In sites where lengths of service were longer, such as Philadelphia, it appears that support services such as case management continued long after children were no longer receiving more intensive clinical services (CMHS, 1999).
Amount of services received has been quantified for the Phase I evaluation in three primary ways: number of service encounters, number of hours of services, and costs of services received. The average number of service encounters per month has been reported individually for five sites (CMHS, 1998). Among these sites, the average number of service encounters per month ranged from four in Charleston to nine in Baltimore and Vermont. Accordingly, the average number of service hours received per month ranged from approximately 3 in Charleston to 11 in Baltimore and 16 in Vermont. Comparison of Baltimore and Vermont indicates that while the average number of service contacts was similar, individual encounters appeared to be longer in Vermont.
Median total hours of services received have been reported for the following five sites: Santa Barbara, California; Wai'anae Coast, Hawai'i; Edgecombe, Nash, and Pitt Counties, North Carolina; Philadelphia, Pennsylvania; and Milwaukee, Wisconsin (CMHS, 1999). The total median hours of services received ranged from 68 in the North Carolina counties to 809 in Milwaukee. Recall that Milwaukee's MIS included residential services in which one bed-day translated to 24 hours of service. Had the other MIS's also recorded residential services, the disparity may not have appeared as large. The costs of services received was reported for these five sites as well. Median total costs per child ranged broadly across sites from $4,689 in the North Carolina counties to $35,469 in Milwaukee (CMHS, 1999). Milwaukee's unadjusted mean costs of $25,742 for a 6-month period (Foster et al., in press) were significantly higher than the other sites, reflecting the inclusion of residential services in the Milwaukee data set. Comparisons to costs data from other children's mental health services programs indicated general comparability, especially when severity level of the population being served in Milwaukee was considered.
For all 10 sites, continuity of care has been measured as number of breaks in service of 30 days or longer (CMHS, 1998, 1999). The number of breaks in service ranged from four in the North Carolina counties to one in Milwaukee (CMHS, 1999). In all other sites, children experienced an average of two breaks during the time they were receiving services, a remarkably consistent finding (CMHS, 1998; 1999). Only 12 percent of the children served in Milwaukee experienced a break in service of 30 days or longer, compared to 64-79 percent of children in other sites (CMHS, 1999). The continuity of care found for Milwaukee may have been related to the considerably higher proportion of children in Milwaukee who were referred to mental health services by the juvenile justice system and were more likely to have been court-ordered to care. Again, however, interpretation of these results is compromised by missing information on residential services in all but one of these MIS's. Some of the breaks reported for sites other than Milwaukee may have in actuality been stays in residential settings not recorded in these MIS's.
The distribution of services across children has been reported for Santa Barbara, Hawai'i, and Milwaukee (Foster et al., in press). Among these sites, case management was used by a greater proportion of children (98-99 percent) than any other single service. Beyond that, the distribution of service use varies considerably across sites, and the meaning of this variation is unclear. For example, substance abuse services reportedly were provided to 11 percent of the children in Santa Barbara, 2 percent in Milwaukee, and no children in Hawai'i. It is unknown, however, whether children received such services in settings not captured in these MIS's. Programmatic differences appear to explain some differences, however. That 35 percent of the children in Santa Barbara received crisis services is likely linked to the site's commitment to serving the children most in need of crisis intervention and stabilization as they entered the system of care.
Findings from grant communities funded in 1993-94 support previous research that severity of clinical status is a powerful predictor of service use. Analysis of data from two sites, East Baltimore and Santa Barbara, showed that children with greater functional impairment (measured with the CAFAS) were more likely to use more services. In the Santa Barbara sample, the more severely impaired children were, the more likely they were to use more hours of services and different types of services (CMHS, 1997). These findings were replicated at the East Baltimore site (CMHS, 1998), where children with the highest levels of impairment at baseline experienced the greatest intensity of services (defined as the total number of hours divided by the length of time over which those services were received).
Risk factors have also been found to predict service use. In Stark County, children who had a history of running away, drug or alcohol problems, or sexual abuse utilized considerably more service hours than children without those risk factors (CMHS, 1998). In contrast, demographic variables (e.g., gender, age, income) did not predict restrictiveness of services received during the subsequent 6 months in the Hawai'i, Milwaukee, or Santa Barbara sites (CMHS, 1999), although CAFAS total score did so in all three sites. In two of the sites, Santa Barbara and Milwaukee, children who had previously received psychiatric inpatient or residential treatment center services were more likely to use those services again. For these analyses, services data obtained from the Santa Barbara and Hawai'i MIS's were augmented by caregiver report data from the Restrictiveness of Living Environment Scale (ROLES). It should be noted that a validation analysis using the more comprehensive Milwaukee MIS data showed that caregivers substantially under-reported their children's use of residential services (CMHS, 1999). Therefore, these findings likely underestimate the relationship between risk factors and use of restrictive services.
There have been consistent findings across several sites that children who used more services tended to experience greater improvements. |
Several interesting findings have emerged from the Phase I services and costs study regarding the relationship between service use and child outcomes. In addition to showing that children with greater challenges at baseline use more services subsequently, there have been consistent findings across several sites that children who used more services tended to experience greater improvements. In both the Charleston and Santa Barbara samples, children who used more hours of services showed greater reductions in impairment as measured with the CAFAS total score than children who received fewer hours of services (CMHS, 1997, 1998). In addition, children in Charleston and Vermont who experienced better continuity of care (e.g., no breaks in service of 30 days or more) showed significantly greater reductions in symptoms on the CBCL Total Problems score and improved functioning as measured by the CAFAS total score than children who experienced poor continuity of care (CMHS, 1998). In Vermont, those gains continued to the 12-month data collection period for children with no breaks in care, while children with breaks experienced some reversals of gains achieved at 6 months (CMHS, 1998).
There also has been some evidence that types of services used impact clinical outcomes. In the Santa Barbara sample, children who received the most intense mix of services (i.e., wider range of service types and more of each service) showed dramatically more improvement in functioning than children who used less intense service mixes (CMHS, 1997). In Stark County, children who received intensive in-home services showed far greater improvement in functioning within 6 months than children who did not receive those services (CMHS, 1998).
These findings provide some support for the position that the amount and type of services children receive as well as the continuity of their care impacts clinical outcomes for children. In addition, these relationships appear stronger for children with the greatest challenges. Caution is warranted, however, because a substantial proportion of children were lost to the 6- and 12-month data collection points due to the replacement sampling design used in Phase I.
The experiences encountered during Phase I led to the crafting of an alternative approach to the services and costs study in Phase II and Phase III of the evaluation. This alternative approach initially involves a complete assessment of each grant community's baseline MIS capacity to determine the existence of existing sources of services and costs data. An additional feature of the approach involves providing technical assistance to the grant communities to develop their MIS capacity further. Furthermore, a multi-tiered approach to data analyses that utilizes data from subsets of grant communities in a stepwise fashion to address increasingly complex services and costs questions will be implemented.
To determine the completeness of computerized services and costs data, a survey was conducted during FY 2000 to assess the MIS capacity of the grant communities funded in 1997-98 (Phase II). The survey provided preliminary information and a foundation for more extensive efforts of the national evaluation team to assist communities in identifying and utilizing community-level sources of MIS data. The survey addressed a variety of MIS characteristics and covered a range of services. The grant communities responded to specific questions in each service area about the presence of electronic records, the location of MIS's, linkages of records to individual children, and methods of recording data about the costs of services. Findings revealed how closely the available data in the communities met the ideal MIS standards described previously in this chapter.
The survey was distributed to the local evaluation teams in the funded grant communities, and follow-up telephone interviews were conducted with those who did not respond or returned incomplete surveys. As of July 2000, 22 sites representing 19 grant communities responded (a response rate of 83 percent). Follow-up is ongoing with grant communities that have not responded in order to describe the MIS capacity of all funded grant communities.
Of the 19 grant communities that completed the survey, an MIS was available in 15 (18 individual sites). An additional grant community reported information about an MIS they were planning to develop; that information is included in these data in anticipation that the MIS will be completed as planned. The MIS's described in the surveys generally included information on most of the mental health services provided within the grant community. However, the range of services recorded electronically varied significantly both within and across grant-funded communities. As displayed in Figure 69, for the 15 grant communities reporting electronic MIS records, most recorded more traditional services such as medication management; outpatient individual, family, and group therapy; and case management. However, innovative, community-based services such as respite, independent living, and transitional services were less likely to be recorded electronically. Therefore, any direct analyses utilizing these electronic sources of data would underestimate the frequency of both costly, restrictive services and unique services that are likely to be important ingredients contributing to the effectiveness of systems of care.
Most of the services and costs information that is available at the grant communities is contained in a single agency MIS, usually housed within the children's mental health agency (53.5 percent). Almost half of the grant communities participating in the survey, however, reported that additional MIS data are housed across multiple agencies. Those grant communities reporting the existence of multiple agency MIS's documented that electronic databases exist in another two to five partner agencies (see Figure 70). However, it is important to consider the percentage of grant communities that reported having access to these electronic data systems within specific agencies. Over 90 percent of the grant communities reported having access to electronic data systems within mental health agencies, but only one-third of the grant communities indicated having access to the MIS's within social services or other community agencies. Only 20 percent of the grant communities had access to the information systems maintained within the educational system, and 13 percent reported direct access to MIS's within the juvenile justice and health agencies (see Figure 71).
As discussed previously, a desirable feature of an MIS is that it include expenditures linked to specific services. There is wide variability across the grant communities in the availability of this information. In at least one grant community, no costs data are recorded for services. At most grant communities, costs (or expenditure) data are recorded for at least some services. There are key omissions, however. In several grant communities, no costs data are available for inpatient care or residential treatment centers. Other grant communities lack costs data for other services such as group or individual therapy. In some instances, the availability of expenditure data depends on payment source. For example, in one grant community, costs data are available for group therapy if the mental health agency paid for the service but not if it was provided by the juvenile justice system.
Some grant communities have developed links across child-serving sectors. Vermont obtains data from mental health, social services, physical health, and education agencies. The Families and Communities Equal Success (FACES) program in Cleveland County, North Carolina, obtains information from education and juvenile justice agencies. |
Some grant communities have developed links across child-serving sectors. Vermont obtains data from mental health, social services, physical health, and education agencies. The Families and Communities Equal Success (FACES) program in Cleveland County, North Carolina, obtains information from education and juvenile justice agencies. Links across information system data within the child-serving sector are typically made through complex computer algorithms that match administrative records rather than through shared identifiers across agencies.
The process of technical assistance for grant communities funded in 1997-98 regarding the development of MIS data has already been initiated. Workshops have been provided at grant communities meetings during the past 2 years to inform grant communities of the sources of data that may exist in their communities and to assist them with crafting a plan for accessing and transferring such data. Transmission of services and costs data from grant communities funded in 1997-98 to the national evaluator will begin in the upcoming year.
As part of an assessment of the evaluation technical assistance needs of grant communities funded in 1999 (Phase III) conducted in spring 2000, agency staff answered a series of questions about the capacity of their MIS's. All grant communities but one indicated that the agency maintained an electronic MIS, and that the data it contained could be linked to specific individuals participating in the national evaluation. However, there are issues of access and data quality that may unfold over time as the evaluators gain experience with the data. One grant community acknowledged that there are limitations in the availability of services and costs data and questions regarding their reliability and validity.
Most grant communities expressed an interest in cross-agency information systems, but there is considerably less progress in developing such systems. Most grant communities mentioned plans for such systems, but those plans appear to be long-term. At one grant community, for example, a pilot project for sharing information will begin in the project's third year. At another grant community, agency-level planning for a State-run or State-level data warehouse is just beginning. Most grant communities seem to have some knowledge of the data available in other agency or sector systems. One grant community, for example, reported that the child welfare agency keeps services information but does not know whether that system includes costs information. Other grant communities are investigating data availability. For example, staff at one grant community are making site visits to assess the availability of MIS data at other key agencies.
Some grant communities that have made more progress are still dealing with technical issues. Project MATCH in Pima County, Arizona, has enlisted the juvenile justice, protective services, and developmental disabilities agencies to record services and costs information. They have engaged a consultant to work toward linking the various systems. In other grant communities, technical problems are apparent. In one grant community, a cross-agency system is being developed, but that system will contain no individual identifiers. As a result, it will be impossible to link children in the national evaluation to those data, at least initially. Other grant communities have successfully addressed technical problems only to encounter administrative problems. In Delaware, a comprehensive cross-agency MIS was developed with significant Federal supports, but the juvenile justice and social services agencies have been reluctant to integrate with the new system.
In sum, the needs assessment results from the grant communities funded in 1999 are encouraging. Most grant communities report having made some progress in developing an MIS, and roughly half indicated that obtaining assistance in developing or obtaining an MIS is a high priority. Most acknowledge the importance of accessing data from other sectors. It is clear, however, that significant barriers await the development and analysis of interagency data.
The next step in the process of working with the grant communities funded in 1999 is to conduct, beginning in winter 2001, a more in-depth survey of MIS capacity similar to the ongoing survey of programs in grant communities funded in 1997-98. Initial contacts with these grant communities suggest that more extensive MIS capacity already exists at the newly funded programs, although there certainly is variability present in existing services and costs data.
Information from Phase I and initial data from Phase II of the national evaluation indicate that to describe the broad array of services offered within systems of care, multiple sources of data (e.g., from various agencies) must be used. However, experience indicates that gathering multi-source data from all grant communities is an inefficient method that does not provide stakeholders with needed information in a timely fashion. Grant communities are in different stages of development with regard to tracking services, costs, and insurance information. Some grant communities have highly developed MIS's that are linked across agencies and are used for continuous program development, reimbursement, third-party billing, and State reporting requirements. Other grant communities may have little or no electronic information and limited resources to develop MIS's. Unfortunately, a multi-agency MIS system is the exception rather than the rule, and gleaning data from less developed information systems may not yield fruitful data by which to evaluate system-of-care programs.
Given these experiences, the national evaluation is implementing a multi-tiered services and costs analysis protocol that compiles essential information from all grant communities about services delivered. The protocol uses parent reports from some grant communities and more detailed and extensive data from progressively smaller groups of grant communities that have electronic fiscal MIS's. In the future, this strategy will result in a more complete assessment of the various services that are being delivered in grant communities. In addition, it will help to establish a fuller understanding of the costs associated with these services and the general costs that result from building and maintaining a system of care. The MIS data that are currently being identified and collected in Phases II and III will provide the necessary information for implementing this approach to services and costs analyses.
The minimum services data that are available across all grant communities funded in 1997-98 and 1999, regardless of the existence of an electronic MIS, are caregiver reports of service use. This information is collected with a comprehensive questionnaire administered to caregivers every 6 months after they enter system-of-care services as part of the outcomes study. Information collected with this measure includes parent perspectives of types of services, extent of services, location of services, and overall evaluation of whether the services met the family's needs. Preliminary analyses of results from this measure obtained at 6 months into services are discussed on pages 88-90. Utilizing these data, the questions that can be addressed at all grant communities include:
Because these data are collected at all grant communities, they improve our understanding of how families in the outcomes study use services. This measurement strategy records services used in multiple settings, including school, jail, and various agency locations (e.g., child welfare). In addition, caregivers report on support services not often recorded in traditional MIS's. However, because no cost data are collected, it is not possible to derive cost estimates from this information.
Fiscal sustainability is defined as the ability of the system of care to maintain the level of funding after the expiration of the Federal grant. |
Fiscal sustainability is defined as the ability of the system of care to maintain the level of funding after the expiration of the Federal grant. This component of the evaluation will include a subset of up to 10 grant communities and will answer several questions:
These questions will be answered by analyzing aggregate budget data (not found in the MIS) that include not only the CMHS Federal grant dollars' contribution, but also other sources (e.g., State, Federal, blended agency funding) that contribute to the development and sustainability of the system of care. In addition to budget information, documentation regarding relevant rules and policies surrounding resource allocation will be analyzed. For example, documentation of rules regarding Medicaid reimbursement will be obtained and reviewed.
With grant communities that record reimbursable services (e.g., outpatient individual and group therapy, medication monitoring, case management) for the purposes of billing, an additional level of analysis will be conducted. Using supplemental evaluation information available in these grant communities, the following questions will be addressed:
To describe amounts of reimbursable services received, service utilization patterns experienced, and the unit cost of services billed, it will be necessary to glean data from mental health administrative databases and fiscal MIS's. As discussed previously, initial surveys have been conducted with grant communities funded in 1997-98 and 1999 to identify the capacity of existing MIS data systems. Further work with these grant communities will isolate a subset that has the most reliable and comprehensive MIS's. These grant communities will then be targeted for this more intensive level of analysis.
For a smaller group of grant communities, supplemental data will be gathered from other child-serving agency MIS's along with concurrent budget information to determine:
Data will be obtained about services from the education, juvenile justice, child welfare, public health, and other participating agencies. In addition, data collected from caregiver reports will be used to supplement agency data. Grant communities will be chosen based on the comprehensiveness of their MIS data across agencies and their ability to access that information across agencies.
A comprehensive viewpoint is necessary to understand fully cost offsets and the societal impact of the services delivered within systems of care. A fully integrated multi-agency MIS is necessary to test fully the utility of the model as a guidepost for predicting service utilization and accompanying costs within systems of care. |
Figure 72 depicts the relationships between various costs incurred at several levels and in multiple parts of the child-serving sector as children with serious emotional disturbance participate in systems of care. This model indicates that a comprehensive viewpoint is necessary to understand fully cost offsets and the societal impact of the services delivered within systems of care. A fully integrated multi-agency MIS is necessary to test fully the utility of the model as a guidepost for predicting service utilization and accompanying costs within systems of care.
Perhaps the greatest contribution made in Phase I to the study of child mental health service use and costs is related not to the findings but to a greater understanding of local capacity for maintaining services and costs MIS's that can be used for program monitoring and evaluation purposes. It is clear that grant communities have varied and typically limited abilities to collect, maintain, and utilize information on the services received by children and the associated costs. Since the beginning of Phase I, issues related to MIS development have entered the national discussion. Indeed, early indications suggest that more recently funded grant communities have a greater capacity for maintaining services and costs MIS's and that more resources are being directed toward this area. As this grant program continues, it is expected that much more will be learned about whether, why, and how children and families use mental health and support services and the impact of service use on children and families.
1These sites included Baltimore, Maryland; Charleston, South Carolina; the Southern Consortium, Ohio; Stark County, Ohio; the State of Vermont; Santa Barbara, California; Wai'anae Coast, Hawai'i; Philadelphia, Pennsylvania; Pitt, Edgecombe, and Nash Counties, North Carolina; and Milwaukee, Wisconsin.
|
| Home | Contact Us | About Us | Awards | Accessibility | Privacy and Disclaimer Statement | Site Map |
