This Web site is a component of the SAMHSA Health Information Network
|
This Web site is a component of the SAMHSA Health Information Network |
| | | | | | |  |
||||||||||
 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This Web site is a component of the SAMHSA Health Information Network. |
|
Section 4: Key Elements of the National Statistical PictureChapter 17. The 16-State Indicator Pilot Grant Project: Selected Performance Indicators and Implications for Policy-and DecisionmakingOlinda González, Ph.D.;* Judy Hall, Ph.D.;† John A. Pandiani, Ph.D.;‡ John McGrew, Ph.D.;§ Amy Elliott, B.A.;+ Alfred M. Volo, Ph.D.;# Sudha Mehta, M.P.H.;# Steve Davis, Ph.D.;** Mary Smith, Ph. D.;†† and Nancy Callahan, Ph.D.†† *Substance Abuse and Mental Health Services Administration/Center for Mental Health Services;† Colorado Mental Health Performance Indicator Grant;‡ Vermont Multi-State Performance Indicator Grant;§ Indiana Performance Indicator Pilot Study;+ Rhode Island State Performance Indicator Pilot;# New York State Indicator Pilot Project; ** Oklahoma Mental Health Indicator Pilot Project;†† Illinois Performance Indicator Pilot Project;‡ Washington State Indicator Pilot Grant: Using Performance Indicators to Shape Quality Mental Health Services Introduction The purpose of this article is to examine selected performance indicators in the 16-State Indicator Pilot Grant Project (hereafter, 16-State Project), highlighting policy-and decisionmaking implications of performance indicator findings. The 16-State Project is funded by the Center for Mental Health Services (CMHS), Substance Abuse and Mental Health Services Administration, in a collaboration of the Survey and Analysis Branch and the State Planning and Systems Development Branch, Division of State and Community Systems Development. Sixteen States have been awarded grants for a 3-year period (fiscal years 1999– 2001) to pilot performance indicators that were selected in the CMHS-funded 5-State Feasibility Study (NASMHPD Research Institute, 1998) and the 1998 National Association of State Mental Health Program Directors' Framework of Mental Health Performance Indicators (NASMHPD President's Task Force, 1998). The primary goal of the project is to pilot and implement these indicators so that they can be collected and reported across States' information systems. The specific aims of the grant are (1) to collect specific performance indicators that can be reported comparably across States for national reporting and (2) to facilitate planning, policy formulation, and decisionmaking at the State level. Additionally, 6 of the 32 performance indicators are tied to the Federal Government Performance and Results Reporting Act (GPRA) as core measures for State reporting. The grant also supports the involvement and participation of key stakeholders, including consumers and family members, at all stages of the grant process. The 16 State grantees are Arizona, Colorado, Connecticut, Illinois, Indiana, Missouri, New York, Oklahoma, Rhode Island, South Carolina, Texas, Utah, Vermont, Virginia, Washington State, and the District of Columbia. Background The 16-State Project conforms to historical de-velopments and contributions of the Mental Health Statistics Improvement Program (MHSIP), which began in the 1970's. An MHSIP document, FN10 Data Standards for Mental Health Decision Support Systems (Leginski et al., 1989), together with subsequent grants to States, initially enabled the State mental health agencies to implement data standards in the areas of organization, patient/client, event, human resources, and financial data; facilitating standardization; and capacity building of State management information systems. The standards were broadly conceived to support decision-making in the mental health system. A second development in 1996, the MHSIP Consumer-Oriented Mental Health Report Card, identified performance measures in the domains of access, quality/appropriateness, outcome, and prevention to be used for assessing the effectiveness of mental health services. The MHSIP consumer survey was made part of the report card to include a consumer assessment of these same domains. Since 1996, 45 States have been awarded MHSIP State Reform Grants to further integrate, synthesize, analyze, and use information based upon the MHSIP Report Card. Grant-funded system changes serve as a foundation for State data integration and performance monitoring activities. In 1997, CMHS funded five States— South Carolina, Massachusetts, Illinois, Texas, and Colorado— to identify and pilot performance indicators that would not only be feasible and meaningful to collect, but that could be compared across States. Using the MHSIP Report Card and other sources, 28 performance indicators were selected and piloted using the domains of access, quality/appropriateness, outcome, and plan/management. The major finding of this project was that the States would be able to collect and report the selected indicators on a comparable basis if given sufficient time and resources. In 1998, the National Association of State Mental Health Program Directors (NASMHPD) incorporated the results of the 5-State Study into the development of the NASMHPD Framework of Mental Health Performance Indicators. It is this refined framework of 32 indicators that is being used by the 16-State Project (table 1). Status of Project The major focus of the 16-State Project is to pilot and implement performance indicators across States and to find the best ways to report these data. In addition to the performance indicators collected, additional data collection includes subgroup information on age (including children), race/ethnicity, gender, and diagnoses for hospital as well as community service settings. Some indicators focus on only adults or children, as appropriate. In the project, the States are addressing complex issues of performance indicator development and comparability as well as the need for collecting and reporting quality data. This article selects specific indicators from the 16-State Project to address their policy and decisionmaking implications. Preliminary results of the project and the potential of these findings for guiding policy and decisionmaking are reviewed. Each performance indicator will be introduced and discussed in terms of (1) background and data collection issues and (2) use of this information in policy-and decisionmaking. Several performance indicators have been selected for discussion in this article: indicators from the MHSIP consumer survey; penetration/utilization rates; assertive community treatment and supported employment services; use of atypical medications; number of days until readmission to a State mental hospital; days elapsed until contact by a provider for persons who have been discharged from a hospital; level of functioning and symptoms; and cost. Authors of these sections have taken responsibility for compiling grantees' data for the performance indicators about which they have written. Each author will introduce the indicator, provide background and significant information related to data collection issues, and present an analysis of the indicator's potential for use in policy-and decisionmaking. Performance indicators will be presented in the order of the 16-State Project domains: access, quality/appropriateness, outcome, and plan management— with the exception of the consumer survey indicators, which address three of these domains and which will be presented first. Performance Indicators in the MHSIP Consumer Survey Many of the indicators being reported in the 16-State Project are based on encounter data, clinician reports, or objective measures such as employment, living situation, or contact with the legal system. The inclusion of indicators based on consumer perspectives illustrates the growing importance of consumer involvement in mental health systems of care. NASMHPD and MHSIP both recognize the value that consumer-based indicators add to comprehensive performance-measurement systems. Studies have demonstrated that consumer satisfaction is related to treatment gains, employment, and other outcome variables (Holcomb, Parker, Leong, Thiele, & Higdon, 1998). Conducting outcome studies using consumer-reported data can produce unbiased population estimates at an affordable cost (Boothroyd, Skinner, Shern, & Steinwachs, 1998). Consumer ratings also are less likely to be provider-biased than are ratings by clinicians or administrators. Consumers rarely have a financial incentive in the results of an evaluation. Their concerns are more often based on need and less on regulatory requirements. Thus, consumer ratings, while subjective, may also provide a relatively unbiased evaluation of system performance. The MHSIP Consumer-Oriented Mental Health Report Card (1996) and other consumer-based measurement systems consider that consumer perspectives are central. The MHSIP Report Card is value based (addressing issues of consumer choice, empowerment, and involvement); emphasizes concerns related to serious mental illness (though it can also address concerns relevant to all people with mental health needs); includes outcomes; and is research based. Additionally, the domains, concerns, indicators, and measures of the MHSIP Report Card were specifically designed to assess consumer concerns with selected aspects of mental health treatment, not merely global satisfaction with mental health services. The MHSIP Consumer Survey was designed to be used in conjunction with other measures included in the Report Card. However, psychometric work conducted on State pilot data suggested that the survey could be used as a stand-alone instrument to measure three of the domains listed in the MHSIP Report Card: access, appropriateness, and outcomes (Wackwitz, 1998). Indicators from these three domains were developed and reported in the CMHS 5-State Feasibility Study. Initial reports of surveys conducted by participating States showed 77 percent of consumers reporting satisfaction with access to services, 74 percent agreeing that services were appropriate, and 65 percent reporting improved outcomes from services (Ganju & Lutterman, 1998). The indicators used for the 16-State Project were selected from the NASMHPD President's Task Force Indicators. Four of the indicators are based on consumer surveys. These four indicators fall across three of the four domains measured by the project:
Data collection for the 16-State Project has been under way for over a year. The MHSIP Consumer Survey is the most widely used instrument for the four indicators. Eleven of the 16 States are currently using the MHSIP Consumer Survey, with two more States planning to implement the survey in the near future. Survey data have been submitted for comparison and analyses by 10 States currently using the MHSIP Consumer Survey. Policy and Decision Applications Consumer survey data can be used to evaluate a provider's performance, compare providers, and assess quality improvement. Consumer surveys have been used to manage programs and to allocate program resources (McCarthy, Gelber, & Dugger, 1993), to evaluate and reimburse contractors, to track programs, and to gauge the overall functioning of programs. Consumer survey data can be used to assess care of underserved or minority populations and to identify barriers to treatment. The most obvious use of consumer survey data is to improve services by identifying problem areas and guiding quality-improvement efforts. Previous research suggests that 43 percent of clients who drop out of treatment do so because of negative experiences in their treatment (Schwartz, 1991). Clearly, addressing the needs of consumers is an important issue. Consum-er survey information can be used to measure whether available services are meeting consumers' needs, whether needed services are available, and how consumers are or are not accessing needed services. The MHSIP Consumer Survey gives specific feedback about areas in which providers can improve services. States are applying the MHSIP Consumer Survey results in a variety of ways. Some States use the indicators to compare providers' performance over the four domains. Others use the survey for quality improvement purposes by providing feedback to their providers based on each item in the survey. Many States are beginning to use the survey results for both provider comparisons and quality improvement. For instance, Colorado uses the survey to derive indicators for local providers as part of its performance incentive system, and Colorado also offers feedback to providers based on individual survey items for quality improvement purposes. Rhode Island publishes a report that incorporates both the indicators calculated from the survey and individual item results to report on performance statewide. Future uses of consumer survey data may include combining survey data with other indicators. For instance, survey data could be used as an outcome measure in cost-effectiveness research (Phillips & Rosenblatt, 1992). Survey data also could be combined with service utilization data to explore relationships between consumer perceptions of treatment and program usage, or treatment compliance. As best-practice treatment models are developed (e.g., supported employment, assertive community treatment [ACT], atypical antipsychotic medication use), consumer survey data could be used as a treatment outcome measure. Outcome indicators derived from administrative data or clinical scales could be used in conjunction with consumer-reported outcomes to provide multiple perspectives on the same treatment effects. Significantly, outcome measures based on consumer reporting could have a major influence on service delivery models and treatment policies, bringing them more in line with consumer preferences. The integration of consumer perspectives into performance measurement systems represents a major advancement in mental health evaluation. Consumer ratings of services can be useful at the provider level or the system level. The inclusion of consumer indicators in the 16-State Project supports the implementation of these measures within the public mental health system. State Hospital Utilization Utilization rates address fundamental issues of the degree to which people in different States make use of public mental health systems of care, and the degree to which these systems of care are responsive to people in different demographic and clinical groups. During its first year, the 16-State Project focused on State mental hospital utilization rates. In subsequent years, utilization rates for community mental health programs will be measured as well. Two quantitative measures of State mental hospital utilization were used. Penetration/utilization rates compare the number of people hospitalized during fiscal year 1998 to the total population of each State. Relative risk compares the hospitalization rates for two groups of people. There was substantial variation in overall State hospital utilization rates among the 16 participating States. State hospitalization rates in the 16 States ranged from less than 25 per 100,000 population in Arizona and Rhode Island to more than 500 per 100,000 population in Washington, DC. Men had higher relative risk of State hospitalization in every State. Men were more than twice as likely as women to spend time in a State mental hospital in eight of the States. The hospitalization rate for men was more than 50 percent greater than the rate for women in six other States. Nonwhite residents had a substantially higher relative risk of State mental hospitalization compared to white residents during 1998 in all 16 States. Nonwhite residents were more than twice as likely as white residents to spend time in a State mental hospital in 9 of the 16 States. The hospitalization rate for non-white residents was more than 50 percent greater than the rate for white residents in all of the other States. Policy and Decision Applications Integration of quantitative findings such as these into policy formulation and evaluation is one of the challenges of the emerging information age. This process should begin by viewing these results from a comparative perspective. Penetration/utilization rates and measures of elevated risk of hospitalization should be compared to the philosophy and values of local and statewide systems of care. They should also be compared to penetration/utilization rates for other systems of care. Perhaps most important, the relationship between these measures and to other indicators of service system performance should be considered for both local and statewide systems of care. Information on the relationship among various indicators of access to care, practice patterns, and treatment outcomes can be a valuable tool for policy development and evaluation. The interpretation of State mental hospital penetration/utilization rates is particularly challenging. Although State hospitals were once thought of as a progressive reform, they have more recently become devalued by many advocates and program administrators. "Appropriate" State hospitalization rates are a matter of debate and are widely seen to be a function of other attributes of systems of care. It is hoped that the information about State mental hospital penetration/utilization rates presented here (and in other places) will prove to be a valuable resource to those engaged in the policy debate that will continue to determine public policy in this area. Interpretation of differences in hospitalization rates for people in different demographic groups is less difficult when it is guided by the principle of equity. Equity dictates that, lacking demonstrated clinical justification, members of different demographic groups have equal access to inpatient care. Where this is not the case, public policy discussion should address this demonstrated inequity. Where there is a reason for this inequity, it should be made explicit and be subjected to public policy consideration. Where there is no reason for this inequity, public policy that will reduce or eliminate the inequity should be developed and implemented. The degree to which variation in State mental hospital penetration/utilization rates and relative risk of State hospitalization are related to differences in the prevalence of mental illness in the States or among different groups of people should be examined and considered. Similarly, differences in community resources, differences in public policy, or other differences should be considered. State mental hospitals, of course, do not represent the totality of inpatient psychiatric care that is available to people in need. In many States, the State mental health agency contracts directly for inpatient care through other mechanisms. In all States, inpatient psychiatric care is provided in a variety of other settings that include general hospitals, private psychiatric hospitals, and Department of Veterans Affairs hospitals, among others. To obtain a full profile of behavioral health care penetration/utilization rates, mechanisms for measuring the utilization of these service sectors will need to be developed as well. Assertive Community Treatment and Supported Employment During the past decade, consensus has formed around the effectiveness of a few selected models of community treatment for persons with severe and persistent mental illness (SPMI). Two such models are ACT (Stein & Test, 1980; Test, 1992), and supported employment (SE) (Wehman, 1986). Both models have a large empirical research base supporting their efficacy and both have been widely disseminated (for reviews of ACT see Latimer, 1999; Mueser, Bond, Drake, & Resnick, 1998; for SE see Bond, Drake, Mueser, & Becker, 1997b). Although formally classified as quality indicators, the ACT and SE indicators measure processes (i.e., what is being done for clients) rather than outcomes (i.e., how clients are doing). ACT and SE were selected as quality indicators because they were thought to represent consensus best practices for persons with SPMI (NASMHPD President's Task Force, 1998). The ACT and SE performance indicators are intended to index the percentage of persons with SPMI receiving one of the services. However, before one can determine who is receiving ACT/SE, it must first be verified that the service was delivered as intended. Departures from fidelity can critically affect outcomes in psychosocial programs; for example, receiving "ACT-like" services may not produce the same benefits as receiving ACT that has been implemented faithfully. For example, in a sample of 18 sites implementing ACT, McGrew, Bond, Dietzen, and Salyers (1994) reported a correlation of 0. 60 between overall measured ACT fidelity and percent reduction in hospital use in the 2 years after admission compared to the 2 years prior to admission to ACT. Thus, the labeled service must closely correspond to the model definition of ACT or SE to truly represent best practice. Without clear standards, measuring the per-centage of clients receiving ACT or SE is problematic. States often use different definitions/ standards for both ACT and SE, making comparisons difficult. A multistate workgroup designed a survey to determine the extent of this problem across the 16 States. The survey addressed two questions: (1) the extent of ACT and SE dissemination and (2) problems in defining and assessing implementation of ACT and SE. Eleven States returned data for SE; 10 States returned data for ACT. The survey results for ACT revealed that 9 of 10 States reported implementing ACT, 6 reported an operational definition of ACT, 6 collected information on who received ACT, but only 2 States reported the information. In addition, just two States had detailed plans to measure ACT implementation (e.g., auditing records, monitoring programs, providing a detailed treatment manual). Finally, some conceptual ambiguity existed in defining ACT; for example, ACT was confused with case management. The survey results for SE showed that 10 of 11 States were implementing SE, 8 had operational definitions of SE, but only 4 collected information on who received SE, and only 2 of those reported the information. Similar to ACT, only two States reported adequate methodology to measure implementation. Moreover, considerable conceptual ambiguity was evident in definitions of SE across the 16 States. Six of the eight States reporting definitions deviated from published definitions of SE (e.g., Bond, Becker, Drake, & Vogler, 1997a), including confusing SE with any vocational programming and defining transitional employment or enclave work as an instance of SE. Both the survey results and the fidelity literature demonstrate the need to measure ACT/SE implementation prior to concluding that services are actually being received. Although fidelity instruments exist for ACT (e.g., McGrew et al., 1994; Teague, Bond, & Drake, 1998), and SE (e.g., Bond et al., 1997a), they tend to require a considerable investment of resources (onsite visits, staff interviews) that was deemed overly taxing for systems attempting to implement the entire performance indicators package. An alternate approach was to develop simplified checklists. Accordingly, preliminary checklists have been developed for both ACT and SE, based on the existing fidelity instruments and on literature specifying the critical ingredients (e. g., Allness & Knoedler, 1998; McGrew & Bond, 1995). Both checklists are one page, contain 14 items, present a simple list of the critical ingredients, and should take less than 15 minutes to complete. As currently conceived, clinical directors would check an item only if it is fully met. The instrument devel-opment plan is (1) to revise the instrument based on feedback from ACT experts/ providers, (2) to pilot the preliminary checklists in several of the 16 States, concurrently gathering criterion fidelity ratings using the DACTS for ACT (Teague et al., 1998) and the IPS fidelity scale for SE (Bond et al., 1997a), and (3) to modify the checklists based on State-, site-and user-level feedback and on the results of the concurrent validity analyses, creating final checklists for use in the 16-State Project. Policy and Decision Applications Possible policy uses of the ACT/ SE indicators include both service and evaluation/ research applications (e. g., correlating ACT/ SE adherence measure data with outcome data). Only policy implications for services will be discussed. Two points seem particularly relevant: (1) the policy use of process indicators is likely to be contingent on corresponding outcome information and (2) process indicators may directly influence the choice of which specific service models to implement. Performance information is complex and is rarely used in isolation. The performance indicators give policymakers access to information about both outcomes, such as percent employed, and best practices, such as percent receiving SE. Although the outcome information alone, such as percent employed, could be used to guide decisions to change or continue current vocational services, it provides no information about how to change services (this is the role of best practice information). However, the process information alone, such as percent receiving SE, likely would not be sufficient to guide policy decisions. For example, in concert with superior performance in achieving employment, low SE penetration rates may have little impact on policy. However, low SE use combined with poor performance in achieving employment would likely lead policymakers to both reexamine current vocational services and strongly consider use of the specific best prac-tice, that is, SE. Thus, the policy use of best practice information may be conditioned by information from the corresponding outcome domain. An exception, however, may be when best practices are strongly supported by important stakeholder groups (e.g., use of atypical antipsychotic medications). In this case, policy may be affected independent of outcome data. An important facet of best practice indicators, as illustrated above, is that they tend to shape policy directly. That is, best practice indicators do not just provide information about the service domains that need targeting (vocational services), but possess additional demand characteristics that tend to promote the provision of specific services for the targeted domain (SE). In essence, best practice indicators deliberately attempt to shape local policymaking authority. Indeed, the previously mentioned controversy over best practice indicators revolves, in part, around a concern that policymaking autonomy is partially coopted by the choice of the specific indicator. This latter concern sets the stage for the possibility of conflict between policymakers at the national, State, and local levels with the use of best practice indicators. New-Generation Antipsychotic Medications New-generation antipsychotic medications, also known as atypicals, were first introduced to the United States in 1990 with the approval of clozapine. Similar to conventional antipsychotic medications, atypical antipsychotic medications are indicated for the treatment of schizophrenia and psychosis and are effective in treating both the negative and positive symptoms of schizophrenia (Brown, Markowitz, Moore, & Parker, 1999). Because of their efficacy and comparatively more tolerable side effects, the administration of atypical antipsychotic medication has been recommended as a first-line treatment for schizophrenia (Collaborative Working Group on Clinical Trial Evaluations, 1998) and is considered a "best practice." For these reasons, the "New-Generation (Atypical) Medication Use" performance measure was identified as a key quality/appropriateness indicator in evaluating mental health organization performance in both the 5-State Feasibility Study and the 16-State Project. In the 5-State Feasibility Study, the "New Generation (Atypical) Medication Use" indicator measured the percentage of persons with a schizophrenia diagnosis (a DSM-IV diagnostic code of 295) who received atypical antipsychotic medications in a given year in both hospital and community settings. Four medications— clozapine, olanzapine, risperidone, and quetiapine— were identified as atypical antipsychotic medications. The 5-State Feasibility Study found that the overall median atypical medication use in the hospital setting was 40.6 percent and that rates differed across gender and race groups. The median rate for females was slightly higher than the rate for males (32.6 percent vs. 29.5 percent). In terms of race/ethnicity, whites had the highest median rate (38.7 percent) and blacks had the lowest (22.6 percent). Only one State was able to provide community data for this measure; interstate comparisons for this setting thus were not possible. The definition of this measure and settings of interest remain unchanged in the 16-State Project. However, at least seven States anticipate that they will be able to provide community data, enabling interstate comparisons for that setting. Additionally, in response to the more widespread use of atypical antipsychotic medications for the treatment of other mental illnesses, the 16-State Project has added a second measure to this indicator. This second measure examines the percentage of clients, regardless of diagnosis, who receive an atypical antipsychotic medication. Policy and Decision Applications Given that administration of atypical antipsychotic medications is viewed as a "best practice," the underlying assumption is that the higher the administration rate, the better the care clients are receiving. However, where should one draw the line? Can we conclude that a State that has a 75 percent atypical medication use rate necessarily provides superior care to a State that has a 65 percent rate? A better way to determine whether a State provides an optimal level of access to atypical antipsychotic medication is to examine not only baseline rates, but also how those rates compare with other quality and outcome measures of care, and whether the rates are consistent across demographic groups and care settings. An initial use for the "New-Generation Antipsychotic (Atypical) Medication Use" indicator is to establish a national-level baseline for atypical antipsychotic medication use rates by calculating average rates, for both hospital and community settings, across States participating in this project. A "first-pass" examination of this information would allow States to assess how they compare nationally in providing access to atypical antipsychotic medication for their clients, and determine whether policy or funding changes may be warranted to adjust their rates. It is worth noting that atypical medication use rates could have policy implications at units as small as the agency level or as large as the State, depending on how States choose to capture data for this measure and the information collected. Once States have established their atypical medication use rates, those rates can be compared to other 16-State indicators, such as rehospitalization rates or symptom severity measures of clients. In patients diagnosed with schizophrenia, Olfson and colleagues (1999) found that the use of conventional rather than atypical antipsychotic medications was one of several factors associated with early (within 3 months of discharge) hospital readmission. Another study found that patients with schizophrenia treated with atypical rather than conventional antipsychotic medication (e.g., olanzapine vs. haloperidol) showed significantly greater symptom reduction, as measured by the Brief Psychiatric Rating Scale (Tollefson et al., 1997). The data from the 16-State Project can be analyzed at the individual client record level to determine whether relationships exist between atypical medication use rates and the aforementioned indicators, and whether different profiles of care emerge across entities. For example, a State agency with the highest atypical medication use rate may show a lower readmission rate but only moderate changes in symptom severity over time, as compared to an agency with a slightly lower atypical medication use rate. Thus, the ideal rate for any given State will be a function of striking a delicate balance between access to atypicals and other system needs. A starting point for finding that balance could be to look to States/agencies with above-average profiles of care to suggest what preferable atypical medication rates might be. Findings at any level (State, regional, agency, program) of a lower-than-average atypical medication rate or less-than-average profile of care could provide mental health administrators with support to lobby at the appropriate level (e.g., State or Federal) for increased funding for atypical antipsychotic medications. Such findings might also provide administrators with support to challenge some managed care policies regarding administration of atypical antipsychotic medication (e.g., requiring that a patient fail on a conventional antipsychotic prior to receiving atypicals). Atypical medication use rates also can be analyzed to determine whether differential practices exist in prescribing atypical antipsychotic medications across demographic groups (e.g., gender, race/ethnicity, and age). Research has not demonstrated any significant gender or racial differences with respect to atypical antipsychotic medications. Thus, no compelling evidence in the literature suggests that atypicals should be prescribed differentially across gender or race/ethnicity groups. Older adults often tend to be even more susceptible to the side ef-fects (extrapyramidal syndrome and tardive dyskinesia) associated with conventional antipsychotic medications, so the use of atypical antipsychotic medication in this age group may be indicated (Jeste et al., 1999). Thus, differing medication use rates across age groups also is not expected. Examining the medication use rates for clients on atypical antipsychotic medications who receive care in hospital vs. community settings is also of interest. There is some concern that patients may receive atypical medications while hospitalized, but are then "stepped down" to conventional antipsychotic medications once in the community. Patients receiving care in both hospital and community settings could be tracked in the 16-State Project to determine whether those concerns are warranted. At various organizational levels (e.g., State, regional, agency, or program), if differences in atypical medication use rates are found among demographic groups or treatment settings, further examination could pinpoint why these differences exist and could result in the implementation of policies to ensure more equitable and consistent prescribing practices. Such policies might further evolve into adopting standard pharmacological treatment guidelines (e.g., guidelines based upon client diagnosis, symptomology, side effect sensitivity, and client preference) and/or working toward increased funding. Readmissions to a State Psychiatric Hospital Within 30 Days of Discharge Hospitalization of psychiatric patients is expensive and is typically indicative of an acute episode of illness. An important goal of mental health treatment is to minimize such episodes and provide services that will allow inpatients to return to the community as soon as possible. A person may be rehospitalized following an episode of inpatient care for many possible reasons, including the following: (1) hospitals may release patients prematurely to reduce cost, (2) persons may not receive adequate followup care, (3) inpatient treatment may be incomplete or ineffective, or (4) continuity of care does not exist between inpatient providers and community providers. This indicator looks at the percentage of consumers discharged that are rehospitalized within 30 days of discharge. If States can report performance on this indicator using a common definition and reporting structure, relative performance among States can be gauged. Those States with higher than expected rates of readmission within 30 days can then further analyze the factors underlying this finding to improve and strengthen the effectiveness of their inpatient and outpatient mental health service systems. The 5-State Feasibility Study measured readmissions to a State psychiatric hospital within 30 days of discharge from any State psychiatric hospital. During the 5-State study, the ability of States to identify readmissions that had an earlier admission to a non-State hospital (e.g., a local or private hospital) within the prior 30 days was examined. The five participating States were largely unable to expand the database by linking client records to other (non-State psychiatric) hospital admissions. For the 16-State Project, State grantees decided to analyze this indicator at 6 months following discharge from an inpatient episode in addition to the 30-day interval. The 6-month interval was felt to reflect the role that community mental health service and support systems play in preventing rehospitalization. The possibility of adding additional periods between discharge and readmission has been left open for future analysis by the 16-State Project group. In addition, an attempt was made to collect data at three different levels: (1) number of readmissions to the same hospital, (2) number of readmissions to any State mental hospital, and (3) number of readmissions to any State mental hospital or community hospital. Policy and Decision Applications Being able to report readmission data across States using common definitions and diagnostic and demographic categories will enable risk-adjusted comparisons between States to be made. Comparing performance for specific demographic and diagnostic groups at 30-and 180-day intervals will inform managers of problem areas and support enhanced monitoring of system initiatives aimed at reducing dependence upon inpatient care or enhancing access to less-restrictive alternatives. This indicator, in conjunction with a second indicator, which looks at the percentage of discharged people who connect with outpatient services in the community within 7 days, is a powerful measure of the success of inpatient treatment and of the coordi-nation postdischarge between service providers. An example from New York State relates to the newly enacted (November 1999) Assisted Outpatient Treatment (AOT) Law. Under this law, eligible individuals are evaluated for legally mandated out-patient treatment. Although only a few of the individuals screened under this law proceed to a court order for mandated treatment, efforts are made to provide case management and other needed services on a voluntary basis to everyone. The indicators that measure inpatient readmission and contact with community mental health services within 7 days are being used as part of a system to monitor the effectiveness of the policies and programs implemented under this law. By having good data available that can be sensibly compared with the experience of other States, individuals diverted or treated under the AOT initiatives can be compared to other recipients of similar services. In summary, having indicator data with national benchmarks will enhance the ability of States to review and evaluate new policy initiatives and to manage programs based on commonly used and valid data. Consumers Contacted by Community Providers Within 7 Days of Hospital Discharge Among the external influences which can have a positive or negative effect on recovery or healing, Ruth Ralph (1999) includes the "policies, procedures, and actions of the mental health system." She also notes these system influences are generally outside the control of the consumer. For this reason, system administrators and service providers have a responsibility to establish and follow policies and procedures that will promote healing and recovery. One means of meeting these goals is to promote continuity of care as service recipients move from one level of care to another within the service system. Continuity of care is especially important for persons who are returning to their communities after receiving inpatient care. Achievement of such continuity can be measured by determining the percentage of persons discharged from inpatient care who are contacted by community providers within 7 days of hospital discharge. In June 1993, a task force of the national MHSIP Ad Hoc Advisory Group submitted its report on "Performance Indicators for Mental Health Services." The task force had been charged "to enhance the MHSIP recommended data standards with the design of a system of performance indicators that can be derived from the content of MHSIP." One of the proposed indicators in that report was "How prompt is the linkage between discharge from inpatient and enrollment in outpatient services?" (MHSIP Task Force, 1993). Although a different formula for measuring the indicator was used, the same issue was addressed by an appropriateness-of-care indicator in the MHSIP Consumer-Oriented Mental Health Report Card published in April 1996. The MHSIP Report Card, another product of a MHSIP task force, included a measure of the percentage of people discharged from inpatient services who receive ambulatory services within 7 days (MHSIP Task Force, 1996). When the 5-State Feasibility Study was implemented, "percent contacted within 7 days of hospital discharge" was again selected as an important indicator of appropriateness and quality of care (NASMHPD Research Institute, 1998). Likewise, when the NASMHPD President's Task Force on Performance Measures (NASMHPD President's Task Force, 1998) established its standardized framework based on the earlier work, one of the quality-of-care indicators was "Consumers are contacted by community providers within 7 days of hospital discharge." It is this last formulation of the indicator that is currently being used in the 16-State Project. Policy and Decision Applications All of the indicators described above have focused on the concern that continuity of care is critical for consumers to avoid recurrence of symptoms and ensure that the process of recovery is not interrupted. The topic is important to consumers, payers, plans, and providers because it reflects the extent to which services provided at different facilities are linked: that is, whether services are provided within a coordinated system of care or among a disconnected set of organizations. The expectation is for clients who receive coordinated care to have better outcomes than those who do not, but linkage data will need to be combined with service utilization and quality-of-life measures to verify that this occurs. The MHSIP Task Force Report on Performance Measures emphasized the multiple perspectives and differential needs for indicators among various mental health system stakeholders. Using the "contact within 7 days of inpatient discharge" indicator, consumers of mental health services, particularly those with serious mental illness who may have episodes of inpatient care, can compare plans to help them decide which is likely to best help them in their recovery from an acute episode. Payers can use this indicator to help determine which plans or provider groups are most effective in coordinating services. Plans and providers, in turn, can identify linkages that can be emulated or improved to provide more effective, less costly care. Improvement in Functioning and Reduction in Symptoms The measurement of improvement in functioning, which has been defined as consumers' increased ability to respond to problems, crises, and everyday situations as a result of mental health treatment, has long been accepted as one of the key indicators of treatment outcome. Similarly, the measurement of change in consumers' level of psychological distress as evidenced by a reduction in symptoms has been widely endorsed. Despite these facts, little agreement exists among States regarding the use of instruments to assess these outcomes, nor does consensus exist as to what amount of change is meaningful. This lack of agreement has resulted in the use of a wide variety of instruments to measure (1) functioning and symptoms and (2) different operational definitions of change. An early survey of the States participating in the 16-State Project revealed that seven different functional assessment instruments were being used with adult populations, and seven instruments were being used for the child and adolescent population. Similar results were found with regard to the measurement of symptoms, although fewer States reported measuring level of symptom distress. Essentially, State mental health agencies have selected instruments that best meet the needs of mental health stakeholders within their State. While this is a reasonable strategy, it presents a problem in terms of a stated goal of the 16-State Project, which is to work towardcomparability of data across States. During the 5-State Feasibility Study, initial efforts to operationally define improvement in terms of positive changes in functioning and symptoms led to the use of a definition that was somewhat arbitrary. However, given the different instruments used by the two States that were able to report, the short timeframe for completing the study, and the recognized importance of including these indicators, results were judged to be reasonable. There was a realization that issues related to comparability of ratings generated from different instruments would have to be addressed in the next phase of the study. The indicator, Percent Improvement in Functioning, was defined as the number of persons receiving community services with a minimum of a 10 percent change in functioning scores divided by the number of persons served in the community during the fiscal year. Maintenance was defined as less than a 10 percent change. The same definition was used for the Symptom indicator, with the focus on a decrease in symptoms. Differences in outcome patterns for closed cases were found for the two States reporting on the indicators, but the issue of whether this change was clinically significant and meaningful from consumer and provider perspectives was an is-sue, as was the comparability of the data. Different patterns were found for the distribution of open cases, adding another degree of complexity to the interpretation of the data. Several additional issues were raised, including the need to risk-adjust the data based on relevant variables, a need for consensus regarding data collection time points, and the reliability with which instruments are used. The complications inherent in using such data for comparisons across States and for demonstrating the impact of public mental health services in a routine summative way are apparent given these issues. However, if the major issue of comparability can be resolved— and it is believed that it can— the extent to which improvement in functioning and symptoms can contribute to the ethos of public accountability will be enhanced. Policy and Decision Applications If outcomes are the bottom line in terms of accountability for the public mental health system, then the implications for decisionmaking and policy development would seem to be clear. Programs and services that result in better outcomes represent those that should be continued and funded. Unfortunately, given the state of the art of the uniform assessment of outcomes and the methodological issues described above, the use of data for policy development such as this must be tempered. However, such data can be used, and are being used, for purposes of accountability and performance contracting to support decisionmaking for quality improvement. Illinois has implemented an ACT model built on a foundation of evidenced-based research. Admission to ACT programs requires preauthorization by Office of Mental Health regional staff. Operating programs are routinely monitored to determine the extent to which the services provided maintain fidelity to the program model. There is an expectation that programs rated high in terms of fidelity should result in better outcomes. Using functioning assessments administered at various points in time, one could determine the outcomes associated with each program. If the outcome pattern for one program was found to be markedly different from patterns produced by other programs, it would be possible to compare the programs to determine what practices account for these differences. This information could then be used to improve program practices by moving the outcomes of the outlier program more into line with other similar types of programs. Although the Illinois Office of Mental Health has not taken this final step of comparing programs in terms of improvement in functioning and symptoms as described in this illustration, the potential to utilize the data for quality improvement purposes such as this is apparent. The Texas Department of Mental Health and Mental Retardation (DMHMR) utilizes performance measures as part of a State agency planning and budgeting system in which appropriations are allocated to objectives and strategies as specified in a strategic plan. As part of this performance measurement system, both outcome and output measures have defined targets tied to funding. Improvement in functioning has been identified by the Texas DMHMR as a key performance measure that provides a means of public accountability. The underlying expectation is that mental health services will lead to improvement in consumers' functioning. Quarterly reports, using functional impairment data submitted by mental health providers, are prepared and submitted to the State legislature for review. The reports reflect how the DMHMR is doing in terms of performance contracting with providers to achieve defined targets for performance. These reports, which contain measures for each local service area, are also shared with local mental health agencies and local mental health planning advisory committees. Thus, the Texas DMHMR is accountable in a very public way to a variety of mental health stake-holders for demonstrating and documenting the outcomes of services purchased on behalf of the public it serves. Texas has also used historical functional impair-ment data, specifically for children, as a basis for performance contracting with service providers. There is an expectation that 85 percent of the children and adolescents seen for services will maintain or improve their level of functioning during the contractual time period. Agencies not meeting this standard are at risk of losing $1,000 in contractual dollars. Although this may seem a small amount, the Texas DMHMR has implemented more than 20 indicators with similar standards attached to them. Thus, providers who fall below the standard on a large number of indicators will experience a serious fiscal impact. The use of data for performance contracting such as this requires the use of audit procedures to ensure that the reported data accurately represent consumers' status. The Texas DMHMR periodically audits medical records to ensure accurate data reporting, and the validity of the data is audited through quality assurance reviews. The Indiana Division of Mental Health utilizes functional assessment ratings as a basis for risk adjustment and to provide information to consumers in the form of a mental health report card that can be used as a basis for decisionmaking. Indiana utilizes a managed care model for the provision of mental health treatment— providers are certified as managed care providers. Functional assessment ratings are required for each consumer at the point of enrollment in treatment. This initial rating is one of several key variables used to develop risk-adjusted groups. The premise underlying this strategy is that severity of functional impairment, diagnosis, and other key variables are related to consumers' level of need. Service packages that differ in type and intensity of services have been designed specifically to address the level of need associated with the risk-adjusted groups. The arrays of services that constitute the packages determine the associated reimbursement rates. The Division also collects functional assessment ratings at the end of the treatment (or at the end of the contract period), using this information to calculate change scores for each risk-adjusted group. Results are normalized using the average change score and the standard deviation across all agencies submitting the data. The data are then partitioned into thirds representing greatest improvement in functioning, moderate improvement, and least amount of improvement. This information is then fed back to providers in the form of a mental health report card that displays the pattern of outcomes for each risk-adjusted group by agency. Statewide values are also displayed for comparison purposes. The ultimate goal of this public report card is to provide information to consumers for use in decisionmaking regarding selection of providers for treatment given their diagnosis and severity of functional impairment. In summary, despite issues associated with comparability of ratings across instruments, the States are using improvement in functioning and improvement in symptoms as a basis for accountability, performance contracting, decisionmaking, and, potentially, quality improvement purposes. Cost Indicator The MHSIP paradigm identifies cost as one of the core components of a mental health information system: who receives what from whom at what cost with what outcome. When a manager of a mental health system considers the management of resources, it is likely that money is the first resource that comes to mind. However, financial data typically have not been recorded and reported in a manner that facilitates comparing information across organizations or for aggregating financial information on organizations to describe systems of care (Leginski et al., 1989). Currently, the management and analysis of cost data continue to challenge mental health administrators. The provision of cost-efficient services has been an ongoing goal of public mental health service delivery, yet the ability to track and evaluate costs meets many barriers (Broskowski & Chalk, 1998; Hargreaves, Shumway, Hu, & Cuffel, 1998; Larson et al., 1998; Wurster, 1997). Mental health programs are funded through multiple funding sources including Medicaid, Medicare, State and local funds, and private insurance. In many States, service utilization and cost data are not available from some of these funding sources, and dollars are tracked through cost reports that are completed months or years after services are delivered. Despite the complexity of collecting and analyzing cost data, this information is critical for understanding service delivery systems. As systems become increasingly competitive, timely knowledge of revenues and expenditures is critical. Fiscal data also provide information to managers on the cost of delivering a unit of service, the cost per client, and a system's ability to generate revenue. One of the performance indicators for the 16-State Project is cost. It is being combined with several of the other indicators to provide a better understanding of access, utilization, and cost of services. When gauged against system values and goals and tracked over time, this information supports informed decisionmaking at all levels: Federal, State, local agencies, programs, and treatment teams. By examining these components together, managers can begin to understand trends in system-level changes across time. For example, combining information on access (e.g., penetration rates and number of clients per 1,000 population or per eligible member), service utilization (e.g., number of units per client), and cost (e.g., cost per unit of service) provides managers with a better understanding of service performance and helps them to manage risk. When information is examined across time, the resulting trends in system changes can also help evaluate the impact of policy changes. Policy and Decision Application A model for using cost data for decisionmaking is described below (Figure 1) and presents data from one State. This type of data is useful as a decision tool to examine access, service utilization, and dollars across programs and/or across years. This model shows two fiscal years, 1993–94 and 1997–98, to demonstrate policy trends across time. This model is useful in understanding the impact of system-level changes upon service patterns and costs. As a first step in understanding cost, administrators may examine total system dollars and service dollars across broad cost and service areas (e.g., inpatient, crisis/emergency, outpatient, day services). In this State's analysis, Residential Services are included in Day Services. Administrative costs associated with overall program operation are not shown, but could be added to the model. The total dollars and the proportion of dollars spent in each service area for each year show global trends in service delivery practices and reflect the State's values and policy changes. During this 5-year period, this State consolidated inpatient services and allowed counties to use savings to develop community-based services. As a result, counties negotiated lower bed day rates and expanded the delivery of outpatient services. Using this model, an administrator interested in understanding cost issues would start with an examination of the total system expenditures and each service area's expenditures, both in absolute number and as a share of the total system expenditures. Trends in these expenditures reveal the growth or constriction of each service sector and/or the total system. For example, while inpatient service dollars declined, paid claims (dollars) for outpatient services increased. A comparison of such trends with the strategic plan will inform the ad-ministrator regarding the actual implementation of their direction of change. Next, the total number of unique clients served statewide, as well as the unduplicated client count for each service area, informs the administrator about system access and capacity. When the number of clients in a service area is expressed as a proportion of all clients, the data show the relative salience of the services in the system. For example, while 11 percent of all clients used inpatient services in 1997–98, more than 95 percent used outpatient services. The number of units of service for each area can serve as an indicator of service access. Measures of units of service, of course, differ depending upon the type of service. Normally, the units for inpatient and day-treatment services reflect a day of services. Crisis and outpatient services reflect client contacts. Consistency in counting units is important for comparing similar organizations within a specific program type. To compare (or aggregate data across) different program types, it may be necessary to translate units to a common denominator, such as hours or quarter-hours. As with expenditures and client counts, units of service reflect trends within each program and as a proportional share relative to the total system. Certain obvious red flags should be paid immediate attention, such as when expenditures increase but units of service decrease; when the rate of growth in clients served is matched by the rate of growth in units of service; or when units of service vary across similar provider organizations. These red flags are easily recognized by using ratios: cost per client (expenditures divided by clients served), cost per unit of services (expenditures divided by units of service), and intensity of service (units of service per client). An increase in cost per client accompanied by an increase in cost per unit of service suggests that resources are not being used efficiently or have become more expensive. These administrative red flags can be constructed for each program type and its subcomponents as well as for different client subpopulations as defined by age, ethnicity, funding source, clinical status, and so on. These "drill-down" statistics are extremely useful in identifying not only potential problem spots, but also instances of "best practice." When counties in this example were allowed to manage their own inpatient services, bed day rates were negotiated lower. The savings were used to expand outpatient services. As a result, the bed day rate dropped from $624 to $456 per day over a 5-year period. The result is a large decrease in the average inpatient dollars per client from $9,820 to $6,100 and a simultaneous increase in the availability of outpatient services. Additional steps would measure consumer perception of access and quality of services through consumer surveys. Outcome measurement instruments provide information on the effectiveness of services on clients' recovery and ability to function independently. Managers can begin using this simple cost-effectiveness model to understand broad issues of access, utilization, and cost by service area. More sophisticated levels of analysis can then be developed and utilized to understand service delivery to meet a variety of outcomes to ensure access, quality, and cost-effectiveness. Summary The sections in this chapter highlight significant facets that are important in addressing performance indicators and mental health policy-and decisionmaking. Judy Hall emphasizes the importance of including consumer perspectives in a comprehensive management information system; John Pandiani states that a need exists to take into account the philosophy, values, and contexts of hospital or community settings when penetration/utilization rates are being considered; John McGrew demonstrates new methods of collecting comparable best practices information in States; Amy Elliott describes how comparison of atypical antipsychotic medication baselines for demographic and hospital and community settings can inform and assist in policy-and decisionmaking; Al Volo and Sudha Mehta discuss how data on readmissions to psychiatric hospitals can inform and influence State policy; Steve Davis illustrates how data on 7-day contact following hospital discharge could be utilized for consumer selection of managed care plans; Mary Smith illustrates how existing State systems are utilizing data on level of functioning and symptoms to assess and manage programs; and Nancy Callahan demonstrates an approach that can inform States on how to identify cost and service utilization change across time for program and policy decisionmaking. Some trends that appear in these commentaries include the need to assess subgroup findings, the need to address combinations of indicators for a more informed picture, and the ability to target findings and planning at the local as well as State and national levels. Some closing comments are important to note in regard to the 16-State Project effort. First, it must be kept in mind that the work of the 16-State Project is part of an historical effort that has been developing over a period of years. The work to develop a conceptual framework, identify indicators and measurements, and pilot and then apply them in divergent and changing State systems is a difficult and arduous task. Both time and resources will have been required to finally arrive at findings on selected performance indicators. Once actual findings are available, information must be considered in terms of the context of existing policies, programs, and populations which may explain differences. One must be able to dig deeper into what an indicator "implies," as other factors may be influencing a finding. There must also be risk adjustment to understand information by subgroup, such as age, race/ethnicity, gender, and diagnostic groupings, as appropriate to each indicator. Once there is knowledge of what indicator findings mean, policy-and decisionmaking can be addressed. The point is that information must be fully understood before it is used for policy-and decisionmaking application. Also, within this process, the development, piloting, and implementing of performance indicators must involve key stakeholders at every stage. These key stakeholders must include providers, consumers, and family members. The process must involve, inform, and train these persons, and they must have input into the content and direction of the effort. This involvement and representation are grant requirements in the 16-State Project. Third, persons involved in these efforts must not lose sight of the overarching goal within these efforts: to improve information so that we can better support the recovery of persons with mental illness. Refernces Allness, D. J., & Knoedler, W. H. (1998). The PACT model of community-based treatment for persons with severe and persistent mental illness: A manual for PACT startup. Waldorf, MD: National Alliance for the Mentally Ill. Bond, G. R., Becker, D. R., Drake, R. E., & Vogler, K. M. (1997a). A fidelity scale for the Individual Placement and Support model of supported employment. Rehabilitation Counseling Bulletin, 40, 265–284. Bond, G. R., Drake, R. E., Mueser, K. T., & Becker, D. R. (1997b). An update on supported employment for people with severe mental illness. Psychiatric Services, 48, 335–346. Boothroyd, R. A., Skinner, E. A., Shern, D. L., & Stein-wachs, D. M. (1998). Feasibility of consumer-based outcome monitoring: A report from the National Outcomes Roundtable. In R. W. Manderscheid & M. J. Henderson (Eds.), Mental health, United States, 1998. DHHS Pub-lication No. (SMA) 99–3285. Washington, DC: Government Printing Office, U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services. Broskowski, A., & Chalk, M. (1998). Estimating and managing risks for the utilization and cost of mental health and substance abuse services in a managed care environment. Volume Four: Managed Care Technical Assistance Series. U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration. Brown, C. S., Markowitz, J. S., Moore, T. R., & Parker, N. G. (1999). Atypical antipsychotics: Part II. Adverse effects, drug interactions, and costs. The Annals of Pharmacotherapy, 33, 210–217. Campbell, J. (1999). Consumerism, outcomes, and satis-faction. In K. Coughlin, (Ed.), Behavioral outcomes and guidelines sourcebook. New York: Faulkner & Gray, 1999. Collaborative Working Group on Clinical Trial Evaluations. (1998). Adverse effects of the atypical antipsychotics. Journal of Clinical Psychiatry, 59(12), 17–22. Ganju, V., & Lutterman, T. (1998). The Five-State Feasibility Study: Implementing performance measures across state mental health systems. In R. W. Mander-scheid & M. J. Henderson (Eds.), Mental health, United States, 1998. DHHS Publication No. (SMA) 99–3285. Washington, DC: Government Printing Office, U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services. Hargreaves, W. A., Shumway, M., Hu, T., & Cuffel, B. (1998). Cost-outcome methods for mental health. San Diego, CA: Academic Press. Holcomb, W. R., Parker, J. C., Leong, G. B., Thiele, J., & Higdon, J. (1998). Customer satisfaction and self-reported treatment outcomes among psychiatric inpatients. Psychiatric Services, 49(7), 929–934. Jeste, D. V., Lacro, J. P., Bailey, A., Rockwell, E., Harris, M. J., & Caligiuri, M. P. (1999). Lower incidence of tardive dyskinesia with risperidone compared with haloperidol in older patients. Journal of the American Geriatrics Society, 47, 716–719. Larson, M. J., Farrelly, M. C., Hodgkin, D., Miller, K., Lubalin, J. S., Whitt, E., McQuay, L., Simpson, J., Pepitone, A., Keme, A., & Manderscheid, R. W. (1998). Payments and use of services for mental health, alcohol, and other drug abuse disorders: Estimates from Medicare, Medicaid, and private health plans. In R. W. Manderscheid & M. J. Henderson, Mental health, United States, 1998. U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services. Latimer, E. (1999). Economic impacts of assertive commu-nity treatment: A review of the literature. Canadian Journal of Psychiatry, 44, 443–454. Leginski, W. A., Croze, C., Driggers, J., Kamis-Gould, E., Namerow, M. J., Patton, R. E., Wilson, N. Z., & Wurster, C. R. (1989). Data standards for mental health decision support systems. Series FN No. 10. Rockville, MD: U.S. Department of Health and Human Services. Mayer, J. E., & Rosenblatt, A. (1994). Clash in perspective between mental patients and staff. American Journal of Orthopsychiatry, 44, 432–441. McCarthy, P. R., Gelber, S., & Dugger, E. (1993). Outcome measurement to outcome management: The critical step. Administration and Policy in Mental Health, 21, 59–68. McGrew, J., & Bond, G. (1995). Critical ingredients of assertive community treatment: judgments of the experts. Journal of Mental Health Administration, 22(2): 113–125. McGrew, J., Bond, G., Dietzen, L., & Salyers, M. (1994). Measuring the fidelity of implementation of a mental health program model. Journal of Consulting and Clinical Psychology, 62, 670–678. Mental Health Statistics Improvement Program (MHSIP) Task Force on a Consumer-Oriented Mental Health Report Card. (1996, April). The MHSIP consumer-oriented mental health report card. Mental Health Statistics Improvement Program Ad Hoc Advisory Group. Mental Health Statistics Improvement Program (MHSIP) Task Force on the Design of Performance Indicators Derived from the MHSIP Content. (1993, June 5). Performance indicators for mental health services: Values, accountability, evaluation, and decision support. Mental Health Statistics Improvement Program Ad Hoc Advisory Group. Mueser, K. T., Bond, G. R., Drake, R. E., & Resnick, S. G. (1998). Models of community care for severe mental illness: A review of research on case management. Schizophrenia Bulletin, 24, 37–74. National Association of State Mental Health Program Directors (NASMHPD) President's Task Force on Performance Measures Technical Workgroup. (1998, December 4). Performance measures for mental health systems: A standardized framework. NASMHPD. National Association of State Mental Health Program Directors (NASMHPD) Research Institute. (1998, September). Five-State Feasibility Study on state mental health agency performance measures. NASMHPD Research Institute. Olfson, M., Mechanic, D., Boyer, C. A., Hansell, S., Walkup, J., & Weiden, P. J. (1999). Assessing clinical predictions of early rehospitalization in schizophrenia. The Journal of Nervous and Mental Disease, 187, 721–729. Phillips, K., & Rosenblatt, A. (1992). Speaking in tongues: Integrating economics and psychology into health and mental health services outcome research. Medical Care Review, 49, 191–23. Ralph, R. (1999, November 2). What is recovery? Paper presented at the Southern Regional Conference on Mental Health Statistics, New Orleans, LA. Schwartz, D. (1991). A follow up study of very early terminators from an outpatient clinic. Psychotherapy Research, 2(3), 164–180. Stein, L. I., & Test, M. A. (1980). An alternative to mental hospital treatment. I: Conceptual model, treatment program, and clinical evaluation. Archives of General Psychiatry, 37, 392–397. Teague, G. B., Bond, G. R., & Drake, R. E. (1998). Program fidelity in assertive community treatment: Development and use of a measure. American Journal of Orthopsychiatry, 68, 216–232. Test, M. A. (1992). Training in community living. In A. P. Liberman (Ed.), Handbook of psychiatric rehabilitation. New York: MacMillan. Tollefson, G. D., Beasley Jr., C. M., Tran, P. V., Street, J. S., Krueger, J. A., Tamura, R. N., Graffeo, K. A., & Thieme, M. E. (1997). Olanzapine versus haloperidol in the treatment of schizophrenia and schizoaffective and schizophreniform disorders: Results of an international collaborative trial. American Journal of Psychiatry, 154, 457–465. Wackwitz, J. (1998). Modeling analysis of the MSHIP consumer survey. Paper presented at the Southern Regional Conference, New Orleans, LA. Wehman, P. (1986). Supported competitive employment for persons with severe disabilities. Journal of Applied Rehabilitation Counseling, 17, 24–29. Wurster, C. R. (1997). Changes in data needs and requirements for cost-effective mental health. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services. |
|
|
| Home | Contact Us | About Us | Awards | Accessibility | Privacy and Disclaimer Statement | Site Map |
