Community Support

Research and Evaluation

the Evaluation Center@HSRI
May 1999
Volume 1, Issue 5

This is the fifth in a series of briefings on new and current mental health services evaluations, resources, and/or methods. We hope FastFacts will be a quick and easy way for you to learn important information in the field of evaluation. If you have any ideas on how FastFacts could be more useful to you, please contact Dawna Phillips at 617-876-0426 x325 or phillips@hsri.org.

the Evaluation Center@HSRI is a technical assistance center funded by the federal Center for Mental Health Services (CMHS), Substance Abuse and Mental Health Services Administration (SAMHSA) and operated by the Human Services Research Institute (HSRI). The mission of the Center is to provide evaluation technical assistance to state and non-profit public and private entities including but not limited to consumers, families and provider groups. The Center presently has six programs designed to fulfill this mission -

The Consultation Program,
The Topical Evaluation Network Program,
The Toolkit Program,
The Evaluation Materials Program,
The Conferences & Training Program, and
The Multicultural Issues in Evaluation Program.

Mortality Can be a Powerful Performance Indicator

Growing evidence suggests that measures of mortality in a mental health service recipient population can be a powerful indicator of system performance. As performance measurement has become nearly ubiquitous in mental health systems, from local to state to federal levels, there has been a search to identify quantifiable indicators of system performance. Ideal performance measures not only reflect overall system performance, but also stimulate systems improvements. Recent experience suggests measures of mortality might meet these dual goals.

Mortality rates are commonly used as global measures of health status for populations (Grob, 1983; Zopf, 1992). Furthermore, they are increasingly being used as indicators of performance of public health efforts; for example, mortality rates are used a number of times in Healthy People 2000 as performance measures.

Mortality may be particularly appropriate for application to mental health systems because of the extensive literature demonstrating that persons with severe mental illness die at higher rates and at younger ages than the overall population, from both natural and medicolegal (i.e., homicide, suicide, or accidents/injuries) causes (e.g., Segal & Kotler, 1991; Winokur & Black, 1987; Babigian & Ordoroff, 1969). This literature indicates that the overall health status of recipients of mental health services is typically poor, and measuring mortality is a way of tracking population health status as improvement efforts are made. Recent experience in at least one state suggests, further, that tracking mortality rates can stimulate system improvements.

The Massachusetts Case

Massachusetts was one of the first states to use measures of service recipient mortality as a gauge of system performance. Its experience highlights the ability of mortality to stimulate system improvement efforts, and how this performance measure can be misinterpreted if appropriate methodological steps are not taken.

In an article published on June 11, 1995, the Boston Globe indicated that "while the caseload [for the DMH] has remained steady or decreased slightly, the number of people who die while under the care of the State Department of Mental Health has climbed dramatically in recent years" (Bass, 1995).

Following this article, the Department of Mental Health convened a Task Force to examine trends in the client mortality rate and related issues. In a report issued in January 1996, the Task Force concluded that the DMH client mortality rate was actually falling, not rising, since the number of persons counted as DMH clients had increased at a greater rate than the number of deaths (See Figure 1). This highlights the importance of using a mortality rate, not simply the number of deaths, as a measure of system performance.

Figure 1: Massachusetts DMH State wide Mortality Rate 1991-1993

State Vide Mortality Rate Graph

Despite this positive finding, the Task Force did find that DMH clients were dying from all causes at a younger age than average. The problem of "preventable or postponable deaths" was considered of paramount importance by the Task Force, and its report included several recommendations specifically designed to ameliorate this problem.

State Mental Health Directors Include Mortality in Framework of Measures

The growing recognition of the importance and utility of mortality as a performance measure is evidenced by its inclusion in the National Association of State Mental Health Program Directors (NASMHPD) standardized framework for performance measurement. The framework, developed by NASMHPD's President's Task Force on Performance Measures, includes 33 indicators intended to guide performance measurement efforts of comprehensive mental health systems. The indicator on mortality suggests three different measures: (1) The crude mortality rate for the population of persons who received at least one service in the past year, (2) The standardized mortality ratio (SMR) for the population of persons who received at least one service in the past year, defined as the ratio of the number of observed deaths in a population to the number of expected deaths based on an overall population controlling for age and sex, and (3) The average number of years of life lost (YLL) for service recipients who died in the last year, defined as the difference between the age at death and the current life expectancy for an individual.

The NASMHPD Task Force Technical Workgroup makes several points with respect to use and interpretation of this indicator:

"First, as with other measures, appropriate risk adjustment methods need to be employed before certain comparisons can be made; this is particularly important for the first measure, since the SMR adjusts for age, sex and the overall mortality rate of a geographic area by definition. Second, mortality is the result of complex processes that may be influenced by events from the immediate and the more distant past. As such, mortality may be influenced by events occurring before the period for which performance is to be measured. Unless these events are irreversible, however, we can expect successful service interventions aimed at improving the health status of a population to be reflected in these measures. Third, this indicator is most useful when the served population is large, for example in a statewide system. If the system serves only a relatively small number of persons, random variation over time may be misinterpreted as reflecting system performance" (NASMHPD, 1998).

Another benefit of using mortality rates as performance measures is that they can be calculated from existing data. The most direct way to extract data to calculate a mortality rate is to use unique identifiers to link electronic records of persons served with electronic death records (typically available from state public health authorities). Dembling (in press) shows how public health data can be used to generate rates of deaths by cause for mental health system consumers. If linking by identifiers is not desired due to confidentiality concerns, a statistical procedure can be used to approximate an actual one-to-one link. This procedure, called probabalistic population estimation, is described by its developers, John Pandiani, Ph.D. and Steve Banks, Ph.D., in an Evaluation Center publication, "Methodology for Estimating Caseload Size and Overlap."

In sum, if attention is paid to appropriate measurement and methodological issues, mortality can serve as a useful performance indicator, and one that may be particularly powerful in its ability to promote system improvement efforts.

For more information on using mortality as a performance measure, contact Lawrence Woocher at woocher@hsri.org or (617) 876-0426 x326.

For more information on the Center, please contact us at:
Telephone: 617-876-0426 ext.4
Email: hrsi@hsri.org
Website: http://tecathsri.org/
Address: 2269 Massachusetts Avenue
Cambridge, MA 02140

References

Babigian, H.M., & Odoroff, C.L. (1969). The Mortality Experience of a Population with Psychiatric Illness. American Journal of Psychiatry, 126, 470-480.

Bass, A. (1995, June 11). DMH Sees Increase in Deaths. The Boston Globe, pp. 1.

Dembling, B. (in press). Toolkit for Linking Mental Health Consumer Files with State Death Records: Using Mortality Information to Evaluate the Needs and Outcomes of Consumer Populations. (Toolkit soon to be released by the Evaluation Center@HSRI. For more information contact Theresa Conley at (617) 876-0426 x332).

Grob, R. (1993). Toward a comparative overview of dependence potential and acute toxicity of psychoactive substances used nonmedically.American Journal of Drug and Alcohol Abuse, 19, 263-281.

NASMHPD President’s Task Force on Performance Measures Technical Workgroup (1998) Draft Recommended Operational Definitions and Measures to Implement the NASMHPD Framework of Mental Health Performance Indicators. To obtain, go to http://nri.rdmc.org/ or call (703) 739-9333.

Segal, S., & Kotler, P. (1991). A ten year perspective of mortality among mentally ill patients in sheltered care. Hospital and Community Psychiatry, 42, 708-713.

Winokur, G., & Black, D. February, 1987). Psychiatric and Medical Diagnoses as Risk Factors for Mortality in Psychiatric Patients: A Case-Control Study. American Journal of Psychiatry.

Mortality patterns and trends in the United States. In Studies in Population and Urban Demography, 7. Westport, Connecticut: Greenwood Press.