 |
|
|
Chapter 9
The MHSIP Mental Health Quality Report:
The Next Generation of Performance Measures
Mary E. Smith, Ph.D.
Illinois Department of Human Services, Division of Mental Health;
Co-Chair, Mental Health Statistics
Improvement Program (MHSIP) Quality Report Task Force
Vijay Ganju, Ph.D.
National Association of State Mental Health Program Directors Research Institute,
Center for Mental Health Quality and Accountability; Chair, MHSIP Quality Report
Task Force
Introduction
In April 1996, the report of the Mental Health Statistics Improvement Program
(MHSIP) Consumer-oriented Report Card Task Force was published and released
at a widely attended press conference held in Washington, DC. This event was
the culmination of a 3-year effort initiated by the MHSIP Advisory Group to
develop a prototype consumer-oriented report card that could be used to compare
and evaluate the quality of mental health services on the basis of concerns
identified by mental health consumers.
The development of the MHSIP Report Card was a seminal event providing the
foundation for a wide range of national performance measurement activities
that have had an important impact on the mental health field over the past
10 years. The Report Card has provided the basis for performance measurement
initiatives implemented by the National Association of State Mental Health
Program Directors (NASMHPD), the American College of Mental Health Administration
(ACMHA), the American Managed Behavioral Healthcare Association (AMBHA), the
National Committee on Quality Assurance (NCQA), and the Substance Abuse and
Mental Health Services Administration’s (SAMHSA) Center for Mental Health
Services (CMHS). CMHS initiatives include the Five-State Feasibility Study,
the 16-State Performance Indicator Pilot, the Data Infrastructure Grants, the
Forum on Performance Measurement, and the SAMHSA National Outcome Measures
(NOMs).
New instruments and measures developed through various performance measurement
efforts have also refined and enhanced the original MHSIP Report Card. Instruments
related to children’s measures, the measurement of recovery, and measures
developed for inpatient settings are currently under development or being tested.
As a result of these efforts, the behavioral health field has learned many
lessons about implementing performance measurement systems and using performance
measures to evaluate the quality of mental health services. Recognizing the
need to document and consolidate the lessons learned by the field to improve
and enhance MHSIP Report Card measures and indicators, in the fall of 2001
the MHSIP Policy Group convened a task force to revise and update the MHSIP
Report Card.
The purposes of the revision are as follows:
- Incorporate the lessons learned from the development and implementation
experiences of MHSIP Report Card 1.0.
- Incorporate refinements to existing measures, add new measures,
and delete measures that did not work.
- Propose analytical and data presentation reports that could
be adapted for various uses, including systems accountability, quality improvement,
contract management, and consumer choice.
This chapter describes the process undertaken to revise the MHSIP Report Card,
the framework and the indicators and measures comprising the revised Report
Card, the relationship of the MHSIP revision to other performance measurement
initiatives, and the testing plan for the new indicators and measures.
During the course of its work on the MHSIP revision, the task force decided
to change the name of the Report Card to reflect the quality orientation of
the initiative more directly. Thus, the MHSIP Report Card was renamed the MHSIP
Mental Health Quality Report.
Historical Context
for the MHSIP Report Card
During the mid-1990s, the focus on national health care reform led to the development
of a number of health-related report cards aimed at measuring the cost and
effectiveness of care. Report cards were envisioned as a vehicle for comparing
health care plans along these dimensions. However, these efforts focused primarily
on developing reports related to physical health care. As a means of developing
a comparable vehicle for comparing mental health services using quality and
effectiveness measures, the CMHS MHSIP Ad Hoc Committee convened a task force
to develop a mental health report card. The MHSIP approach to this task was
groundbreaking in the following respects:
- The focus of the report card was on consumers of mental health services.
As stated in the overview of the Task Force Report, “the domains, concerns,
indicators and measures of the MHSIP report card are specifically designed
to assess consumer concerns with various aspects of mental health treatment,
not merely global satisfaction with mental health services” (Report
of the Task Force of the MHSIP Consumer-oriented
Report Card, 1996).
- The Report Card’s value was that it “explicitly
addressed issues of consumer choice, empowerment and involvement.” The
focus of the indicators and measures that were adopted included the expectation
that “appropriate services will be available, easily accessible, developed
with and by consumers, and offered in the least restrictive setting.” (Report
of the MHSIP Report Card Task Force, 1996).
- Consumers of mental health services were involved
in all aspects of the design and development of the Report Card.
- The Report Card was outcome-focused. None of the report cards
developed during the mid-1990s in the context of national health care reform
systematically addressed, or focused on, the outcomes, and certainly none focused
on the outcomes of mental health treatment. The MHSIP task force developed
specific indicators and measures to evaluate the outcome of treatment from
both the consumer’s and the clinician’s perspectives. While it
was acknowledged that there were difficulties associated with this focus (e.g.,
additional costs and the burden associated with collecting outcome data across
time), the task force noted that the ability to assess outcomes was a crucial
element of the Report Card framework.
- Other key features of the Report Card included its focus on
individuals with serious mental illnesses and the research base upon which
its measures and indicators were built.
Report Card Framework
The framework adopted for the MHSIP Report Card was composed of four broad
domains: Access, Appropriateness/Quality, Outcomes, and Prevention. Each domain
addressed consumer-identified concerns. The performance indicators and performance
measures developed by the task force reflected these concerns.
Use of the Report Card
The MHSIP Report Card Task Force recommended that the next phase of work focus
on pilot testing the proposed indicators
and measures. Further, it recommended that CMHS issue a specific Request for
Applications (RFAs) for pilot sites to test the Report Card. Shortly afterward,
CMHS developed a grant program for States to implement mental health performance
measurement systems using the indicators and measures in the Report Card as
a model. The adoption of Report Card measures by States for testing varied considerably.
However, some instruments and measures, such as the MHSIP Consumer Survey, were
used extensively across the States and were adopted by others working in the
mental health field. Several indicators directly derived from the MHSIP Survey
form the basis for current national SAMHSA initiatives, such as the State Data
Infrastructure Grants, NOMs, and the Forum on Performance Measures Modular Survey.
Some have been incorporated into many performance measurement initiatives, while
others have been abandoned because of difficulties associated with implementation,
or because they were deemed not to be useful. Some measures have been modified.
The Revision
Principles
The major impetus for the revision of the Report Card is to maintain the momentum
to build a consumer-centered system
that helps consumers move toward recovery. The objective of this effort is to
provide useful information for consumers, their families, authorities who oversee
mental health services, providers, and other stakeholders in promoting evidence-based
recovery, quality of care, accountability, and system improvement.
Values
Work on the MHSIP revision continues to be value-based. Implicit in its measures
are the following key values and expectations of the mental health system:
- Consumers and their families will have quick and easy
access to services.
- Consumers and their families will receive state-of-the-art
services appropriate to individual needs and preferences.
- The treatment and support that consumers and family
members receive will address the problems and concerns for which services
were sought.
- Consumers and family members will receive services
that do no harm, either directly through the services received or in the
environment within which services are provided.
Framework
The framework for the revision retains its focus on the domains of access,
quality/appropriateness, and outcomes. There is also a focus on the extent to
which mental health services facilitate or hinder recovery of individuals, through
the MHSIP collaboration with the Recovery
Research Workgroup. To ensure that the field is prepared to implement the Quality
Report, a toolkit has been released concurrently as a companion document
to the Quality Report that discusses the methodological and implementation
issues related to the proposed measures.
Differences Between the
MHSIP Report Card and the
MHSIP Quality Report
This second-generation effort is different from the original MHSIP Report Card
in three important ways. First, this new effort recognizes that different sets
of measures may be needed for different populations in different settings,
but a major aspect is to develop consistency and commonality across these sets.
Second, as noted above, the Quality Report builds on lessons learned
from performance measurement initiatives that have been implemented over
the last 6 years. In preparation for the revision, information was reviewed
from the following organizations’ initiatives: AMBHA, ACHMA, the National
Alliance for the Mentally Ill (NAMI), the NASMHPD Research Institute (NRI)
President’s Taskforce on Performance Measures, CMHS 16-State Study,
the Outcomes Roundtable for Children and Families, the Recovery Advisory
Group and the Recovery Measurement Workgroup, the Performance Measurement
Forum (Adult and Child Workgroups), the MHSIP Consumer-Oriented Report Card
(Version 1), and the work of various accreditation agencies. Additional input
was incorporated from representatives of the National Mental Health Association
(NMHA), the National Council for Community Behavioral Healthcare (NCCBH),
the Human Services Research Institute (HSRI), the National Association of
Consumer/Survivor Mental Health Administrators (NAC/SMHA), and the National
Association of Mental Health Planning and Advisory Councils (NAMHPAC).
A third difference between the two efforts is the emphasis on the use of the
proposed measures and performance indicators. When the original Report Card
was released, most mental health systems did not adopt the full set of
Report Card Measures. The MHSIP Quality Report emphasizes that a range
of measures across domains is needed to evaluate the quality and effectiveness
of mental health services.
The Development Process
Indicator Selection/Development Process
Combinations of several methods were used to consider and select indicators
for inclusion in Version 2 of the MHSIP Mental Health Quality Report. First,
members of the Task Force were asked to describe “lessons learned” from
the performance measurement initiatives in which their constituents had been
engaged. They were also asked to identify performance measures that would be
useful for inclusion in Version 2. Several Task Force members represented organizations
that were actively working to develop performance measures (e.g., the Children’s
Outcome Roundtable and the Recovery Workgroup). These members were asked
to discuss how their initiatives could relate to the goals of the Quality
Report workgroup and to share information when interim products were developed.
The Task Force systematically reviewed a set of performance measures and indicators
that are used across performance measurement systems. For this purpose, a matrix
was constructed listing the performance measures currently used by each system
or initiative. It was then possible to identify which measures have been adopted
by multiple performance measurement systems. Based on this information, the
Task Force selected initial measures and indicators to be considered for inclusion
in the MHSIP Quality Report.
It also decided to systematically review the concerns, rationale, and each individual
measure and indicator from the MHSIP Report Card. The purpose of this review
was to determine if the concerns were still relevant and if they had been adequately
addressed over time and to discuss alternative ways to address the concerns.
The third method built on the previous two methods. Each individual in the
Task Force was asked to identify additional measures for possible inclusion
in the Quality Report. This brainstorming process generated additional
measures and indicators, some of which were subsets of measures and indicators
previously considered. Although the size of this pool of indicators was considerable,
there was still concern that key issues of mental health consumers might
have been excluded. To address this possibility, a consumer expert panel
was established to review the set of indicators and measures developed by
the Task Force. This review resulted in the inclusion of additional indicators,
such as those related to safety, provider competence, availability of services,
and peer support services. The Task Force then winnowed down the pool of
indicators into unique sets. Fifty-two indicators comprised the final set.
Finally, the Task Force reviewed this material and reached consensus on a proposed
set of performance measures. To gather as much feedback as possible regarding
these measures, the MHSIP Policy Group developed a Web-based survey that was
posted on the MHSIP Web site for approximately 2 months. Invitations to comment
on the proposed measures were sent to a wide array of stakeholders, including
mental health consumers, family members, staff of State mental health authorities,
researchers, local and county-level mental health providers, and other interested
parties. Workgroup liaisons representing organizations to which invitations
were sent helped facilitate the process.
The survey asked respondents to answer four demographic questions to identify
their primary perspective. First they were asked which stakeholder group they
represented (e.g., advocate, consumer, State mental health authority), then
the organizational affiliation they might represent, then the primary population
in which they were interested, and finally any particular treatment setting
in which they had an interest. Respondents were then asked to rate each of
the 52 indicators as high, medium, or low priority, based on their specific
perspectives. The average rating for each indicator was calculated for each
perspective and overall. The 52 indicators were then ranked by perspective.
To further summarize the data, if an indicator was selected in the top 5 rankings
of any perspective, the number of times this occurred was tallied.
A total of 982 respondents completed the Web-based survey. Of those who identified
their perspective, 1 was from an accreditation organization, 117 were mental
health advocates, 270 were consumers of mental health services, 283 were family
members, 33 were from local mental health authorities, 8 were from managed
care organizations, 132 were providers, 74 were from State mental health authorities,
and 64 represented miscellaneous or unnamed groups. In addition, 6,953 comments
from respondents were reviewed, indicator by indicator, and were incorporated
in the final prioritization process.
To ensure that perspectives of people who had interests in particular populations
or specific treatment settings were represented in the findings, the rating
and ranking analysis described above was repeated for each reported population
interest category and each setting category. These analyses were the basis
for prioritizing the population-specific and setting-specific indicators. The
results of these analyses were used to develop the final set of proposed indicators
and measures.
MHSIP Quality Report Indicators and Measures
The indicators and measures proposed in the MHSIP Quality Report consist
of a universal set (which applies to all population subgroups and settings)
and additional indicators that apply to specific populations or specific
settings. For example, quality of treatment or services is a concern that
applies across populations and settings. Cultural competence is another universal
concern. On the other hand, an indicator such as improvement in school functioning
applies specifically to children. Similarly, a system’s recovery orientation
applies primarily to adults with serious mental illnesses. Some measures
apply more to the settings in which services are delivered than to the population
being served. For example, seclusion and restraint measures apply more to
inpatient and residential settings than to community outpatient programs.
Listed below are the indicators and definitions for each of the proposed
sets.
Universal Indicators
- Consumer Outcomes—An indicator
related
to improvement in functioning (i.e., how consumers handle social roles and
problems, address family and social situations, and cope with crises and psychological
distress).
- Active Participation in Treatment Planning—An
indicator of the degree to which consumers (or, for children, family members)
participate in treatment decisionmaking.
- Recovery Orientation—An indicator focused on the degree
to which an agency or organization
is recovery-oriented.
- Quality of Interaction Between Clinicians
and Consumers—An indicator of the degree to which consumers
feel they are treated with respect and dignity and feel safe and involved in
their treatment.
- Quality of Treatment—An indicator
of what consumers think about the overall quality of the treatment they receive.
- Safety—An indicator related
to patient safety, focused on medication errors.
- Availability of Services—An
indicator of the range of service options and treatments that are available.
- Availability of Information/Education—An
indicator of the degree to which consumers and family members receive information
and education that helps them make informed choices about mental health services.
- Initiation of Treatment—An indicator of whether persons
with mental illness have access to
appropriate care.
- Cultural Competence—An indicator
of the degree to which a consumer’s needs related to language, culture,
ethnicity, gender, sexual orientation, age, and disability are taken into
account.
- Co-occurring Problems/Screening—An
indicator of how often screenings are performed to detect substance abuse problems.
- Reduction of Symptoms—An indicator
of whether mental health treatment results in a reduction of a consumer’s
symptoms and an improved ability to function.
- Social Support/Connectedness—An
indicator of whether social support/connectedness is facilitated and supports
recovery.
Population-Specific Indicators
1. All Adults
- Peer Support—An indicator reflecting
the availability of consumer-operated or peer support services, including drop-in
centers, peer case management, peer professional services, and social clubs.
- Improvement in Work Functioning—An indicator of how
much consumers recently entering the
workforce think their ability to do paid work has improved.
2. Adults with Serious Mental Illness
- Adults with Schizophrenia—New Generation
Medications—An indicator of how available “new generation” medications
are in the mental health care system.
- Illness Self-Management—An
indicator of how available illness self-management training is in the mental
health care system.
3. All Children (Including Children with Serious Emotional Disturbances)
- Improvement in School Functioning—
An indicator of improvement in children’s attendance and
school performance.
- Social Relationships—An indicator
related to how social and personal relationships play important roles in facilitating
recovery.
- Involvement with Juvenile Justice System—An
indicator of a consumer’s contact with the criminal justice (or juvenile
justice) system.
- Illness Self-Management—An
indicator of how available illness self-management training is in the mental
health system.
Setting-Specific Indicators
(Note: The only settings identified as having specific measures were hospitals
and comprehensive community systems. The proposed universal measures applied
to all other settings.)
1. Hospitals/Inpatient
- Seclusion and Restraints—An indicator of how often
restrictive therapies are used or that treatment providers lack training or
respect for client autonomy and dignity.
2. Comprehensive Community Systems
- Perception of Access—An indicator
of how consumers feel about access to services—are they available at
times that are convenient, is location convenient, etc.
MHSIP Consumer Surveys
The revision of the MHSIP surveys reflects the general approach undertaken
in the development of the MHSIP Quality Report. That is, there are
universal items, as well as items for specific populations and specific settings
(e.g., recovery orientation).
Changes to the MHSIP Adult Consumer Survey have been based on three strands
of work: (1) recommendations from the February 2000 consumer survey workgroup;
(2) recommendations from the consumer survey workgroup convened under the umbrella
of the MHSIP Quality Report; and (3) feedback obtained from the Web-based
survey that was used for the MHSIP Quality Report. The work on the
Youth Services Surveys has begun with the formation of a task force that
will be reviewing the current survey forms to ensure “fit” with
the Quality
Report framework. Similar work will be conducted on the Inpatient Survey.
In addition to items on consumer perceptions of care, MHSIP Youth Services
Surveys contain a separate section for self-report items related to involvement
in the juvenile justice system, school attendance, access to primary health
care, and medication prescribed for emotional/behavioral problems. It has been
recommended that the use of self-report items be expanded and used as a source
of information on several performance measures that may be more difficult to
obtain, though perhaps less reliable than using alternative sources, such as
cross-system data matching. The MHSIP Quality Report Workgroup has
recommended that the testing of the revised MHSIP surveys incorporate a section
focusing on self-report items—thus several measures, such as involvement
with the juvenile justice system and school attendance, will be included
as part of the survey testing plan.
Implementation of the
Quality Report
The intent of the MHSIP Quality Report performance measures is
that they be used to reflect critical domains of an organization’s performance.
When the original MHSIP Report Card was proposed, many organizations selected
a few of the measures. This could happen again, but the goal of any performance
measurement system is to obtain a systemic view of an organization’s
operation. Use of individual indicators precludes a systemic view. At a minimum,
performance measures from all the domains must be obtained to reflect the
intent of the MHSIP Quality Report.
The MHSIP Quality Report can be used for various purposes: management,
planning, quality improvement, and providing information to consumers and
family members regarding an organization’s performance. To reflect
such performance accurately, attention must be given to data completeness
and quality; the methodologies for sampling, analysis, and benchmarking;
and the types of reports produced for different audiences and different uses.
The MHSIP Quality Report Toolkit
developed by the Human Services Research Institute, Cambridge, Massachusetts,
for the MHSIP Quality Report Task Force addresses these issues.
Testing of Indicators
The next step of the MHSIP Quality Report Task Force is to coordinate
efforts to test the newly proposed measures and indicators individually and
as a set. In fall of 2004, CMHS and the MHSIP Policy Group convened a Technical
Workgroup composed of representatives of various national testing initiatives,
including the Forum on Performance Measures, the Data Infrastructure Grant
initiative, the Recovery Measurement Work Group, SAMHSA’s Co‑Occurring
Disorder Infrastructure Grant (CODIG) initiative, NCQA, the Joint Commission
on Accreditation of Healthcare Organizations (JCAHO), and the Washington
Circle. The goal of this meeting was to develop parameters for testing the MHSIP
Quality Report indicators and to coordinate testing efforts with the groups
represented at the meeting.
The proposed testing plan that was developed is composed of two phases:
- Testing of individual indicators in
which the objective is to examine operational definitions for new measures
(i.e., measures not previously implemented) and to evaluate differences in
multiple definitions for existing measures.
- Testing the set of performance measures in
the MHSIP Quality Report in a subsequent
phase in which the focus is on evaluating the entire set of measures in different
settings. The objective is to test whether the set of measures can be implemented
to reflect the performance of the system and be used effectively for quality
improvement. An integral value of the Quality Report is that multiple
domains must be monitored simultaneously to be useful for management, quality
improvement, and planning purposes. To measure some performance indicators
and not others undermines the systemic nature of these indicator sets. This
phase of testing seeks to understand the relationships and potential redundancy
across the proposed measures. This phase will also test the measures for use
with various populations and settings.
While these are proposed as two distinct phases, they could occur simultaneously.
The plan is to test measures derived from both administrative databases and
from surveys under development. The measures will also be tested in different
settings and, if possible, for different uses.
The secondary goal of the Technical Workgroup, focusing on the coordination
of testing efforts of Quality Report measures with other performance
measurement testing efforts, was addressed through the development of a set
of recommendations specific to this goal. These recommendations and the specific
testing plan may be accessed on the MHSIP Web site at www.mhsip.org.
Next Steps
The MHSIP Quality Report and the MHSIP Quality Report Toolkit
were published in May 2005 and have been disseminated widely. Both documents
are also posted at www.mhsip.org. The MHSIP
Policy Group is currently recruiting organizations to participate in testing
proposed MHSIP Quality Report indicators and measures. Some testing
is being initiated through the CMHS Data Infrastructure Grants for Quality Improvement.
Other organizations have indicated an interest, and negotiations are under way.
The development of the MHSIP Quality Report has occurred at a propitious
point in time. The report of the President’s New Freedom Commission
has led many organizations to focus on developing strategies to transform
mental health care so that recovery is the expected outcome. Implementing
change that leads to a transformed mental health system, one that is consumer-driven
and recovery focused, requires the use of quality tools to assess progress
toward transformation and the outcomes of system transformation. The Mental Health
Statistics Improvement Program (MHSIP) Mental Health Quality Report provides
a cutting-edge framework composed of performance measures and strategies for
assessing and measuring transformation.
Reference
Task Force on a Mental Health Consumer-oriented Report Card. (1996). The
MHSIP Consumer-oriented Mental Health Report Card. Rockville, MD: Center
for Mental Health Services.
|
|
