 |
This Web site is a component of the SAMHSA Health Information Network. |
|
Blueprint for Change: Ending Chronic
Homelessness for Persons with Serious
Mental Illnesses and Co-Occurring Substance Use Disorders
Chapter 7: Measure Results
Significant changes in the financing and delivery of health care,
including the rapid development of managed care, have increased the need to
monitor and evaluate costs, quality, and access (Kamis-Gould and Hadley,
1996). In the same vein, the Federal, state, and local agencies that fund
services for people who are homeless increasingly are using outcome measures
to balance service quality and effectiveness with limited resources.
Many states and localities have been influenced by the Government
Performance and Results Act of 1993 (P.L. 103-62), which requires Federal
agencies to set specific performance goals and to measure outcomes for
Federal programs (GAO, 1999b). SAMHSA’s new Performance Partnership grants
will give states more flexibility in how they spend their Federal mental
health and substance abuse block grant funds, and, in turn, the states will
have to show that they have been effective in meeting the goals they’ve set.
For programs that serve homeless people, a General Accounting Office
report notes, "The use of outcome measures shifts the focus from outputs,
such as the types and numbers of activities performed, to the outcomes, or
results achieved" (GAO, 1999b). This means, for example, that a provider
should include a measure of the number of people who become permanently
housed, along with the number of people the program serves.
This chapter examines (1) the rationale for measuring outcomes, (2) the
type of outcomes that should be measured, (3) the barriers to measuring
outcomes for people with serious mental illnesses and/or co-occurring
substance use disorders who are homeless, and (4) the use of management
information systems to track client data.
Why Measure Outcomes?
Measuring outcomes, first and foremost, is a way to ensure accountability.
Positive outcomes provide justification for continued services, which may help
programs sustain activities in difficult fiscal environments and/or when
start-up funding ends. Agencies also use outcome measures to evaluate their
progress in meeting strategic goals and objectives. Beyond these broader aims,
some specific reasons to measure outcomes of services for people who are
homeless include (HRSA BPHC, 1996):
- To demonstrate improvements in clients’ health status, level of
functioning, and quality of life. Residential stability also is a key
indicator that a program has met its goal of helping individuals exit
homelessness.
- To know what works and what doesn’t, and to be able to make appropriate
interventions more effective. People with serious mental illnesses and/or
co-occurring substance use disorders who are homeless are a heterogeneous
group. Measuring the effectiveness of specific interventions helps clinicians
know what works for whom, and at what cost.
- To assist with and assess internal quality improvement efforts.
Agencies can use outcome measures to make internal course corrections that
improve the quality of services they offer.
- To assess cost-effectiveness. Resources available to serve people who
are homeless and people who have serious mental illnesses and/or co-occurring
substance use disorders are limited. Providers can demonstrate that timely and
appropriate interventions result in cost savings.
- To assist in resource allocation. Limited resources necessitate
difficult decisions about how to allocate funds among providers and services.
Outcome measures can be used to help determine which providers and which
interventions are best able to meet clients’ needs.
- To exchange successful strategies. There is no need for programs that
serve people who are homeless to reinvent the wheel. As noted previously in
this report, states, communities, and individual agencies can adopt or adapt
practices that prove promising, as indicated by the outcomes they achieve.
- To build support for specific interventions that are effective with people
who are homeless. Showing positive results with vulnerable, high-need
individuals can help justify the expenditure of resources for a particular
program or approach.
- To increase client satisfaction. Though client satisfaction measures
may not always relate to successful clinical outcomes, individuals who are
satisfied with the type of services offered and the way in which services are
delivered may be more likely to complete treatment. For example, offering
people a choice in housing relates directly to their success in remaining
housed (Srebnik et al., 1995).
- To demonstrate positive impact on public health and social issues.
Ending homelessness for people with serious mental illnesses and/or
co-occurring substance use disorders reduces human suffering; it also reduces
the burden on the broader service system. For example, outcome measures show
that people who are housed and receiving appropriate treatment have less
contact with the criminal justice system and make less use of emergency room
and inpatient care.
Population-specific outcome measures also can help gauge the appropriateness
and effectiveness of managed care contracts. In an effort to improve quality and
contain costs, many states contract with managed behavioral health care
companies to provide mental health and substance abuse treatment to Medicaid
recipients. Serious questions have been raised about the ability of managed care
companies to respond appropriately to the needs of people with serious mental
illnesses or co-occurring disorders who are homeless. State agencies that
contract with these organizations can build in specific quality assurance
activities and outcome measures relevant to people who are homeless, as well as
incentives and sanctions to ensure compliance (Wunsch, 1998).
The Types of Outcomes That Should Be Measured
Providers who work with people who are homeless and have serious mental
illnesses and/or co-occurring substance use disorders agree on the need to
measure positive changes in both the client and in the service system. Outcome
assessment requires a thorough planning process and the involvement of key
stakeholders, including consumers (HRSA BPHC, 1998a).
Programs will measure different client-level and system-level outcomes
depending on a number of factors:
- Population they serve;
- Specific services they offer;
- Overall goals and objectives of the organization;
- Requirements of their primary funding source(s);
- Role they play in their community’s Continuum of Care for people who are
homeless; and
- Political climate in their state or community.
Outcomes measures frequently are expressed in broad terms such as
"reduction of barriers" (system-level outcome) or "mental health or substance
use status" (client-level outcome). Data collected to support the outcomes
typically are articulated as a series of specific measures, often called
performance indicators. For example, "the number of access points to the
system" is a performance indicator for reduction of barriers. Similarly, "the
number of psychiatric emergency admissions" is a performance indicator for
mental health status.
Some organizations compile outcomes and performance indicators in a document
called a report card. Report cards have been developed by national
oversight organizations, such as the National Committee on Quality Assurance (NCQA)
and the Joint Commission on Accreditation of Health Care Organizations (JCAHO);
by the insurance industry; and by large corporations. The SAMHSA Center for
Mental Health Services’ Mental Health Statistics Improvement Program (MHSIP) has
developed the MHSIP Consumer-Oriented Mental Health Report Card (MHSIP, 1996). A
report card informs key stakeholders, including policymakers, payers, providers,
and consumers, whether the agency or health care system does what is expected
and whether it does it well (MHSIP, 1996).
Client-Level Outcomes
Ultimately, the goal of an integrated system of care is to improve client
outcomes. For people with serious mental illnesses and/or co-occurring substance
use disorders who are homeless, this means improvements in mental symptoms,
substance use, housing status, and quality of life. Measures of improved
functioning for people with serious mental illnesses or co-occurring disorders
who are homeless may include the following outcomes (in bold) and suggested
performance indicators (in italics). Although some of these results are
difficult to quantify, they are a clear indication that all the preparatory work
has been successful (NASMHPD, 1995).
- Housing status: Number of days homeless; number of days in housing;
length of time in most recent housing placement; possession of housing
subsidy;
- Mental health status: Number of psychiatric emergency admissions;
number of days in inpatient treatment; self-report of mental health status;
- Substance use status: Number of days drinking and/or using drugs;
number or severity of problems associated with substance use; self-report of
substance use;
- Employment: Number of days employed; number of work days lost to mental
symptoms or substance use;
- Income: Average monthly income; receipt of SSI/SSDI or other public
benefits;
- Health status: Number of acute illnesses; number of inpatient days;
self-report of health status; has private or public health insurance;
- Family relationships: Self-report of quality of family relationships;
- Criminal justice involvement: Number of arrests; number of days
incarcerated;
- Social supports: Self-report of degree of social support;
- Consumer satisfaction: Self-report of satisfaction with a broad range
of variables, including housing, mental health, substance use, health status,
and degree of social support; and
- Quality of life: Measurable improvements in the expected direction in
the areas noted above; self-report of perceived quality of life.
System-Level Outcomes
At the system-level, most programs that serve people with serious mental
illnesses and/or co-occurring substance use disorders who are homeless will
measure access to services, quality of care, and cost-effectiveness. Other
system-level outcomes for programs that provide health care services to people
who are homeless may include availability of comprehensive services, continuity
of care, prevention activities, and degree of client involvement (HRSA BPHC,
1996).
Some specific system-level outcomes for programs that serve people with
serious mental illnesses and/or co-occurring substance use disorders who are
homeless also might include (NASMHPD, 1995):
- Attention to those not in the system;
- Degree of choice;
- Cultural competence;
- Reduction of barriers;
- Affordable housing options;
- Array of service options;
- Access to services clients want and need;
- Degree to which the mainstream system is responsive; and
- Level of support to maintain progress.
It also is possible to measure the level of systems integration. Measures of
improved system performance, designed to gauge the extent to which agencies
share information, resources, and clients, include the following performance
indicators (HRSA BPHC, 1996; Glover and Gustafson, 1999):
- Number and type of formal interagency agreements;
- Degree of blended funding;
- Number of joint activities between and among providers;
- Extent to which staff from participating agencies are trained in each
other’s disciplines;
- Degree to which application procedures have been streamlined and
exclusionary program rules have been waived;
- Extent to which the system offers "no wrong door" access; and
- Degree to which program planning and development incorporates the
participation of key community stakeholders, including consumers.
The Massachusetts Division of Medical Assistance has set certain
performance standards related to people who are homeless in its contract
with the company that provides mental health and substance abuse services
for many of the State’s Medicaid recipients. For example, one performance
standard requires the company to implement measures that will reduce the
inappropriate discharge of people from psychiatric facilities to shelters or
the streets. Another provides incentives to the company for increasing
enrollment of Medicaid-eligible individuals who are homeless (GAO, 1999b).
Barriers to Measuring Outcomes
There are numerous barriers to measuring outcomes for people with serious
mental illnesses and/or co-occurring substance use disorders who are homeless.
Indeed, mental illness, substance use, and homelessness each pose special
problems in valid and reliable assessment (Mercer-McFadden and Drake, 1992).
These problems are further complicated by the interaction of these conditions,
which makes categorizing their progress challenging at best.
If progress is difficult to measure, measuring "success" is even more
challenging. Much depends on how success is defined. For people with serious
mental illnesses and/or co-occurring substance use disorders, recovery may be
incremental and long term, often marked by numerous flare-ups and relapses.
Outcome measures therefore, must reflect the intensity of services required to
serve people with serious mental illnesses or co-occurring disorders who are
homeless, and must recognize the small steps that constitute success (HRSA BPHC,
1996).
Further, the availability of many services that individuals require to exit
homelessness, such as supportive housing, may be beyond the ability of
individual programs to control. Also, some outcome measures, such as "quality of
life," are difficult to quantify, especially when they rely on individual
self-reports.
In addition, data collected on program clients cannot capture information on
the individuals who are not in the system but whose needs may be just as
great, if not greater, than those of the individuals being served. Finally,
measuring outcomes requires development and implementation of data systems that
are sophisticated and user-friendly, as well as staff trained in the use of
these tools. Even when the systems and personnel are in place, however,
fragmented, duplicative, and inconsistent reporting and evaluation requirements
attached to different funding streams pose an ongoing challenge for many
providers.
Use of Management Information Systems
Collecting client-level information can help streamline services, reduce
duplication of effort, improve access, and inform public policy. Beginning in
2004, the U.S. Department of Housing and Urban Development will require all
government and nonprofit agencies receiving McKinney-Vento Homeless Assistance
funds to implement homeless information management systems (HMIS) (University of
Massachusetts, 2001).
HMIS have already been implemented in 20 to 25 jurisdictions, including
several statewide plans and a handful of communities that are pooling resources
to build local tracking networks. In other jurisdictions, State and local HMIS
efforts are in various stages of planning, pilot testing, and implementation.
The benefits of such systems to individuals who are homeless include
streamlined referrals, coordinated case management, and reduction of duplicative
intakes and assessments. Service agencies gain reporting tools, mechanisms for
internal and external service coordination, and information that can inform
service and systems planning and advocacy. Policymakers benefit from data that
can include the types and number of services provided, an unduplicated count of
individuals being served, population characteristics and service needs, and
trends over time.
States and local jurisdictions implementing HMIS must ensure this data
collection is done in compliance with the Health Insurance Portability and
Accountability Act of 1996 (HIPAA). HUD is preparing security and data element
guidelines for compliance with HIPAA. Under the guidelines, jurisdictions will
be able to decide what information can be shared. To be compliant with HIPAA,
any sharing of health information must be done with client consent, and
individuals must be informed of what information is being shared and why (NAEH,
2003).
In December 2000, the Wisconsin Division of Housing and Intergovernmental
Relations purchased a commercial software package to implement a statewide
system using a central database, and offered bonus points to agencies
applying for funds as an incentive to participate. To date, 84 agencies
representing 68 counties participate in the statewide HMIS. The system is
designed to protect confidential data and allow clients to determine which
records are shared. The HMIS system has reduced or eliminated monthly or
quarterly reporting requirements for many agencies, decreased duplicative
client intakes, helped coordinate services and streamline referrals,
provided better access to data for funders and other stakeholders, and
offered on-line access to a statewide database of service providers
(University of Massachusetts, 2001).
Connecting to Mainstream Resources
Weaving best practices for people with serious mental illnesses and/or
co-occurring substance use disorders who are homeless into a seamless system of
care and measuring the results is a tall order indeed. In the last decade, the
homeless service system has created a wealth of programs and services that show
positive results for this very vulnerable group. However, the homeless service
system does not have the human or financial resources to address this problem
fully. The final chapter of this report examines some ways in which mainstream
resources can play a vital role in ending homelessness among people with serious
mental illnesses or co-occurring disorders.
Table of Contents | Previous | Next
|
|
