Evidence-Based Practices:
Shaping Mental Health Services Toward Recovery
General Organizational Index (GOI)
Item Definitions and Scoring
G1. Program Philosophy
Definition: The program is committed to
a clearly articulated philosophy consistent with the specific evidence-based
practice (EBP), based on the following 5 sources:
- Program leader
- Senior staff (e.g., executive director, psychiatrists)
- Practitioners providing EBP
- Clients and/or family members (depending on EBP focus)
- Written materials (e.g., brochures)
Rationale: In psychiatric rehabilitation programs
that truly endorse EBPs, staff members at all levels embrace the program philosophy
and practice it in their daily work.
Sources of Information: During the course of a site
visit, fidelity assessors should be alert to indicators of program philosophy
consistent with or inconsistent with the EBP including observations from casual
conversations, staff and client activities, etc. Statements that suggest mis-conceptions
or reservations about the practice are negative indicators, while statements
that indicate enthusiasm for and understanding of the practice are positive
indicators. The intent of this item is to gauge the understanding of and commitment
toward the practice. It is not necessary that every element of the practice
is currently in place (this is gauged by the EBP-specific fidelity scale), but
rather whether all those involved are committed to implementing a high fidelity
EBP.
The practitioners rated for this item are limited to those implementing
this practice. Similarly, the clients rated are those receiving the
practice.
a) Program leader interview, b) Senior staff interview and c) Practitioner
interview:
- At the beginning of interview, have the staff briefly describe the program.
- “What are the critical ingredients or principles of your services?”
- “What is the goal of your program?”
- “How you define [EBP area]?”
d) Client interview:
- “What kind of services do you receive from this program?”
- Using a layperson’s language, describe to the client/family, the
principles of the specific EBP area; probe if the program offers services
that reflect each principle.
- “Do you feel the staff of this program competent and
helpful to you in addressing your problems?”
e) Written material review (e.g., brochure):
- Does the site have written materials on the EBP? If no written material,
then item is rated done one scale point (i.e., lower fidelity).
- Does the written material articulate program philosophy consistent with
EBP?
Item Response Coding: The goal of this item is not
to quiz every staff worker to determine if they can recite every critical ingredient.
The goal is to gauge whether the understanding is generally accurate and not
contrary to the EBP. If, for example, a senior staff member says, “most
of our clients are not work ready,” then that would be a red flag for
the practice of supported employment. If all sources show evidence of a clear
understanding of the program philosophy, the item is coded as a “5”.
For a source type that is based on more than one person (e.g., Practitioner
interviews) determine the majority opinion when rating that source as endorsing
or not endorsing a clear program philosophy. Note: If no written material, then
count that source as being unsatisfactory.
Difference between a major and minor area of discrepancy (needed to distinguish
between a score of “4” and a score of “3”): An
example of a minor source of discrepancy for ACT might be larger caseload sizes
(e.g., 20-1) or some brokering of services. An example of a major discrepancy
would be if the team seldom made home visits or if the psychiatrist was uninvolved
in the treatment team meetings.
G2. Eligibility/Client Identification
Definition: For EBPs implemented in a mental health
center: All clients in the community support program, crisis clients, and
institutionalized clients are screened using standardized tools or admission
criteria that are consistent with the EBP.
For EBPs implemented in a service area: All clients within the jurisdiction
of the services area are screened using standardized tools or admission criteria
that are consistent with the EBP. For example, in New York, county mental health
administrations are responsible for identifying clients who will be served by
assertive community treatment programs.
- The target population refers to all adults with severe mental illness
(SMI) served by the provider agency (or service area). If the agency serves
clients at multiple sites, then assessment is limited to the site
or sites that are targeted for the EBP. If the target population
is served in discrete programs (e.g., case management, residential, day treatment,
etc.), then ordinarily all adults with SMI are included in this definition.
- Screening will vary according to the EBP. The intent is to
identify any and all who could benefit from the EBP. For Integrated
Dual Disorder Treatment and Assertive Community Treatment, the
admission criteria are specified by the EBP and specific assessment tools
are recommended for each. For Supported Employment, all clients are
invited to receive the service because all are presumed eligible (although
the program is intended for clients at the point they express interest in
working). The screening for Illness Management & Recovery includes
an assessment of the skills and issues addressed by this EBP. For Family
Psychoeducation, the screening includes the assessment of the involvement
of a family member or significant other. In every case, the program should
have an explicit, systematic method for identifying the eligibility of every
client.
- Screening typically occurs at program admission, but for a program that
is newly adopting an EBP, there should be a plan for systematically reviewing
clients already active in the program.
Rationale: Accurate identification of clients who
would benefit most from the EBP requires routine review for eligibility, based
on criteria consistent with the EBP.
Sources of Information:
a) Program leader interview, b) Senior staff interview and c) Practitioner
interview:
- “Describe the eligibility criteria for your program.”
- “How are clients referred to your program? How does the agency identify
clients who would benefit from your program? Do all new clients receive screening
for [substance abuse or SMI diagnosis]?”
- “What about crisis [or institutionalized] clients?”
- Request a copy of the screening instrument used by the agency
d) Chart review
- Review documentation of screening process & results
e) (Where applicable) County mental health administrators. If
eligibility is determined at the service area level (e.g., the New York example),
then the individuals responsible for this screening should be interviewed.
This item refers to all clients with SMI in the community support program
or its equivalent at the site(s) where the EBP is being implemented; it is not
limited to the clients receiving EBP services only. Calculate this percentage
and record it on the fidelity rating scale in the space provided. If 100% of
these clients receive standardized screening, the item would be coded as a “5.”
G3. Penetration
Definition: Penetration is defined as the
percentage of clients who have access to an EBP as measured against the total
number of clients who could benefit from the EBP. Numerically, this proportion
is defined by:
# of clients receiving an EBP
# of clients eligible for the EBP
As in the preceding item, the numbers used in this calculation are
specific to the site or sites where the EBP is being implemented.
Rationale: Surveys have repeatedly shown that persons
with SMI often have a limited access to EBPs. The goal of EBP dissemination
is not simply to create small exclusive programs but to make these practices
easily accessible within the public mental health system.
Sources of Information: The calculation of the penetration
rate depends of the availability of the two statistics defining this rate.
- Numerator: The number receiving the service is based on
a roster of names maintained by the program leader. Ideally, this total should
be corroborated with service contact sheets and other supporting evidence
that the identified clients are actively receiving treatment. As a practical
matter, agencies have many conventions for defining “active clients”
and dropouts, so that it may be difficult to standardize the definition for
this item. The best estimate of the number actively receiving treatment should
be used.
- Denominator: If the provider agency systematically tracks
eligibility, then this number is used in the denominator. (See rules listed
above in G2 to determine target population before using estimates below.)
If the agency does not, then the denominator must be estimated by multiplying
the total target population by the corresponding percentage based on the literature
for each EBP. According to the literature, the estimates should be as follows:
- Supported Employment – 60%
- Integrated Dual Disorders Treatment – 40%
- Illness Management & Recovery – 100%
- Family Psychoeducation – 100% (some kind of significant other)
- Assertive Community Treatment – 20%
Example for calculating denominator: Suppose you don’t know how many
consumers are eligible for supported employment (i.e., the community support
program has not surveyed the clients to determine those who are interested).
Let’s say the community support program has 120 clients. Then you would
estimate the denominator to be:
120 x .6 = 72
Item Response Coding: Calculate this ratio and record
it on the fidelity scale in the space provided. If the program serves >80%
of eligible clients, the item would be coded as a “5”.
G4. Assessment
Definition: All EBP clients receive standardized,
high quality, comprehensive, and timely assessments.
- Standardization refers to a reporting format that is easily interpreted
and consistent across clients.
- High quality refers to assessments that provide concrete, specific
information that differentiates between clients. If most clients are assessed
using identical words, or if the assessment consists of broad, noninformative
checklists, then this would be considered low quality.
- Comprehensive assessments include: history and treatment of medical,
psychiatric, and substance use disorders, current stages of all existing disorders,
vocational history, any existing support network, and evaluation of biopsychosocial
risk factors.
- Timely assessments are those updated at least annually.
Rationale: Comprehensive assessment/re-assessment
is indispensable in identifying target domains of functioning that may need
intervention, in addition to the client’s progress toward recovery.
Sources of Information:
a) Program leader interview, b) Senior staff interview
and c) Practitioner interview:
- “Do you give a comprehensive assessment to new clients? What are
the components that you assess?”
- Request a copy of the standardized assessment form, if available, and have
the practitioners go through the form.
- “How often do you re-assess clients?”
d) Chart review:
- Look for comprehensiveness of assessment by looking at multiple completed
assessments to see of they address each individual component of the comprehensive
assessment each time an assessment is performed.
- Is the assessment updated at least yearly
Item Response Coding: If >80% of clients receive
standardized, high quality, comprehensive, and timely assessments, the item
would be coded as a “5”.
G5. Individualized Treatment Plan
Definition: For all EBP clients, there is an explicit, individualized treatment
plan (even if it is not called this) related to the EBP that is consistent with
assessment and updated every 3 months. “Individualized” means
that goals, steps to reaching the goals, services/ interventions, and intensity
of involvement are unique to this client. Plans that are the same or similar
across clients are not individualized. One test is to place a treatment plan
without identifying information in front of the supervisor and see if they can
identify the client.
Rationale: Core values of EBP include individualization
of services and supporting clients’ pursuit of their goals and progress
in their recovery at their own pace. Therefore, the treatment plan needs ongoing
evaluation and modification.
Sources of Information: Note: This item and the next
are assessed together; i.e., follow up questions about specific treatment plans
with question about the treatment.
a) Chart review (treatment plan):
- Using the same charts as examined during the EBP-specific fidelity
assessment, look for documentation of specific goal(s) and client-based
goal-setting process.
- Are the treatment recommendations consistent with assessment?
- Evidence for a quarterly review (and modification)?
b) Program leader interview:
- “Please describe the process of developing a treatment plan. What
are the critical components of a typical treatment plan and how are they documented?”
c) Practitioner interview:
- When feasible, use the specific charts selected above. Ask the practitioners
go over a sample treatment plan.
- “How do you come up with client goals?” Listen for
client involvement and individualization of goals.
- “How often do you review (or follow up on) the treatment plan?”
d) Client interview:
- “What are your goals in this program? How did you set these goals?”
- “Do you and your practitioner together review your progress toward
achieving your goal(s)?” [If yes] “How often? Please describe
the review process.”
e) Team meeting/supervision observation, if available:
- Observe how treatment plan is developed. Listen especially for discussion
of assessment, client preferences, and individualization of treatment.
- Do they review treatment plans?
Item Response Coding: If >80% of EBP clients have
an explicit individualized treatment plan that is updated every 3 months, the
item would be coded as a 5. IF the treatment plan is individualized but updated
only every 6 months, then the item would be coded as a 3.
G6. Individualized Treatment
Definition: All EBP clients receive individualized treatment meeting the goals
of the EBP. “Individualized” treatment means that steps, strategies,
services/interventions, and intensity of involvement are focused on specific
client goals and are unique for each client. Progress notes are often a good
source of what really goes on. Treatment could be highly individualized despite
the presence of generic treatment plans.
An example of a low score on this item for Integrated Dual Disorders Treatment:
a client in the engagement phase of recovery is assigned to a relapse prevention
group and constantly told he needs to quit using, rather than using motivational
interventions.
An example for a low score on this item for Assertive Community Treatment:
the majority of progress notes are written by day treatment staff who see the
client 3-4 days per week, while the Assertive Community Treatment team only
sees the client about once per week to issue his check.
Rationale: The key to the success of an EBP is implementing
a plan that is individualized and meets the goals for the EBP for each client.
Sources of Information:
a) Chart review (treatment plan):
- Using the same charts as examined during the EBP-specific fidelity
assessment, examine the treatment provided. Limit the focus to a
recent treatment plan related to the EBP. The assessor should judge whether
an appropriate treatment occurred during the time frame indicated by the treatment
plan.
b) Practitioner interview:
- When feasible, use the specific charts selected above. Ask the practitioners
to go over a sample treatment plan and treatment.
c) Client interview:
- “Tell me about how this program or practitioner is helping you
meet your goals.”
Item Response Coding: If >80% of EBP clients
receive treatment that is consistent with the goals of the EBP, the item would
be coded as a 5.
G7. Training
Definition: All new practitioners receive standardized
training in the EBP (at least a 2-day workshop or its equivalent) within 2 months
of hiring. Existing practitioners receive annual refresher training (at least
1-day workshop or its equivalent).
Rationale: Practitioner training and retraining are
warranted to ensure that evidence-based services are provided in a standardized
manner, across practitioners and over time.
Sources of Information:
a) Program leader interview, b) Senior staff interview and
c) Practitioner interview:
- “Do you provide new practitioners with systematic training for
[EBP area]?” [If yes] Probe for specifics:
mandatory or optional, length, frequency, content, group or individual format,
who trains, in-house or outside training, etc.
- “Do Practitioners already on the team receive refresher trainings?”
[If yes] Probe for specifics.
d) Review of training curriculum and schedule, if available:
- Does the curriculum appropriately cover the critical ingredients for [EBP
area]?
e) Practitioner interview:
- “When you first started in this program, did you receive a systematic/formal
training for [EBP area]?” [If yes]
Probe for specifics: mandatory or optional, length, frequency, content, group
or individual format, who trained, in-house or outside training, etc.
“Do you receive refresher trainings?” [If yes]
Probe for specifics.
Item Response Coding: If >80% of practitioners
receive at least yearly, standardized training for [EBP area],
the item would be coded as a “5”.
G8. Supervision
Definition: EBP practitioners receive structured,
weekly supervision from a practitioner experienced in the particular EBP. The
supervision can be either group or individual, but CANNOT be peers-only supervision
without a supervisor. The supervision should be client-centered and explicitly
address the EBP model and its application to specific client situations.
Administrative meetings and meetings that are not specifically devoted to the
EBP do not fit the criteria for this item. The client-specific EBP supervision
should be at least one hour in duration each week.
Rationale: Regular supervision is critical not only
for individualizing treatment, but also for ensuring the standardized provision
of evidence-based services.
Sources of Information:
a) Program leader interview, b) Senior staff interview and
c) Practitioner interview:
- Probe for logistics of supervision: length, frequency, group size, etc.
- “Please describe what a typical supervision session looks like.”
- “How does the supervision help your work?”
d) Team meeting/supervision observation, if available:
- Listen for discussion of [EBP area] in each case reviewed.
e) Supervision logs documenting frequency of meetings.
Item Response Coding: If >80% of practitioners
receive weekly supervision, the item would be coded as a “5”.
G9. Process Monitoring
Definition: Supervisors/program leaders monitor the
process of implementing the EBP every 6 months and use the data to improve the
program. Process monitoring involves a standardized approach, e.g., use of a
fidelity scale or other comprehensive set of process indicators. An example
of a process indicator would be systematic measurement of how much time individual
case managers spend in the community versus in the office. Process indicators
could include items related to training or supervision. The underlying principle
is that whatever is being measured is related to implementation of the EBP and
is not being measured to track billing or productivity.
Rationale: Systematic and regular collection of process
data is imperative in evaluating program fidelity
to EBP.
Sources of Information:
a) Program leader interview, b) Senior staff interview and
c) Practitioner interview:
- “Does your program collect process data regularly?” [If
yes] Probe for specifics: frequency, who, how (using [EBP area]
Fidelity Scale vs. other scales), etc.
- “Does your program collect data on client service utilization
and treatment attendance?”
- “Have the process data impacted how your services are provided?
For example?”
d) Review of internal reports/documentation, if available
Item Response Coding: If there is evidence that standardized
process monitoring occurs at least every 6 months, the item would be coded as
a “5”.
G10. Outcome Monitoring
Definition: Supervisors/program leaders monitor the
outcomes of EBP clients every 3 months and share the data with EBP practitioners
in an effort to improve services. Outcome monitoring involves a standardized
approach to assessing clients.
Rationale: Systematic and regular collection of outcome
data is imperative in evaluating program effectiveness. Effective programs also
analyze such data to ascertain what is working and what is not working, and
use the results to improve the quality of services they provide.
The key outcome indicators for each EBP are discussed in the implementation
resource kits. A provisional list is as follows:
- Supported Employment – competitive employment rate
- Integrated Dual Disorders Treatment – substance use (such as the
Stages of Treatment Scale)
- Illness Management & Recovery – hospitalization rates; relapse
prevention plans; medication compliance rates
- Family Psychoeducation – hospitalization and family burden
- Assertive Community Treatment – hospitalization and housing
Sources of Information:
a) Program leader interview, b) Senior staff interview and
c) Practitioner interview:
- “Does your program have a systematic method for tracking outcome
data?” [If yes] Probe for specifics: how (computerized vs.
chart only), frequency, type of outcome variables, who collects data, etc.
- “Do you use any checklist/scale to monitor client outcome (e.g.,
Substance Abuse Treatment Scale)?”
- “What do you do with the outcome data? Do your practitioners review
the data on regular basis?” [If yes] “How is the
review done (e.g., cumulative graph)?”
- “Have the outcome data impacted how your services are provided?
For example?”
d) Review of internal reports/documentation, if available.
Item Response Coding: If standardized outcome monitoring
occurs quarterly and results are shared with EBP Practitioners, the item would
be coded as a “5”.
G11. Quality Assurance (QA)
Definition: The agency's QA Committee has an explicit
plan to review the EBP or components of the program every 6 months. The steering
committee for the EBP can serve this function. Good QA committees help the agency
in important decisions, such as penetration goals, placement of the EBP within
the agency, hiring/staffing needs. QA committees also help guide and sustain
the implementation by reviewing fidelity to the EBP model, making recommendations
for improvement, advocating/promoting the EBP within the agency and in the community,
and deciding on and keeping track of key outcomes relevant to the EBP.
Rationale: Research has shown that programs that
most successfully implement evidence-based practices have better outcomes. Again,
systematic and regular collection of process and outcome data is imperative
in evaluating program effectiveness.
Sources of Information:
a) Program leader interview:
- “Does your agency have an established team/committee that is in
charge of reviewing the components of your [EBP area]
program?” [If yes] Probe for specifics: who, how, when,
etc.
b) QA Committee member interview:
- “Please describe the tasks and responsibilities of the QA Committee.”
Probe for specifics: purpose, who, how, when, etc.
- “How do you utilize your reviews to improve the program’s
services?”
Item Response Coding: If agency has an established
QA group or steering committee that reviews the EBP or components of the program
every 6 months, the item would be coded as a “5”.
G12. Client Choice Regarding Service Provision
Definition : All clients receiving EBP services are offered
a reasonable range of choices consistent with the EBP; the EBP practitioners
consider and abide by client preferences for treatment when offering and providing
services.
Choice is defined narrowly in this item to refer to services provided. This
item does not address broader issues of client choice, such as choosing to engage
in self-destructive behaviors.
To score high on this item, it is not sufficient that a program offers choices.
The choices must be consonant with EBP. So, for example, a program implementing
supported employment would score low if the only employment choices it offered
were sheltered workshops.
A reasonable range of choices means that EBP practitioners offer realistic
options to clients rather than prescribing only one or a couple of choices or
dictating a fixed sequence or prescribing conditions that a client must complete
before becoming eligible for a service.
Sample of Relevant Choices by EBP:
- Supported Employment: Type of occupation; Type of work setting; Schedules
of work and number of hours; Whether to disclose; Nature of accommodations;
Type and frequency of follow-up supports
- Integrated Dual Disorders Treatment: Group or individual interventions;
Frequency of DD treatment; Specific self-management goals
- Family Psychoeducation: Client readiness for involving family; Who to
involve; Choice of problems/issues to work on
- Illness Management & Recovery: Selection of significant others to
be involved; Specific self management goals; Nature of behavioral tailoring;
Skills to be taught
- Assertive Community Treatment: Type and location of housing; Nature
of health promotion; Nature of assistance with financial management; Specific
goals; Daily living skills to be taught; Nature of medication support; Nature
of substance abuse treatment
Rationale: A major premise of EBP is that clients are
capable of playing a vital role in the management of their illnesses and in making
progress towards achieving their goals. Providers accept the responsibility of
getting information to clients so that they can become more effective participants
in the treatment process.
Sources of Information:
a) Program leader interview:
- “Please tell us what your program philosophy is regarding client
choice. How do you incorporate their preferences in the services you provide?”
- “What options are there for your services? Please give examples.”
b) Practitioner interview:
- “What do you do when there is a disagreement between what you
think is the best treatment for a client and what he/she wants?”
- “Please describe a time when you were unable to abide by a client’s
preferences.”
c) Client interview:
- “Does the program give you options for the services you receive?
Are you receiving the services you want?”
d) Team meeting/supervision observation:
- Look for discussion of service options and client preferences.
e) Chart review (especially treatment plan):
- Look for documentation of client preferences and choices.
Item Response Coding: If all sources support that
type and frequency of EBP services always reflect client choice, the item would
be coded as a “5”. If agency embraces client choice fully, except
in one area (e.g., requiring the agency to assume representative payeeships
for all clients), then the item would be coded as a “4”.
Back to Top
Back to Assertive Community Treatment
TOC | Cover Sheet
|
