SAMHSA's National Mental Health Information Center

This Web site is a component of the SAMHSA Health Information Network

About the Toolkits

Illness Management and Recovery

Assertive Community Treatment

Family Psychoeducation

Supported Employment

Co-occuring Disorders: Integrated Dual Diagnosis Treatment

Feedback Form

Evidence-Based Practices: Shaping Mental Health Services Toward Recovery

Illness Management and Recovery

General Organizational Index (GOI)—Item Definitions and Scoring

G1. Program Philosophy

Definition
The program is committed to a clearly articulated philosophy consistent with the specific evidence-based practice (EBP), based on the following 5 sources:

Program leader
Senior staff (e.g., executive director, psychiatrists)
Practitioners providing EBP
Clients and/or family members (depending on EBP focus)
Written materials (e.g., brochures)

Rationale

In psychiatric rehabilitation programs that truly endorse EBPs, staff members at all levels embrace the program philosophy and practice it in their daily work.

Sources of Information

Overview

During the course of a site visit, fidelity assessors should be alert to indicators of program philosophy consistent with or inconsistent with the EBP including observations from casual conversations, staff and client activities, etc. Statements that suggest mis-conceptions or reservations about the practice are negative indicators, while statements that indicate enthusiasm for and understanding of the practice are positive indicators. The intent of this item is to gauge the understanding of and commitment toward the practice. It is not necessary that every element of the practice is currently in place (this is gauged by the EBP-specific fidelity scale), but rather whether all those involved are committed to implementing a high fidelity EBP.

The practitioners rated for this item are limited to those implementing this practice. Similarly, the clients rated are those receiving the practice.

a) Program leader interview, b) Senior staff interview and c) Practitioner interview:

At the beginning of interview, have the staff briefly describe the program.
“What are the critical ingredients or principles of your services?”
“What is the goal of your program?”
“How you define [EBP area]?”

d) Client interview:

“What kind of services do you receive from this program?”
Using a layperson’s language, describe to the client/family, the principles of the specific EBP area; probe if the program offers services that reflect each principle.
“Do you feel the staff of this program competent and helpful to you in addressing your problems?”

e) Written material review (e.g., brochure):

Does the site have written materials on the EBP? If no written material, then item is rated done one scale point (i.e., lower fidelity).
Does the written material articulate program philosophy consistent with EBP?

Item Response Coding

The goal of this item is not to quiz every staff worker to determine if they can recite every critical ingredient. The goal is to gauge whether the understanding is generally accurate and not contrary to the EBP. If, for example, a senior staff member says, “most of our clients are not work ready,” then that would be a red flag for the practice of supported employment. If all sources show evidence of a clear understanding of the program philosophy, the item is coded as a “5”. For a source type that is based on more than one person (e.g., Practitioner interviews) determine the majority opinion when rating that source as endorsing or not endorsing a clear program philosophy. Note: If no written material, then count that source as being unsatisfactory.

Difference between a major and minor area of discrepancy (needed to distinguish between a score of “4” and a score of “3”): An example of a minor source of discrepancy for ACT might be larger caseload sizes (e.g., 20-1) or some brokering of services. An example of a major discrepancy would be if the team seldom made home visits or if the psychiatrist was uninvolved in the treatment team meetings.

G2. Eligibility/Client Identification

Definition

For EBPs implemented in a mental health center: All clients in the community support program, crisis clients, and institutionalized clients are screened using standardized tools or admission criteria that are consistent with the EBP.

For EBPs implemented in a service area: All clients within the jurisdiction of the services area are screened using standardized tools or admission criteria that are consistent with the EBP. For example, in New York, county mental health administrations are responsible for identifying clients who will be served by assertive community treatment programs.

The target population refers to all adults with severe mental illness (SMI) served by the provider agency (or service area). If the agency serves clients at multiple sites, then assessment is limited to the site or sites that are targeted for the EBP. If the target population is served in discrete programs (e.g., case management, residential, day treatment, etc.), then ordinarily all adults with SMI are included in this definition.
Screening will vary according to the EBP. The intent is to identify any and all for who could benefit from the EBP. For Integrated Dual Disorder Treatment and Assertive Community Treatment, the admission criteria are specified by the EBP and specific assessment tools are recommended for each. For Supported Employment, all clients are invited to receive the service because all are presumed eligible (although the program is intended for clients at the point they express interest in working). The screening for Illness Management & Recovery includes an assessment of the skills and issues addressed by this EBP. For Family Psychoeducation, the screening includes the assessment of the involvement of a family member or significant other. In every case, the program should have an explicit, systematic method for identifying the eligibility of every client.
Screening typically occurs at program admission, but for a program that is newly adopting an EBP, there should be a plan for systematically reviewing clients already active in the program.

Rationale

Accurate identification of clients who would benefit most from the EBP requires routine review for eligibility, based on criteria consistent with the EBP.

Sources of Information

a) Program leader interview, b) Senior staff interview and c) Practitioner interview:

“Describe the eligibility criteria for your program.”
“How are clients referred to your program? How does the agency identify clients who would benefit from your program? Do all new clients receive screening for [substance abuse or SMI diagnosis]?”
“What about crisis [or institutionalized] clients?”
Request a copy of the screening instrument used by the agency.

d) Chart review

Review documentation of screening process & results.

e) (Where applicable) County mental health administrators.

If eligibility is determined at the service area level (e.g., the New York example), then the individuals responsible for this screening should be interviewed.

Item Response Coding

This item refers to all clients with SMI in the community support program or its equivalent at the site(s) where the EBP is being implemented; it is not limited to the clients receiving EBP services only. Calculate this percentage and record it on the fidelity rating scale in the space provided. If 100% of these clients receive standardized screening, the item would be coded as a “5.”

G3. Penetration

Definition

Penetration is defined as the percentage of clients who have access to an EBP as measured against the total number of clients who could benefit from the EBP. Numerically, this proportion is
defined by:

# of clients receiving an EBP
# of clients eligible for the EBP

As in the preceding item, the numbers used in this calculation are specific to the site or sites where the EBP is being implemented.

Rationale

Surveys have repeatedly shown that persons with SMI often have a limited access to EBPs. The goal of EBP dissemination is not simply to create small exclusive programs but to make these practices easily accessible within the public mental health system.

Sources of Information

The calculation of the penetration rate depends of the availability of the two statistics defining
this rate.

Numerator: The number receiving the service is based on a roster of names maintained by the program leader. Ideally, this total should be corroborated with service contact sheets and other supporting evidence that the identified clients are actively receiving treatment. As a practical matter, agencies have many conventions for defining “active clients” and dropouts, so that it may be difficult to standardize the definition for this item. The best estimate of the number actively receiving treatment should be used.
Denominator: If the provider agency systematically tracks eligibility, then this number is used in the denominator. (See rules listed above in G2 to determine target population before using estimates below.) If the agency does not, then the denominator must be estimated by multiplying the total target population by the corresponding percentage based on the literature for each EBP. According to the literature, the estimates should be as follows:
- Supported Employment – 60%
- Integrated Dual Disorders Treatment – 40%
- Illness Management & Recovery – 100%
- Family Psychoeducation – 100% (some kind of significant other)
- Assertive Community Treatment – 20%

Example for calculating denominator: Suppose you don’t know how many consumers are eligible for supported employment (i.e., the community support program has not surveyed the clients to determine those who are interested). Let’s say the community support program has 120 clients. Then you would estimate the denominator to be:
120 x .6 = 72

Item Response Coding

Calculate this ratio and record it on the fidelity scale in the space provided. If the program serves >80% of eligible clients, the item would be coded as a “5”.

G4. Assessment

Definition

All EBP clients receive standardized, high quality, comprehensive, and timely assessments.

Standardization refers to a reporting format that is easily interpreted and consistent across clients.
High quality refers to assessments that provide concrete, specific information that differentiates between clients. If most clients are assessed using identical words, or if the assessment consists of broad, noninformative checklists, then this would be considered low quality.
Comprehensive assessments include: history and treatment of medical, psychiatric, and substance use disorders, current stages of all existing disorders, vocational history, any existing support network, and evaluation of biopsychosocial risk factors.
Timely assessments are those updated at least annually.

Rationale

Comprehensive assessment/re-assessment is indispensable in identifying target domains of functioning that may need intervention, in addition to the client’s progress toward recovery.

Sources of Information

a) Program leader interview, b) Senior staff interview and c) Practitioner interview:

“Do you give a comprehensive assessment to new clients? What are the components that you assess?”
Request a copy of the standardized assessment form, if available, and have the practitioners go through the form.
“How often do you re-assess clients?”

d) Chart review:

Look for comprehensiveness of assessment by looking at multiple completed assessments to see of they address each individual component of the comprehensive assessment each time an assessment is performed.
Is the assessment updated at least yearly?

Item Response Coding

If >80% of clients receive standardized, high quality, comprehensive, and timely assessments, the item would be coded as a “5”.

G5. Individualized Treatment Plan

Definition

For all EBP clients, there is an explicit, individualized treatment plan (even if it is not called this) related to the EBP that is consistent with assessment and updated every 3 months. “Individualized” means that goals, steps to reaching the goals, services/ interventions, and intensity of involvement are unique to this client. Plans that are the same or similar across clients are not individualized. One test is to place a treatment plan without identifying information in front of the supervisor and see if they can identify the client.

Rationale

Core values of EBP include individualization of services and supporting clients’ pursuit of their goals and progress in their recovery at their own pace. Therefore, the treatment plan needs ongoing evaluation and modification.

Sources of Information

Note: This item and the next are assessed together; i.e., follow up questions about specific treatment plans with question about the treatment.

a) Chart review (treatment plan):

Using the same charts as examined during the EBP-specific fidelity assessment, look for documentation of specific goal(s) and client-based goal-setting process.
Are the treatment recommendations consistent with assessment?
Evidence for a quarterly review (and modification)?

b) Program leader interview

“Please describe the process of developing a treatment plan. What are the critical components of a typical treatment plan and how are they documented?”

c) Practitioner interview:

When feasible, use the specific charts selected above. Ask the practitioners go over a sample treatment plan.
“How do you come up with client goals?” Listen for client involvement and individualization of goals.
“How often do you review (or follow up on) the treatment plan?”

d) Client interview:

“What are your goals in this program? How did you set these goals?”
“Do you and your practitioner together review your progress toward achieving your goal(s)?” [If yes] “How often? Please describe the review process.”

e) Team meeting/supervision observation, if available:

Observe how treatment plan is developed. Listen especially for discussion of assessment, client preferences, and individualization of treatment.
Do they review treatment plans?

Item Response Coding

If >80% of EBP clients have an explicit individualized treatment plan that is updated every 3 months, the item would be coded as a 5. IF the treatment plan is individualized but updated only every 6 months, then the item would be coded as a 3.

G6. Individualized Treatment

Definition

All EBP clients receive individualized treatment meeting the goals of the EBP. “Individualized” treatment means that steps, strategies, services/interventions, and intensity of involvement are focused on specific client goals and are unique for each client. Progress notes are often a good source of what really goes on. Treatment could be highly individualized despite the presence of generic treatment plans.

An example of a low score on this item for Integrated Dual Disorders Treatment: a client in the engagement phase of recovery is assigned to a relapse prevention group and constantly told he needs to quit using, rather than using motivational interventions.

An example for a low score on this item for Assertive Community Treatment: the majority of progress notes are written by day treatment staff who see the client 3-4 days per week, while the Assertive Community Treatment team only sees the client about once per week to issue his check.

Rationale

The key to the success of an EBP is implementing a plan that is individualized and meets the goals for the EBP for each client.

Sources of Information

a) Chart review (treatment plan):

Using the same charts as examined during the EBP-specific fidelity assessment, examine the treatment provided. Limit the focus to a recent treatment plan related to the EBP. The assessor should judge whether an appropriate treatment occurred during the time frame indicated by the treatment plan.

b) Practitioner interview:

When feasible, use the specific charts selected above. Ask the practitioners to go over a sample treatment plan and treatment.

c) Client interview:

“Tell me about how this program or practitioner is helping you meet your goals.”

Item Response Coding

If >80% of EBP clients receive treatment that is consistent with the goals of the EBP, the item would be coded as a 5.

G7. Training

Definition

All new practitioners receive standardized training in the EBP (at least a 2-day workshop or its equivalent) within 2 months of hiring. Existing practitioners receive annual refresher training (at least 1-day workshop or its equivalent).

Rationale

Practitioner training and retraining are warranted to ensure that evidence-based services are provided in a standardized manner, across practitioners and over time.

Sources of Information

a) Program leader interview, b) Senior staff interview and c) Practitioner interview:

“Do you provide new practitioners with systematic training for [EBP area]?” [If yes] Probe for specifics: mandatory or optional, length, frequency, content, group or individual format, who trains, in-house or outside training, etc.
“Do Practitioners already on the team receive refresher trainings?” [If yes] Probe
for specifics.

d) Review of training curriculum and schedule, if available:

Does the curriculum appropriately cover the critical ingredients for [EBP area]?

e) Practitioner interview:

“When you first started in this program, did you receive a systematic/formal training for [EBP area]?” [If yes] Probe for specifics: mandatory or optional, length, frequency, content, group or individual format, who trained, in-house or outside training, etc.
“Do you receive refresher trainings?” [If yes] Probe for specifics.

Item Response Coding

If >80% of practitioners receive at least yearly, standardized training for [EBP area], the item would be coded as a “5”.

G8. Supervision

Definition

EBP practitioners receive structured, weekly supervision from a practitioner experienced in the particular EBP. The supervision can be either group or individual, but CANNOT be peers-only supervision without a supervisor. The supervision should be client-centered and explicitly address the EBP model and its application to specific client situations.

Administrative meetings and meetings that are not specifically devoted to the EBP do not fit the criteria for this item. The client-specific EBP supervision should be at least one hour in duration
each week.

Rationale

Regular supervision is critical not only for individualizing treatment, but also for ensuring the standardized provision of evidence-based services.

Sources of Information

a) Program leader interview, b) Senior staff interview and c) Practitioner interview:

Probe for logistics of supervision: length, frequency, group size, etc.
“Please describe what a typical supervision session looks like.”
“How does the supervision help your work?”

d) Team meeting/supervision observation, if available:

Listen for discussion of [EBP area] in each case reviewed.

e) Supervision logs documenting frequency of meetings.

Item Response Coding

If >80% of practitioners receive weekly supervision, the item would be coded as a “5”.

G9. Process Monitoring

Definition

Supervisors/program leaders monitor the process of implementing the EBP every 6 months and use the data to improve the program. Process monitoring involves a standardized approach, e.g., use of a fidelity scale or other comprehensive set of process indicators. An example of a process indicator would be systematic measurement of how much time individual case managers spend in the community versus in the office. Process indicators could include items related to training or supervision. The underlying principle is that whatever is being measured is related to implementation of the EBP and is not being measured to track billing or productivity.

Rationale

Systematic and regular collection of process data is imperative in evaluating program fidelity to EBP.

Sources of Information

a) Program leader interview, b) Senior staff interview and c) Practitioner interview:

“Does your program collect process data regularly?” [If yes] Probe for specifics: frequency, who, how (using [EBP area] Fidelity Scale vs. other scales), etc.
“Does your program collect data on client service utilization and treatment attendance?”
“Have the process data impacted how your services are provided? For example?”

d) Review of internal reports/documentation, if available

Item Response Coding

If there is evidence that standardized process monitoring occurs at least every 6 months, the item would be coded as a “5”.

G10. Outcome Monitoring

Definition

Supervisors/program leaders monitor the outcomes of EBP clients every 3 months and share the data with EBP practitioners in an effort to improve services. Outcome monitoring involves a standardized approach to assessing clients.

Rationale

Systematic and regular collection of outcome data is imperative in evaluating program effectiveness. Effective programs also analyze such data to ascertain what is working and what is not working, and use the results to improve the quality of services they provide.

The key outcome indicators for each EBP are discussed in the implementation resource kits. A provisional list is as follows:

Supported Employment – competitive employment rate
Integrated Dual Disorders Treatment – substance use (such as the Stages of Treatment Scale)
Illness Management & Recovery – hospitalization rates; relapse prevention plans; medication compliance rates
Family Psychoeducation – hospitalization and family burden
Assertive Community Treatment – hospitalization and housing

Sources of Information

a) Program leader interview, b) Senior staff interview and c) Practitioner interview:

“Does your program have a systematic method for tracking outcome data?” [If yes] Probe for specifics: how (computerized vs. chart only), frequency, type of outcome variables, who collects data, etc.
“Do you use any checklist/scale to monitor client outcome (e.g., Substance Abuse Treatment Scale)?”
“What do you do with the outcome data? Do your practitioners review the data on regular basis?” [If yes] “How is the review done (e.g., cumulative graph)?”
“Have the outcome data impacted how your services are provided? For example?”

d) Review of internal reports/documentation, if available

Item Response Coding

If standardized outcome monitoring occurs quarterly and results are shared with EBP Practitioners, the item would be coded as a “5”.

G11. Quality Assurance (QA)

Definition

The agency’s QA Committee has an explicit plan to review the EBP or components of the program every 6 months. The steering committee for the EBP can serve this function. Good QA committees help the agency in important decisions, such as penetration goals, placement of the EBP within the agency, hiring/staffing needs. QA committees also help guide and sustain the implementation by reviewing fidelity to the EBP model, making recommendations for improvement, advocating/promoting the EBP within the agency and in the community, and deciding on and keeping track of key outcomes relevant to the EBP.

Rationale

Research has shown that programs that most successfully implement evidence-based practices have better outcomes. Again, systematic and regular collection of process and outcome data is imperative in evaluating program effectiveness.
Sources of Information

a) Program leader interview:

“Does your agency have an established team/committee that is in charge of reviewing the components of your [EBP area] program?” [If yes] Probe for specifics: who, how, when, etc.

b) QA Committee member interview:

“Please describe the tasks and responsibilities of the QA Committee.” Probe for specifics: purpose, who, how, when, etc.
“How do you utilize your reviews to improve the program’s services?”

Item Response Coding

If agency has an established QA group or steering committee that reviews the EBP or components of the program every 6 months, the item would be coded as a “5”.

G12. Client Choice Regarding Service Provision

Definition

All clients receiving EBP services are offered a reasonable range of choices consistent with the EBP; the EBP practitioners consider and abide by client preferences for treatment when offering and providing services.

Choice is defined narrowly in this item to refer to services provided. This item does not address broader issues of client choice, such as choosing to engage in self-destructive behaviors.

To score high on this item, it is not sufficient that a program offers choices. The choices must be consonant with EBP. So, for example, a program implementing supported employment would score low if the only employment choices it offered were sheltered workshops.

A reasonable range of choices means that EBP practitioners offer realistic options to clients rather than prescribing only one or a couple of choices or dictating a fixed sequence or prescribing conditions that a client must complete before becoming eligible for a service.

Sample of Relevant Choices by EBP

Supported Employment
- Type of occupation
- Type of work setting
- Schedules of work and number of hours
- Whether to disclose
- Nature of accommodations
- Type and frequency of follow-up supports
Integrated Dual Disorders Treatment
- Group or individual interventions
- Frequency of DD treatment
- Specific self-management goals
Family Psychoeducation
- Client readiness for involving family
- Who to involve
- Choice of problems/issues to work on
Illness Management & Recovery
- Selection of significant others to be involved
- Specific self management goals
- Nature of behavioral tailoring
- Skills to be taught
Assertive Community Treatment
- Type and location of housing
- Nature of health promotion
- Nature of assistance with financial management
- Specific goals
- Daily living skills to be taught
- Nature of medication support
- Nature of substance abuse treatment

Rationale

A major premise of EBP is that clients are capable of playing a vital role in the management of their illnesses and in making progress towards achieving their goals. Providers accept the responsibility of getting information to clients so that they can become more effective participants in the treatment process.

Sources of Information

a) Program leader interview.

“Please tell us what your program philosophy is regarding client choice. How do you incorporate their preferences in the services you provide?”
“What options are there for your services? Please give examples.”

b) Practitioner interview.

“What do you do when there is a disagreement between what you think is the best treatment for a client and what he/she wants?”
“Please describe a time when you were unable to abide by a client’s preferences.”

c) Client interview.

“Does the program give you options for the services you receive? Are you receiving the services you want?”

d) Team meeting/supervision observation.

Look for discussion of service options and client preferences.

e) Chart review (especially treatment plan).

Look for documentation of client preferences and choices.

Item Response Coding

If all sources support that type and frequency of EBP services always reflect client choice, the item would be coded as a “5”. If agency embraces client choice fully, except in one area (e.g., requiring the agency to assume representative payeeships for all clients), then the item would be coded as a “4”.

TOC | Cover Sheet