Evidence-Based Practices: Shaping Mental Health Services Toward Recovery

Illness Management and Recovery Fidelity Scale

Introduction

What Is Illness Management and Recovery?

Illness Management and Recovery (IMR) is an evidence-based psychiatric rehabilitation practice. The primary aim of IMR is to empower consumers with severe mental illness (SMI) to manage their illness, find their own goals for recovery, and make informed decisions about their treatment by teaching them the necessary knowledge and skills. IMR involves a variety of interventions designed to help consumers improve their ability to overcome the debilitating effects of their illness on social and role functioning. Evidence supporting the effectiveness of illness management is available in studies focusing on the specific interventions as well as those evaluating comprehensive intervention packages.

The core evidence-based components of illness management are psychoeducation, behavioral tailoring for medication, relapse prevention training, and coping skills training. Psychoeducation provides the basic information about the nature of specific psychiatric disorders and the principles of treatment. Behavioral tailoring helps consumers manage daily medication regimes by teaching them strategies that make taking medication a part of their daily routine. Relapse prevention training teaches consumers to identify triggers of past relapses and early warning signs of an impending relapse, and helps them develop plans for preventing relapse. Coping skills training involves identifying people’s current coping strategies for dealing with persistent psychiatric symptoms and either increasing their use of these strategies or teaching new strategies. To effectively teach these components and to ensure that knowledge is put into practice, practitioners use a variety of techniques including motivational, educational and cognitive-behavioral strategies. Throughout the IMR program, practitioners help people to set and achieve their personal goals.

Overview of the Scale

The IMR Fidelity Scale contains 13 items that have been developed to measure the adequacy of implementation of IMR programs. Each item on the scale is rated on a 5-point behaviorally-anchored rating scale ranging from 1 (“Not implemented”) to 5 (“Fully implemented”). The standards used for establishing the anchors for the “Fully implemented” ratings were determined through a variety of expert sources as well as empirical research.

What is Rated

The scale is rated on current behavior and activities, not planned or intended behavior. For example, in order to get full credit for Item 3 (“Comprehensiveness of Curriculum”), it is not enough that the program is currently developing a curriculum.

How the Rating Is Done

The fidelity assessment is conducted through a site visit. It requires a minimum of 4 hours to complete, although a longer period of assessment will offer more opportunity to collect information and hence should result in a more valid assessment. The data collection procedures include chart review, review of educational handouts, and semi-structured interviews with the IMR program leader, IMR practitioners, and IMR consumers. When feasible, fidelity assessors should observe one or more IMR sessions (either live or a videotaped session).

The IMR fidelity assessment is primarily based on documentation in progress notes. Consequently, if these notes do not exist or are not easily available, the fidelity assessment will take a very different course. The goal is to examine the charts and 5 most recent progress notes of IMR sessions for each of 5 IMR consumers (preferably ones who have received IMR training for several months) for each of 3 IMR practitioners. If a practitioner has fewer than 5 IMR consumers, then use the charts and progress notes for all IMR consumers for that practitioner. If the site has more than 3 IMR practitioners, then the program leader should select 3 IMR practitioners for review. The fidelity assessors should aim to interview at least 3 consumers (one each per practitioner) for whom progress notes available.

The ideal is that the consumers chosen for review are randomly selected. It is also possible that the progress notes will not be integrated into the consumer charts (although this is optimal). In any situation, both the charts and the progress notes should be reviewed.

Who Does the Ratings

Fidelity assessments should be administered by individuals who have experience and training in interviewing and data collection procedures (including chart reviews). In addition, interviewers need to have an understanding of the nature and critical ingredients of IMR. We strongly recommend that all fidelity assessments be conducted by at least two assessors.

IMR Fidelity Assessor Checklist

Before the Fidelity Site Visit:

Establish a contact person at the program. The fidelity visit is coordinated by the site IMR program leader (or equivalent), with whom you arrange your visit. The program leader should communicate beforehand the purpose and scope of your assessment to the IMR practitioners. Schedule your visit when key staff are available and ideally when you can observe an IMR session. Exercise common courtesy in scheduling well in advance, respecting the competing time demands on practitioners, etc.
Create a timeline for the fidelity assessment. Fidelity assessments require careful coordination and good communication, particularly if there are multiple assessors. For instance, the timeline might include a note to make reminder calls to the program site to confirm interview dates and times.
Establish a shared understanding with the site being assessed. It is essential that the fidelity assessment team communicates to the each site the goals of the fidelity assessment. Assessors should also inform sites about who will see the report, whether the program site will receive this information, and exactly what information will be provided. The most successful fidelity assessments are those in which there is a shared goal among the assessors and the program site to understand how the program is progressing according to evidence-based principles. If administrators or line staff fear that they will lose funding or look bad if they don’t score well, then the accuracy of the data may be compromised. It is particularly important that fidelity assessors explain during a baseline interview that the goal is to obtain an initial picture of the program, and that high fidelity is not expected when a program is just starting.
Indicate what you will need from respondents during your fidelity visit. In addition to the purpose of the assessment, briefly describe what information you will need, who you will need to speak with, and how long each interview or visit will take to complete. The first step is to determine which practitioners are providing IMR training and which consumers are receiving IMR. The site visit will go more smoothly if the contact person can assemble the following information beforehand:
- List of names of IMR practitioners
- Roster of IMR consumers assigned to each practitioner
- A copy of agency brochure
- A copy of IMR program mission statement
- Copies of curriculum and educational handouts
- Total number of consumers served by the IMR program in the previous year
- Number of consumers who dropped out of the IMR program in the previous year
- Charts and progress notes
  The goal is to examine the charts along with the 5 most recent progress notes for IMR sessions for each of 5 IMR consumers (preferably ones who have received IMR training for several months) for each of 3 IMR practitioners. (See Appendix for sample progress note.)

During Your Fidelity Site Visit:

Overview: The general strategy in conducting program fidelity assessments is to obtain data from as many sources as possible. When all these data sources converge, then one can be more confident in the validity of the ratings. However, experience suggests that the sources often disagree. A review of progress notes should precede practitioner and consumer interviews. If the IMR interventions are well documented in progress notes, and the fidelity assessor is convinced that practitioners both understand the forms and have completed them conscientiously (by querying the practitioners and the program leader), then these progress notes can be used as the primary source for making ratings on the fidelity scale. If the information from different sources is not in agreement, (for example, if the program leader indicates a higher rate of use of a particular technique than is documented in the records), then ask the program leader to help you understand the discrepancy. The results from a chart or progress note review can be overruled if other data (e.g., team leader interview, internal statistics) refute it.
The first step in the fidelity assessment is an interview with the IMR program leader. The fidelity assessors should begin by reviewing the purpose for the visit and the schedule for the day. Explain that after the interview with the program leader, you will begin by reviewing charts and that the goal is to examine 5 progress notes for each of 5 IMR consumers (preferably ones who have received IMR training for several months) for each of 3 IMR practitioners. The schedule will then include interview with the 3 practitioners and with 1 IMR consumer for each practitioner.
The recommended schedule is as follows:
- Interview with program leader
- Review of progress notes and charts
- Observation of IMR session(s)
- Interviews with 3 IMR practitioners
- Interviews with 3 IMR consumers
- Final interview with program leader (to clarify information from the day;
  fill in gaps, etc.)

Tips:

Tailor terminology used in interviews to the site. For example, if the site uses the term “client” for consumer, use that term. Similarly, if practitioners are referred to as clinicians, use that terminology. Every agency has specific job titles for particular staff roles. By adopting the local terminology, the assessor will improve communication.
During the interview, record all the important names and numbers (e.g., numbers of practitioners, active IMR consumers, etc.) on the cover sheet.
If discrepancies between sources occur, query the IMR program director/coordinator, practitioners, or program leaders to get a better sense of the program’s performance in a particular area. The most common discrepancy is likely to occur when the interview with the program staff gives a more idealistic picture of the program’s functioning than do chart reviews or consumer interviews. For example, on Item 5 (“Involvement of Significant Others”), the practitioners may report that involvement of significant others in the IMR program was a common practice, while the majority of the charts reviewed may not document involvement of significant others. To understand and resolve this discrepancy, the assessor may need to go back to the practitioners and say something like, “Involvement of family members or friends was rarely documented in the charts we reviewed. Since you had reported that you always try to involve significant others, we wanted your help to understand the difference.”
Before you leave, check for missing data. The scale is designed to be filled out completely. If information cannot be obtained at time of the site visit, it will be important for you to be able to collect at a later date

After Your Fidelity Site Visit:

If necessary, follow up on any missing data (e.g., phone calls to the program site).
Assuming there are two assessors, both should independently rate the fidelity scale. The assessors should then compare their ratings and resolve any disagreements. Come up with a consensus rating.
Tally the item scores and determine which level of implementation was achieved (See Score Sheet).
Send a follow-up letter to the site. In most cases, this letter will include a fidelity report, explaining (to the program) their scores on the fidelity scale and providing some interpretation of the assessment, highlighting both strengths and weaknesses. The report should be informative, factual, and constructive. The recipients of this report will vary according to the purposes, but would typically include the key administrators involved in the assessment.
As fidelity is assessed over time, it is useful to create an Excel spreadsheet from which a graph of the total fidelity scores over time can be created and incorporated into the fidelity report.

TOC | NEXT