Evidence-Based Practices: Shaping Mental Health Services Toward Recovery

Assertive Community Treatment Fidelity Scale

Introduction

ACT Overview
As an evidence-based psychiatric rehabilitation practice, ACT provides a comprehensive approach to service delivery to consumers with severe mental illness (SMI). ACT uses a multidisciplinary team, which typically includes a psychiatrist, a nurse, and at least two case managers. ACT is characterized by (1) low client to staff ratios; (2) providing services in the community rather than in the office; (3) shared caseloads among team members; (4) 24-hour staff availability, (5) direct provision of all services by the team (rather than referring consumers to other agencies); and (6) time-unlimited services.

Overview of the Scale
The ACT Fidelity Scale contains 28 program-specific items. The scale has been developed to measure the adequacy of implementation of ACT programs. Each item on the scale is rated on a 5-point scale ranging from 1 (“Not implemented”) to 5 (“Fully implemented”). The standards used for establishing the anchors for the “fully-implemented” ratings were determined through a variety of expert sources as well as empirical research. The scale items fall into three categories: human resources (structure and composition); organizational boundaries; and nature of services.

What Is Rated
The scale ratings are based on current behavior and activities, not planned or intended behavior. For example, in order to get full credit for Item O4 (“responsibility for crisis services”), it is not enough that the program is currently developing an on-call plan.

Unit of Analysis
The scale is appropriate for organizations that are serving clients with SMI and for assessing adherence to evidence-based practices, specifically for an ACT team. If the scale is to be used at an agency that does not have an ACT team, a comparable service unit should be measured (e.g., a team of intensive case managers in a community support program). The DACTS measures fidelity at the team level rather than at the individual or agency level.

How the Rating Is Done
To be valid, a fidelity assessment should be done in person, i.e., through a site visit. The fidelity assessment requires a minimum of 6 hours to complete, although a longer period of assessment will offer more opportunity to collect information; hence, it should result in a more valid assessment. The data collection procedures include chart review, team meeting observation, home visits, and semi-structured interview with the team leader. Clinicians who work on the ACT teams are also valuable sources of data; most frequently the assessors obtain this information when accompanying them on home visits. Data may be obtained through other sources (e.g., supervisors, consumers) as appropriate.

Some items require calculation of either the mean or the median value of service data (e.g., median number of community-based contact contacts); specific administration instructions are given as needed for individual items (see below).

For some items that require chart review for rating, the intent is to use charts selected at random. Some processes for randomly selection are suggested below; assessors should feel free to use whatever method is most convenient or practical for the particular visit.

Prior to site visit, request the team leader select 20 charts for review; from these 20 charts, assessors select 10 at random.
Center provides a de-identified list of clients (i.e., ID numbers) and the assessors use random selection to choose 10.
It is important to select the most representative sample of charts; if a team assigns clients to different levels of service intensity, the sample should reflect this (e.g., a team with 30% of its clients on Level 1, 60% of clients on Level 2, and 10% on Level 3, 30% of reviewed charts should come from Level 1 clients, 60% of reviewed charts from Level 2, and so on).

How to Rate a Newly-Established Team
For ACT teams in the start-up phase, the time frame specified in individual items may not be met. For example, item H5 asks for the turnover rate during the last two years; Item O2 asks for the number of new clients during the last six months. Assessors should prorate time frames for teams that have been in operation for a shorter amount of time than specified in the individual items. (Specific instructions given for pertinent items.)

How to Rate Programs Using Other Program Models
The DACTS is designed to assess programs following the ACT model. If a case management or other program is rated on the DACTS, some items do not apply. This protocol does not cover every possible case of program model. In most instances, if an item cannot be rated, the assessor should assign a value of “1” for that item.

Who Does the Ratings
The scale can be administered internally by an agency/program or by an external review group. If it is administered internally, it is obviously important for the ratings to be made objectively, based on hard evidence, rather than made to “look good.” Circumstances will dictate decisions in this area, but we encourage agencies to choose a review process that fosters objectivity in ratings, e.g., by involving a staff person who is not centrally involved in providing the service. With regard to external reviews, there is a distinct advantage in using assessors who are familiar with the program, but at the same time are independent. The goal in this process is the selection of objective and competent assessors.

Fidelity assessments should be administered by individuals who have experience and training in interviewing and data collection procedures (including chart reviews). In addition, raters need to have an understanding of the nature and critical ingredients of ACT. We recommend that all fidelity assessments be conducted by at least two raters in order to increase reliability of the findings.

Missing Data
The scale is designed to be filled out completely, with no missing data on any items. It is essential that raters obtain the required information for every item. It is critical that raters record detailed notes of responses given by the interviewees. If information cannot be obtained at time of the site visit, it will be important for the raters to collect it at a later date.

Fidelity Assessor Checklist
Before the Fidelity Site Visit

Review the sample cover sheet. This sheet is useful for organizing your fidelity assessment, identifying where the specific assessment was completed, along with general descriptive information about the site. You may need to tailor this sheet for your specific needs (e.g., unique data sources, purposes for the fidelity assessment).
Create a timeline for the fidelity assessment. Fidelity assessments require careful coordination of efforts and good communication, particularly if there are multiple assessors. Therefore, it may be useful to list all the necessary activities leading up to and during the visit. For instance, the timeline might include a note to make reminder calls to the program site to confirm interview dates and times.
Establish a contact person at the program. You should have one key person who arranges your visit and communicates beforehand the purpose and scope of your assessment to program staff. Typically this will be the ACT team leader. Exercise common courtesy in scheduling well in advance, respecting the competing time demands on clinicians, etc.
Establish a shared understanding with the site being assessed. It is essential that the fidelity assessment team communicates to each program site the goals of the fidelity assessment; assessors should also inform the program site about who will see the report, whether the program site will receive this information, and exactly what information will be provided. The most successful fidelity assessments are those in which there is a shared goal among the assessors and the program site to understand how the program is progressing according to evidence-based principles. If administrators or line staff at the study site fear that they will lose funding or look bad if they don’t score well, then the accuracy of the data may be compromised. The best agreement is one in which all parties are interested in getting at the truth.
Indicate what you will need from respondents during your fidelity visit. In addition to the purpose of the assessment, you will need to briefly describe what information you will need, who you will need to speak with, and how long each interview or visit will take to complete. The fidelity visit will be most efficient if the team leader gathers in advance as much as possible of the following information:
- Roster of ACT staff – (roles, full-time equivalents (FTEs))
- Staff vacancies each month for last 12 months (or as long as program has existed, if less than 12 months)
- Number of staff who have left the team over the last two years (or since program started if less than two years old)
- A written description of the team’s admission criteria
- Roster of ACT clients
- Number of clients with dual disorders
- Number of clients admitted to ACT program, per month, for last six months
- How many clients have terminated from the program in the last year, broken down in these categories:
  - Graduated (left because of significant improvement)
  - Left town
  - Closed because they refused services or team cannot find them
  - Deceased
  - Other (explain)
- List of the last 10 clients admitted to psychiatric hospital
- List of the last 10 clients discharged from psychiatric hospital
- Number of clients living in supervised group homes
- Number of clients for whom the ACT team contacts their informal support network (e.g., family member, landlord, etc.) at least once. (Helpful for team leader to have a list of names at the time of interview.)

Note: Reassure the team leader that you will be able to conduct the fidelity assessment even if not all of the above information is available. You should indicate that some information is more critical (e.g., staffing and number of active clients).

Inform the contact person that you will need to observe at least one team meeting during your visit. This is an important factor in determining when you should schedule your assessment visit to the program.
Alert your contact person that you will need to sample 20 charts. It is preferable from a time efficiency standpoint that the charts be drawn beforehand, using a random selection procedure. Obviously, a program can falsify the system by hand picking charts and/or updating them right before the visit. If there is a shared understanding that the goal is to better understand how a program is implementing services, this is less likely to occur.

During Your Fidelity Site Visit

Tailor terminology used in the interview to the site. For example, if the site uses the term “member” for consumer, use that term. If “practitioners” are referred to as clinicians, use that terminology. Every agency has specific job titles for particular staff roles. By adopting the local terminology, the assessor will improve communication.
During the interview, record the names of all relevant programs, the total number of consumers, and the total number of clinicians.
Obtain a random sample of charts:
- For the chart review, select 10 charts at random. One appropriate method is to examine the roster of client names. Divide the number of clients by 10 and round down. Suppose there are 65 clients, then the number would be 6. Starting at an arbitrary name, select every 6th name on the roster.
- If the caseload is known to be stratified, for example if the team uses a level of care system in which every client is classified, and if this level of care is related to intensity of services, then a preferred sampling method is to stratify the sample according the level of care. Example: Suppose the team has 50 Level 1, 30 Level 2, and 20 Level 3 clients. Then select 5 Level 1, 3 level 2, and 2 level 3 clients, using a random sampling strategy.
- In some cases, there may be a lag between when a service is rendered and when it is documented in the client’s chart. When sampling chart data, try to gather data from the most recent time period where documentation is completed in full to get the most accurate representation of services rendered. The most up-to-date time period might be ascertained by asking the team leader, clinicians, or administrative staff. The point is to avoid getting an inaccurate sampling of data where office-based services (e.g., nurses visits or weekly groups) might be charted more quickly than services rendered in the field (e.g., Case manager progress notes).
If discrepancies between sources occur, query the team leader to get a better sense of the program’s performance in a particular area. The most common discrepancy is likely to occur when the Team leader interview gives a more idealistic picture of the team’s functioning than do the chart and observational data. For example, on item S1, the chart review may show that client contact takes place largely in the office; however, the team leader may state that the clinicians spend the majority of their time working in the community. To understand and resolve this discrepancy, the assessor may say something like, “Our chart review shows xx% of client contact is office-based, but you estimate the contact at yy%. What is your interpretation of this difference?”
Before you leave, check for missing data. It is a good idea to check in with the program leader at the end of the visit to review and resolve any discrepancies if possible.

After Your Fidelity Site Visit

If necessary, follow up on any missing data (e.g., by phone calls or email to the program site). This would include a discussion with the team leader about any discrepancies between data sources that arise after the visit has been completed.
Assuming there are two assessors, both should independently rate the fidelity scale. The assessors should then compare their ratings and resolve any disagreements. Come up with a consensus rating.
Tally the item scores and determine which level of implementation was achieved (See Score Sheet).
Send a follow-up letter to the site. In most cases, this letter will include a fidelity report, explaining to the program their scores on the fidelity scale and providing some interpretation of the assessment, highlighting both strengths and weaknesses. The report should be informative, factual, and constructive. The recipients of this report will vary according to the purposes, but would typically include the key administrators involved in the assessment.
If the fidelity assessment is given repeatedly, it is often useful to provide a visual representation of a program’s progress over time by graphing the total fidelity scale using an EXCEL spreadsheet, for example. This graph may be included in the fidelity report

Back to Assertive Community Treatment

TOC | Next