 |
This Web site is a component of the SAMHSA Health Information Network. |
|
2001 Annual Report to Congress on the Evaluation
of the Comprehensive Community Mental Health
Services for Children and Their Families Program
Home | Summary | Table of Contents | Figures | Tables | I | II | III | IV | V | VI | VII | VIII | Appendix | Appendix B
APPENDIX B
Measures Used in the National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program
THE CHILD AND ADOLESCENT FUNCTIONAL ASSESSMENT SCALE (CAFAS)
The CAFAS (Hodges, 1990b) yields scores for each of eight psychosocial subscales (School/Work role performance, Home role performance, Community role performance, Behavior Toward Others, Self-Harmful Behavior, Moods/Emotions, Substance Use, and Thinking) and generates a total score indicating the child's highest level of functioning during the past 6 months. For each of these psychosocial subscales, scores are assigned based on information available about the child and how this information aligns with the scoring criteria of the CAFAS.
The CAFAS originally was designed to be completed by a clinician who obtained information from multiple informants such as the child, caregivers, schools, and official records. For the purposes of this evaluation, information to score the CAFAS may also be obtained through a structured interview with the caregiver. Completed CAFAS forms are rated by clinicians or interviewers who have been established as reliable raters by a standardized reliability assessment. The rater determines the level of a youth's functional impairment by reviewing specific identifiers across levels of severity in each domain and assigns the highest level of impairment based on available information. A score of 0 indicates minimal or no impairment, 10, mild impairment, 20, moderate impairment, and 30, severe impairment in a given domain (Hodges, 1990a). The eight psychosocial subscales scores are then summed to produce a total score ranging from 0 to 240. When combining the eight subscale scores to generate a total functioning score, total scores of 40 or below indicate minimal impairment; scores from 50 to 90 indicate moderate impairment, while scores from 100 to 130 indicate marked impairment and those 140 or higher indicate severe impairment. In addition, scores above the clinical cutoff of 40 are considered to indicate impairment in social functioning at a level requiring clinical care.
The CAFAS is intended to be administered by any rater who has met the standardized rater reliability criteria, and the scale has been used successfully by a variety of raters, including mental health intake workers, direct service providers, paraprofessionals, and graduate students, resulting in good interrater reliability (Hodges, 1990a). Adequate test-retest reliability has been reported by Cross and McDonald (1995). In addition, construct, concurrent, and discriminant validity have been demonstrated (Hodges, Lambert, & Summerfelt, 1994; Hodges & Wong, 1996).
THE CHILD BEHAVIOR CHECKLIST (CBCL) AND THE YOUTH SELF-REPORT (YSR)
The CBCL (Achenbach, 1991a) has been identified as the most reliable and valid parent report measure currently available for assessing children's emotional and behavioral problems (Reitman, Hummel, Franz, & Gross, 1998). Information concerning the child's behavior during the previous 6 months is obtained directly from the primary caregiver. The CBCL consists of 118 problem behavior items categorized as internalizing or externalizing behaviors. Internalizing behaviors include sadness, depression, and anxiety. Externalizing behaviors include opposition, aggressiveness, and hyperactivity. Parents rate each item on a 3-point scale: 0 (not true [as far as you know]), 1 (somewhat or sometimes true), and 2 (very true or often true). In addition, 20 social competence items are used to measure school performance, functioning at home, peer relationships, and social involvement. These items are rated on a 4-point scale with varying response sets (don't know to more than average, don't know to above average, and don't know to more active).
Scoring procedures using Achenbach's Cross-Informant Scoring Program (Achenbach, 1996) generate standardized scores for eight syndrome subscales (Withdrawn, Anxious/Depressed, Somatic Complaints, Social Problems, Attention Problems, Delinquent Behavior, Thought Problems, Aggressive Behavior), two broadband syndrome scores (Internalizing behavior and Externalizing behavior), a Total Problems score, a total competence score, and three competence subscales (Activities, Social, and School). Internalizing behavior scores are derived from the subscales assessing withdrawal, somatic complaints, and anxious/depressed behaviors, and Externalizing behavior scores are derived from the delinquent and aggressive subscales. Standardized T-scores for each syndrome subscale range from 50 to 100, Internalizing behavior T-scores range from 31 to 100, Externalizing behavior T-scores range from 30 to 100, and Total Problems T-scores range from 23 to 100. For each syndrome subscale, standardized T-scores above 70 are considered in the clinical range (i.e., indicating emotional and behavioral problems above the 90th percentile in the population of children as a whole), although T-scores ranging from 67 to 70 are considered to be in the borderline range. Internalizing, Externalizing, and Total Problems scale T-scores are considered in the clinical range if they are above 63, while scores from 60 to 63 are borderline. Scores in the clinical range indicate a need for clinical care.
Total competence scores are calculated from the 20 competence items. The competence in activities subscale is restricted to items related to hobbies, games, chores, and jobs; social competence to participation in organizations, clubs, teams or groups, and interaction with friends, siblings, parents, or others; and school competence to items pertaining to school performance. Total competence T-scores range from 10 to 80, with T-scores in the clinical range falling below 37 and those considered borderline ranging from 37 to 40. Subscale scores range from 20 to 55, with those below 30 falling in the clinical range and those between 30 and 33 considered borderline.
Youth assess their own emotional and behavioral problems and competencies with the YSR (Achenbach, 1991b), which is similar in format to the CBCL. The same scoring criteria, range of values, and clinical ranges apply to the YSR. The YSR consists of 112 items and does not include a School competence subscale. The instruments have been nationally normed on a proportionally representative sample of children across income and racial-ethnic groups. Achenbach (1991a, 1991b) has reported high internal consistency, test-retest reliability, construct validity, and criterion-related validity for both the CBCL and the YSR.
THE BEHAVIORAL AND EMOTIONAL RATING SCALE (BERS)
For the purposes of the national evaluation, the Behavioral and Emotional Rating Scale (BERS; Epstein & Sharma, 1998) is employed to measure child strengths. This "is a standardized, norm-referenced scale designed to assess the behavioral and emotional strengths of children instead of their problems and deficits (p. 1)."
The BERS is designed to be completed by caregivers or professionals (e.g., clinicians or teachers) to rate the behaviors of children aged 5 to 18. It has 52 statements about a child's behaviors and emotions that are categorized into the following five domains: Interpersonal Strength, Family Involvement, Intrapersonal Strength, School Functioning, and Affective Strength. Each item is rated on the following scale: 0 (not at all like the child), 1 (not much like the child), 2 (like the child), and 3 (very much like the child).
Standard scores for the five domains and an overall strength quotient are derived from a nationally representative normative sample to indicate the level of a child's strengths. A domain-specific score of 10 and an overall strength quotient of 100 is considered average (50 percent of all children will have scores at or above this standard score).
The BERS has demonstrated test-retest reliability, interrater reliability, and internal consistency (Epstein, Cullinan, Harniss, & Ryser, 1999). Test-retest reliability coefficients for the BERS subscales ranged from .85 to .99, with a 10-day interval between the two ratings. Interrater reliability was tested using a sample of 96 students with emotional and behavioral disorders rated by their special education teachers. Cronbach's alpha coefficients for the scales were .83 or above.
THE RELIABLE CHANGE INDEX
Because numeric change may vary in magnitude and implications for actual behavioral change are often difficult to interpret, we provide a quantitative indicator of clinical change for these measures: the reliable change index (RCI; Jacobson & Traux, 1991; Jacobson, Roberts, Berns, & McGlinchey, 1999; Speer & Greenbaum, 1995). The analysis of clinical change focuses on groups of children and families from the outcomes study sample who had complete data at two evaluation points. This statistic compares a child's scores at two different points in time and indicates whether a change in scores shows clinically significant improvement, stability, or deterioration.
THE FAMILY RESOURCE SCALE (FRS)
The FRS (Dunst & Leet, 1985) includes 30 questions that measure the adequacy of a variety of resources needed by households with young children. Questions address the adequacy of resources in a number of areas, including growth and support, health and necessity, nutrition and protection, physical shelter, intrafamily support, communication and employment, childcare, and income. Each questions is answered using the following scale: 1 (not at all adequate), 2 (seldom adequate), 3 (sometimes adequate), 4 (usually adequate), and 5 (almost always adequate).
The reliability and validity information on the FRS comes from studies of two populations: children with mental retardation or other handicaps, or who were at risk for other developmental challenges (Dunst & Leet, 1987) and children with emotional and behavioral disturbance (Heflinger et al., 1998). In both samples, good internal consistency among all the items was demonstrated, with alpha coefficients ranging from .92 to .95 (Dunst & Leet, 1987; Heflinger et al., 1998).
THE CAREGIVER STRAIN QUESTIONNAIRE (CGSQ)
The Caregiver Strain Questionnaire (CGSQ; Brannan et al., 1998), designed for use with caregivers of children with severe emotional and behavioral disturbances, assesses the extent to which caregivers experienced additional difficulties, strains, and other negative effects as the result of their caregiving responsibilities in the past 6 months. The instrument assesses global strain (i.e., the total impact on the family), objective strain (i.e., the extent to which observable negative events or consequences related to the child's disorder have been a problem for the family), subjective externalizing strain (i.e., negative feelings about the child such as anger, resentment, or embarrassment), and subjective internalizing strain (i.e., negative feelings that the caregiver experiences such as worry, guilt, and fatigue). This measure includes 21 items regarding the level of disruption, strain, or other form of stress resulting from a child's emotional or behavioral problems. Each item is rated on the following scale: 1 (not at all), 2 (a little), 3 (somewhat), 4 (quite a bit), and 5 (very much). Higher scores on each subscale indicate more strain on the caregivers.
The CGSQ demonstrated good reliability and validity in previous research (e.g., Brannan et al., 1998; Heflinger et al., 1998). Confirmatory factor analysis findings from previous research with the CGSQ have supported the existence of three related dimensions of caregiver strain (Brannan et al., 1998). The three CGSQ subscales demonstrated adequate internal consistency, with alpha coefficients ranging from .73 to .91 (Heflinger et al., 1998). In addition, the CGSQ subscales were found to correlate with measures of family functioning and caregiver distress in expected ways, providing evidence of construct validity (Brannan et al., 1998). The predictive validity of the CGSQ is supported by findings that the CGSQ was a better predictor of service utilization patterns than measures of the child's clinical and functional status (Foster, Saunders, & Summerfelt, in press; Lambert, Brannan, Heflinger, Breda, & Bickman, 1998).
THE FAMILY ASSESSMENT DEVICE (FAD)
The Family Assessment Device (FAD) and the Family Assessment Device for Youth (FAD-Y) (Epstein et al., 1983) are 60-item self-report measures of family functioning designed to measure how families interact, communicate, and work together. The instruments assess overall family health and functioning along six dimensions: Problem Solving, Communication, Family Roles, Affective Responsiveness (the extent to which family members experience appropriate affect with each other in a variety of situations), Affective Involvement (the degree to which family members are interested in and place value on each other's activities and concerns), and Behavior Control. There is also a General Functioning scale that assesses overall family functioning. Each item is rated on the following scale: 1 (strongly disagree), 2 (disagree), 3 (agree), and 4 (strongly agree).
The measurement quality of the FAD has been studied over the years by a variety of researchers. Internal consistency has been good across many studies and samples, with alphas ranging from .71 to .92 across the seven subscales (Byles, Byrne, Boyle, & Offord, 1988; Heflinger et al., 1998; Perosa & Perosa,1990). Test-retest reliability after 7 days was also found to be good, with correlations for most subscales above .70 (Kabacoff, Miller, Bishop, Epstein, & Keitner, 1990; Miller, Epstein, Bishop, & Keitner, 1985).
Construct validity has been supported in several studies, with the FAD correlating in the expected directions with other measures of family functioning, family cohesion, marital satisfaction, and other family factors (Byles et al., 1988; Epstein et al., 1983; Fristad, 1989; Heflinger, et al., 1998; Miller et al., 1985). The FAD demonstrated good predictive validity by distinguishing families in clinical treatment from those who were not (e.g., Epstein et al., 1983; Fristad, 1989).
MULTI-SECTOR SERVICE CONTACTS QUESTIONNAIRE (MSSC)
This self-report questionnaire was designed specifically for the national evaluation to assess the types and frequencies of services that children and families receive across different settings and child-serving sectors as well as caregiver perceptions about whether services met their needs. The MSSC provides standard descriptions for 22 types of services, with names of services and service settings customized for each site. Services examined include assessment, case management, individual therapy, group therapy, family therapy, medication treatment or monitoring, crisis stabilization, family preservation, day treatment, residential therapeutic camp or wilderness program, inpatient hospitalization, residential treatment center services, therapeutic group home services, therapeutic foster care, behavioral or therapeutic aide services, independent living services, transition services, caregiver or family support, recreational activities, transportation, respite care, and flexible funds.
The MSSC first asks respondents whether families received services in the last 6 months and, if so, for how long and from which major child-serving agencies. If no services were received, caregivers are asked to document the reason no services were received and to record the date of the last service.
THE FAMILY AND YOUTH SATISFACTION QUESTIONNAIRES (FSQ-A and YSQ-A)
The FSQ-A is an abbreviated and revised version of the Family Satisfaction Questionnaire developed through the evaluation of grant communities funded in 1993-94 (Brunk, Santiago, Ewell, & Watts, 1997).
The first part of the FSQ-A contains seven items that assess the caregiver's satisfaction in the past 6 months with services as a whole, the child's progress, and the cultural competence and family-focused nature of services. Respondents report their satisfaction on a 5-point scale: 1 (very dissatisfied), 2 (dissatisfied), 3 (neutral), 4 (satisfied), and 5 (very satisfied).
The second part of the FSQ-A contains seven items that assess whether the services the family received improved the caregiver's (or other family member's) ability to work for pay, and quantifies the impact in terms of days worked. For example, "Have the services (child's name) or your family received helped you miss fewer days or fewer hours of work?" is an item that assesses the impact of services on the caregiver's work life and the amount of work missed.
The YSQ-A also was developed specifically for this evaluation and contains one screening item and eight satisfaction-related items that mirror those in the family satisfaction questionnaire. Like the FSQ-A, the YSQ-A assesses satisfaction in the past 6 months with services as a whole, the youth's progress, and the cultural competence and individualization of services received. Respondents report their satisfaction on a 5-point scale: 1 (very dissatisfied), 2 (dissatisfied), 3 (neutral), 4 (satisfied), and 5 (very satisfied). The last item, an open-ended question, asks whether services were helpful and, if so, what was the most helpful thing about the services received.
The measures from which the satisfaction items were extracted have demonstrated internal consistency in their original forms (Brunk et al., 1997). The items collected for the abbreviated versions also demonstrated internal consistency. Based on reliability analysis of the evaluation of grant communities funded in 1997-98, Cronbach's alpha coefficient for the seven family satisfaction rating items was .88. Cronbach's alpha coefficient for the seven youth satisfaction rating items was .89.
Table of Contents | Previous
Back to Top
|
|
