This Web site is a component of the SAMHSA Health Information Network

Enter Keywords Choose Website This Site Mental Health at HHS All Mental Health

Online Publications Order Publications National Library of Medicine National Academies Press Publications Homepage

printer friendly page e-mail this page bookmark this page shopping cart  current or new account

This Web site is a component of the SAMHSA Health Information Network.

1999 Annual Report to Congress on the Evaluation
of the Comprehensive Community Mental Health
Services for Children and Their Families Program

Home | Summary | Table of Contents | Figures | Tables | I | II | III | IV | V | VI | VII | VIII | IX | Appendix

VII. SPECIAL STUDIES

INTRODUCTION

Even though the evaluation is designed to capture a vast amount of standardized information on multiple levels, emerging issues of importance must also be addressed. These issues cannot always be pre-determined; however, the evaluation must incorporate a method for inquiring into such issues. Special studies have been built into the evaluation to address this need. Two special studies were conducted during Phase I of the evaluation. The first, called a family scan, examined how extensively families were involved in evaluating systems of care. This study delineated what kinds of roles family members have performed with respect to the national evaluation and tapped the perspectives of three major groups: families, project directors, and evaluators at the local grant communities. The second special study assessed the impact of managed care initiatives on the development and operation of local systems of care. These two studies are presented below.

FINDINGS FROM A FAMILY SCAN STUDY

Family involvement has become a capstone of the system-of-care philosophy and implementation. Families participate extensively as collaborators, advisors, and advocates, and run family-based organizations. One area that has not experienced extensive family involvement, however, is in the research and evaluation of systems of care. Collaboration between family members and researchers is relatively new, and little is known about the extent to which families are engaged in these activities.

The purpose of this study was to gain a better understanding of how families are involved in evaluating systems of care and the degree of congruence in the perceptions of project directors, evaluators, and family members about

family involvement. While the field of participatory research is gaining prominence, most studies reporting on participatory research describe the researcher's perspective about consumer involvement. Very little is known about the consumer's perspective of this involvement.

How are families involved in evaluating systems of care?

A design team consisting of a family advocate, a researcher, and a project director developed a survey instrument for three categories of respondents: family members, project directors, and site evaluators. The survey addressed several areas: (a) initiation and design of the evaluation, (b) implementation of the evaluation, (c) data analysis and utilization, and (d) compensation of families.

To enrich the results of the survey findings, in-depth telephone calls were made to grant communities ranking highest on a combined score of the responses to key survey questions. The purpose of the calls was to gain a better understanding of the emerging roles of families in evaluating systems of care, how these roles evolved, and what impact family involvement had on the system and the evaluation itself.

The survey was mailed to the three groups of respondents in the 22 grant communities funded in FY 1993–94 and the nine grant communities funded in FY 1997. Because the nine grant communities funded in FY 1997 had only begun services, responses from these grant communities were eliminated from the analyses. After follow-up telephone calls to enhance participation, the response rate was approximately 90 percent. Responses were evenly distributed among the three groups, with family members having slightly more responders than the other two groups. Follow-up conference calls were made with three grant communities to obtain further information about family involvement in evaluation.

Results of the study indicate that family members are involved in a wide range of evaluation activities at the grant communities. Levels of involvement across grant communities varied depending upon the particular category of

activity. For example, more than 80 percent of all respondents indicated that family members were involved in the modification of instruments to make them more relevant to the culture, language, or environment of the grant community. Only about 50 percent of the respondents confirmed that family members were involved in collecting data for the evaluation

Families participate in a wide range of evaluation activities, from instrument development to data collection.

and far fewer were involved in data entry (18.6 percent). Only moderate agreement was obtained across family members, project directors, and evaluators about the level of involvement in evaluation activities. Agreement was highest between project directors and evaluators and somewhat lower between family members and the other participants. The most significant discrepancies were found for introducing the study to others and reviewing evaluation findings. Family members reported a much greater involvement with other family members in introducing the study (62.5 percent) than project directors and evaluators (37.5 percent and 36.4 percent, respectively), but they reported only a 50 percent involvement in reviewing evaluation findings as opposed to the 82.6 percent reported by directors and 81.8 percent by evaluators.

The ‘Ohana Project in Hawaii, the first statewide project in Vermont, and Wings in Maine were selected for follow-up interviews because responses from all three perspectives indicated extensive family involvement in site evaluation. A conference call was held with the three respondents from each grant community so that they could interact together to discuss the questions and formulate a response. Results of these discussions indicated that family-researcher collaborations are set in motion intentionally by a key person or event. They depend upon trusting relationships. They are built on a common set of values that are operationalized in daily practice. Finally, these collaborations are perceived to improve the quality of data collected.

SUMMARY OF KEY FINDINGS

The family scan study was an initial attempt to understand family members' involvement in evaluation. Multiple perspectives of consumer involvement in research or the congruence of these perspectives have not been previously investigated. Findings show that family members are involved in a wide range of evaluation activities. However, the level of this involvement was differentially perceived by the three groups of stakeholders. This study provides a baseline of data from Phase I of the evaluation that can be used to understand directions to pursue in the future. These directions include the development and implementation of a curriculum to train family members in evaluation and the initiation of special research and evaluation projects by family members and family organizations.

FINDINGS FROM A STUDY ON MANAGED CARE

BACKGROUND AND METHODS

The communities receiving grant support from CMHS are endeavoring to develop comprehensive, community-based systems of care for children and adolescents with serious emotional disorders and their families. System-building activities, however, are occurring in the context of other significant environmental forces and reforms that inevitably affect children and families as well as service systems. One such contextual influence is the rapid adoption and application of managed care technologies to the delivery of mental health and substance abuse (i.e., behavioral health) services, with goals that often include improving the efficiency of service delivery and containing costs. A special study was designed to assess how managed care reforms affect children and adolescents with emotional disorders, their families, and the systems of care that serve them.

Telephone interviews and site visits were conducted to six CMHS-funded grant communities to obtain the assessments, perceptions, and impressions of multiple key stakeholders regarding the effects of managed care on systems of care. A basic consideration for site selection was the requirement that the grant community was affected by managed care; other criteria considered during the site selection process included stage of implementation of managed care, characteristics of managed care reforms, approach to system-of-care development, and geographic diversity. Based on this process, three grant communities were selected for telephone interviews (Lane County, Oregon; Las Cruces, New Mexico; and the State of Rhode Island) and three grant communities were selected for in-depth site visits (Milwaukee, Wisconsin; Solano County, California; and San Mateo County, California). At the time of the telephone interviews and site visits, Las Cruces was planning for managed care, Lane County had recently implemented managed care, and the other grant communities had implemented managed care for a minimum of a year and had a substantial base of experience.

The study examined the impact of managed care reforms on system-of-care values and principles, and their effects on other system features such as providers, financing, and accountability also are discussed.

FINDINGS

Las Cruces, New Mexico, is an example of a local grant community that was anticipating and planning for managed care implementation at the time of the telephone interviews, since the State was phasing in Medicaid managed care in various regions of the State sequentially. Stakeholders reported considerable activity in the community in order to prepare and "position" for managed care implementation, such as forming provider networks and seeking training in various aspects of managed care from communities that already had experienced managed care reforms. Respondents from Las Cruces conveyed many concerns about the potential impact of managed care reforms on the grant community. The speed with which the managed care plan was developed in New Mexico and the reported lack of opportunity for stakeholder involvement in the planning process were overriding issues for these respondents. They anticipated significant problems with the behavioral health system as a result, particularly for children and adolescents. Another factor contributing to stakeholders' concerns about managed care was their perception that the overriding goal of managed care reforms in New Mexico is cost savings. With the emphasis on cost savings, respondents feared that the reduction in available resources would devastate the local system of care and its component services.

In addition to expressing these general concerns about the reform planning process and the future of the local system of care, the Las Cruces respondents raised a range of specific issues. These included concerns that physical health would dominate and that only a small percentage of health care resources would be allocated to behavioral health, that managed care organizations would have little knowledge about the needs of children with emotional disorders, that resources would be insufficient to expand behavioral health service capacity for children, that services would be inadequate for children with serious disorders, that families would be "disenfranchised," and that small and nontraditional providers would be in jeopardy in a managed care environment.

Unlike Las Cruces, the other five grant communities included in this study had already experienced managed care reforms, and stakeholders in these grant communities could convey their perceptions of the actual impact of managed care reforms in a number of domains.

In Milwaukee, San Mateo County, and Lane County, individuals and agencies involved with the system of care played a significant role in shaping the managed care system and its implementation. In these three grant communities, system-of-care services, principles, and approaches are more integrated with managed care reforms.

In Rhode Island and Solano County, where planning (whether it occurred primarily at the State or local level) incorporated little input from those with expertise in behavioral health services for children and adolescents and their families (and in systems of care), there is little integration of managed care and systems of care and little incorporation of the system-of-care tenets. Thus, Milwaukee, San Mateo County, and Lane County were able to use managed

Involving system-of-care participants in managed care reform can lead to advancements in system-of-care development.

care reforms as a vehicle to further advance the development of systems of care, while the other grant communities had less opportunity to do so. Integration ranges from the system of care being fully incorporated in the managed care system as an expanded benefit for children with serious disorders in Lane County, to being a specialized carve-out for children with serious disorders in Milwaukee, to being considered a tier in the overall system for children with serious disorders in San Mateo County. In both Rhode Island and Solano County, the systems of care are not integrated with the managed care systems and essentially function separately.

Across these five grant communities, respondents noted a range of beneficial effects of managed care reforms, all of which are consistent with system-of-care goals, such as improved interagency collaboration, decreased use of

hospital and residential treatment services, an expanded array of services, an expanded array of providers, improved access to mental health care, and increased clarity and consistency in clinical decisionmaking. However, respondents from the grant communities also noted serious problems that

Managed care reforms had both positive and negative effects on systems of care.

they felt were caused by or associated with managed care reforms. These included the fragmentation in two of the grant communities created by separation of acute and extended care services, a lack of sufficient service capacity for behavioral health services for children and adolescents, insufficient education of stakeholders about the new managed care systems, and generally negative attitudes toward managed care that create resistance and negative expectations.

Although the number of grant communities in the study is small, the study has yielded important insights about the effects of managed care reforms and their compatibility with system-of-care goals and approaches. Taken together, analysis of the six grant communities identified a number of variables that appear to influence how well managed care works with systems of care:

• Whether an ongoing system development initiative and commitment to the system-of-care philosophy pre-existed managed care.
• Whether individuals familiar with the system-of-care philosophy and approach participated in the design and implementation of the managed care system (at State and local levels).
• Whether extended care services for children and adolescents with serious disorders are integrated into managed care systems, or, if not, whether clear criteria exist for movement of children from managed care systems to systems of care and for coordination between the two.
• Whether systems of care are incorporated into managed care systems as a level of care for children with serious and complex needs (i.e., the high utilizer population) when the managed care system is responsible for extended care.
• Whether special provisions and risk adjustment mechanisms are incorporated into managed care systems for children and adolescents with serious and complex disorders.
• Whether the managed care system benefit design includes a broad array of community-based service options.
• Whether the managed care system incorporates system-of-care principles as requirements and guidelines, such as individualized and flexible care, family involvement, cultural competence, service coordination, and interagency service planning.
• Whether managed care organizations are familiar with the system-of-care philosophy and approach and are knowledgeable about the needs of children and adolescents with emotional disorders and their families.
• Whether prior authorization and other utilization control mechanisms impinge on the ability to provide appropriate types and levels of care or excessively restrict flexibility in service delivery.
• Whether financial resources within the system are sufficient to provide the range and level of services needed by children and adolescents with emotional disorders and to prevent underservice, particularly of those with serious disorders.
• Whether risk adjustment mechanisms are incorporated into managed care systems to prevent incentives for underserving children and adolescents with more serious and complex disorders.
• Whether incentives maximize access to treatment in the least restrictive appropriate setting.
• Whether provisions are incorporated into the managed care system to encourage, and to allow billing for, service coordination and interagency service planning activities.
• Whether a broad array of providers is included in the managed care provider network.

SUMMARY OF KEY FINDINGS

The adoption of managed care technologies to finance and deliver behavioral health services is continuing to spread rapidly. Additional States and counties are in the process of planning and implementing managed care reforms, and those with existing managed care systems are in the ongoing process of problem-solving and incorporating refinements. This analysis shows that managed care reforms can be designed and implemented in a way that takes into consideration the special needs and system requirements of children and adolescents with emotional disorders and their families. Stakeholders in some of the grant communities have "harnessed" managed care by successfully taking the opportunity presented by the reform to advance the development of systems of care. However, in other grant communities, the managed care reforms have been implemented in a manner that is less attuned to the special needs of children with emotional disorders, leading to a host of difficulties that ultimately must be corrected.

There are examples within this group of grant communities that show that the system-of-care philosophy and approaches (or aspects of it) can, in fact, be maintained in a managed care environment and, under the right circumstances, can even be used to shape managed care systems. There are also, however, clear indications that without specific attention to and planning for children and adolescents with emotional disorders, many system-of-care features can be threatened with managed care reforms. Despite these threats, there is evidence of efforts to stay true to the system-of-care philosophy even in the context of major system changes such as managed care reforms.

SYNTHESIS

The two special studies explored critical issues that have become increasingly important since the initial design of the Phase I evaluation in 1994. Since that time, systems of care have embraced an expanding role for families. The family scan provides a baseline of information about this role vis-à-vis evaluation. Findings from the study can be used to target additional efforts as this role continues to evolve. The study on managed care also provides a baseline of information about the effects of managed care on systems of care and suggests areas for attention and further study. Taken together, the subjects of these special studies are illustrative of the importance of having a component of the evaluation that can respond to emerging issues.

Table of Contents | Previous | Next

Back to Top