Community Support

Regulating Managed Mental Health Care
A Policy Analysis and Discussion
of the Role of Evaluation

Program Evaluation as a Regulatory Tool

One of the most conspicuous gaps in the legislative regulation of managed care is the area of evaluation. In the laws reviewed, virtually no mention occurs of a need for systematic evaluation of the impact of new regulations, much less is there a mandate for such research or assignment of responsibility for it. As noted, the political impetus behind managed mental health controls typically has come from an upsurge in consumer and provider complaints. Some participants in the legislative process are inclined to continue to rely on unresolved complaints as an indicator of MCO performance (Coyne-McCoy, 1996). Yet this is a very flawed measure of regulatory effectiveness, not least because it neglects frustrated silent consumers, as well as potential changes in "complaint-making behavior" that may be due to factors unrelated to regulation. Federal administrators responsible for overseeing mental health waiver programs have voiced a much stronger interest in managed care evaluation research, yet focusing on but one subscriber population, the publicly insured, within the managed care enrollment group.

However, the reasons are compelling for linking new regulatory initiatives with a solid program of evaluation research. First, as the number and variety of government regulations in U.S. society has risen over the past few decades, so too have the criticisms from industry groups about excessive regulatory burden. To the extent regulatory proponents seek to argue that the social benefits of regulation outweigh its costs, both these costs and benefits must be identified, carefully measured, and tracked over time (Congressional Quarterly, Inc., 1982). This research unavoidably will also serve to clarify the trade-offs posed by regulatory interventions that cannot pass a cost/benefit calculation on strictly economic grounds, but which reflect other social values (see, e.g., the report on the cost of restricting selective provider contracting by Sheils, Stapleton, and Haught, 1995). "Like any other intervention, the effects of regulation must be measurable," write Brennan and Berwick (1996, p. 22). "Most regulatory innovations are simply hypotheses; outcomes must be specified and followed." Second, we have noted that public officials possess significant discretion in determining the character of any regulatory activity through the many details of coercion left in their control. According to a substantial public administration literature, effective policy management is a process of "error correction" in which program authorities must seek constant feedback about the impacts of their actions (see, e.g., Bardach, 1977; Levin and Ferman, 1985). These data, in turn, must enter into an ongoing process of administrative adjustment, which, in the regulatory context, might mean altering the balance or methods of coercion to achieve improved compliance. All of this underscores the need for evaluation studies that can provide regulatory administrators with a source of unbiased feedback. Third, the basic framework of regulatory legislation ought also to remain under review and subject to redesign depending on evaluation findings. The potential "substitutability" of policy instruments has been discussed, as well as the likelihood that policymakers will begin regulation of a new area with weaker instruments and then "move up the scale" (Howlett and Ramesh, 1995, p. 159). For legislators to judge whether scaling-up is necessary, they will have to draw on empirical evidence geared to the kinds of policy decisions facing them.

Implicit in these statements is a broad understanding of evaluation research. Experimental studies, which use the most scientifically rigorous designs, have a distinctive value, especially in the traditions of clinical outcomes investigation and cost-benefit analysis. However, the creation of control groups for research purposes is often problematic in public policymaking. Further, experimental-design studies may tell us little about the contribution of specific program components within a complex policy change (Rochefort and Dill, 1994). Studies of this kind also tend to be exceedingly expensive and slow.

In short, administrators and public policymakers alike live in "a rough and tumble world" that leads them to look for programmatic data beyond what classic evaluation studies can provide (Howlett and Ramesh, 1995). Frequently they end up giving heavy emphasis to anecdotal evidence that "puts a human face" on an issue (Scallet, 1996b). Evaluators can help to make such information more reliable by ensuring that it is at least representative. Moreover, there are a variety of quantitative and qualitative research studies that can be done with policy formulation purposes expressly in mind. Included are consultations with members of the "policy susbsystem" (Howlett and Ramesh, 1996, p. 174), meaning, in the area of managed care, regulators, managed care administrators, service providers, patients, and advocacy groups. Large-scale surveys may or may not be a part of this process. In-depth case studies of a particular organization or period of program implementation can be valuable when new regulations are introduced. Equally pertinent, Scallet (1996b) refers to "commando studies" undertaken to gather information on particular narrow questions that have gained administrative or legislative attention. An example for managed care regulation might be the changing situation of a specific population subgroup such as persons with severe mental illnesses.

Beyond these generalities, to attempt to go into the details of conducting managed care research would be beyond the scope of this report. A growing body of literature can be found on the subject. For example, Milstein et al. (1994) discuss four primary techniques for evaluating psychiatric case management, including on-site interviews and inspection; analysis of claims data; examination of provider medical records; and patient satisfaction surveys. Reviewing several studies that have been completed using either an external (policy-relevant) or internal (organizational management) perspective, Wells et al. (1995) offer useful insights into data, methodology, and research sponsorship issues.

When Massachusetts mounted the nation's first managed mental health carve-out under Medicaid, federal administrators required an independent program review. Although inconclusive in certain respects, the evaluation report generated by this requirement (Callahan et al., 1995), as well as a number of other program analyses (Beinecke, Goodman, and Rivera, 1995; Stroup and Dorwart, 1995; Leadholm and Kerzner, 1994), effectively lay the groundwork for similar future investigations in Massachusetts and elsewhere. A main conclusion of this work for evaluation researchers is the importance of looking at management process and the functioning of service networks under managed care, in addition to conventional cost and utilization measures.

In terms of future legislation, the states are well advised to consider establishing consolidated data centers that can facilitate the evaluation research process on a continual basis. This action is consistent with the recommendations of both researchers (e.g., Wells et al., 1995) and mental health advocates who seek a clearly defined state role in overseeing managed care developments (e.g., Bazelon Center, 1995). Such a center would be a repository of data sets from different MCO providers, to include some number of common elements specified by public officials. According to Brennan and Berwick (1996), forty-two states already "have either implemented or plan to implement some sort of system for collecting and disseminating health care data" (p. 204). As this movement proceeds, it will be important to incorporate more concerns that are distinctive to managed mental health care.

The upshot is that program evaluation represents another crucial regulatory tool for legislators and government administrators in their drive to monitor managed care. And like other tools, it can be used with more or less intrusiveness and concentrated on different aspects of the process of service delivery. Referring to the various types of evaluation research alluded to above, it is possible to specify at least five distinct contributions that evaluation can make to the regulatory process:

Informing the establishment of regulatory standards and procedures. Regulators face the daunting task of converting general legislative intent into specific operational rules. Especially in the areas related to data collection, standardized information disclosure, and clinical and service indicators, evaluators can aid regulators in recognizing what data it is possible to collect, in what formats, and for what purposes.
Developing measures of regulatory compliance. The empirical orientation of evaluation research is well-suited to the regulatory task of assessing compliance with new legislative and administrative requirements. Many health plan activities—for example, in regard to quality assurance, access to services, grievance procedures, utilization review, and coordination of generalist and specialist care—are not only amenable to systematic quantitative examination, they have already been studied in this way by evaluation researchers in various health service contexts. This expertise could be brought into the regulatory arena to improve effective monitoring.
Collecting implementation feedback. Ongoing evaluation of the routines of regulatory administration is necessary to keep bureaucratic strategy true to enabling legislation, to identify faulty approaches, and to adapt regulation to changing circumstances in the managed care industry. This should include continuing contact with all the major stake-holders who were involved in law-making, as well as the managers, providers, and consumers who are affected by regulatory changes.
Contributing to information disclosure. By requiring all MCOs to assume responsibility for, or submit to, a formal performance evaluation by independent researchers, regulators can document industry trends on a long-term, comprehensive basis. Such research, which employs a different level of data and methodological approach than health plan managers use for internal quality assurance, amounts to another kind of information disclosure crucial to the policymaking process.
Assessing regulatory impact. Finally, a formal independent evaluation of the regulatory regime is also in order to assess impacts on the managed care industry and service delivery, including overarching issues of efficacy and need-based care.

A natural tension arises between these evaluation research desiderata and the maintenance of patient confidentiality. The only way this tension can be managed is by bringing the two concerns into joint consideration and identifying the trade-offs involved. The situation to be avoided is one where regulated interests are allowed to seize on the patient confidentiality theme merely as a shield against public scrutiny that is legitimate and ultimately in the best interests of plan subscribers. At the same time, calls for data collection for the sake of evaluation research must be tempered in light of patients' appropriate desires for privacy and a standard of reasonableness for the burden being placed on MCOs.

A regular stream of evaluation reports making intelligent use of qualitative and quantitative data can go far in helping to document managed care performance, link together private and public system issues, and assess regulatory costs and benefits. Lacking this touchstone, future public decisionmakers undertake rule-making on this issue in an environment where interest-group politics becomes that much more likely to overshadow policy expertise.

Table of Contents | Previous | Next