Summary of the Second Meeting
Break-out Session Wrap-ups
TA Center Evaluation
The session began with discussion of the best ways to evaluate the Technical Assistance (TA) Centers. Questions centered around topics such as how well the centers are doing and what effect they are having on the mental health field.
One participant suggested that it would be useful to review and aggregate all TA Center evaluation plans as a tool for analyzing the techniques being used by the centers. However, many of the approaches taken by evaluation plans developed by the TA Centers at the time of funding are outdated. One possible option for correcting this problem is to revise evaluation plans at the time of continuation funding. But this may not be feasible, given the vast changes in the mental health services system over the past several years.
Properly evaluating the TA Centers requires agreeing on common goals. For example, is systems change the ultimate outcome measure? If so, are there more proximal outcomes to measure in the process of creating systems change? Is the welfare of people with mental illness the barometer by which outcomes should be measured? Are CMHS and the TA Centers prepared to define which criteria lead more to welfare than others?
Defining the outcomes to be measured and how they are interpreted vary based on the evaluator's perspective. Managed care companies, for instance, use hospitalization rates, which are lower use rates, as a primary outcome measure. Yet consumers are likely to argue that recognizing the need for inpatient treatment in a timely fashion may be more in line with health and recovery.
One participant was uneasy with the idea of selecting systems change as a common goal, asserting that too many variables, such as the political and fiscal environments, affect the pace and degree to which systems can and will change. It was suggested that a better approach may be to measure the impact of TA Centers on the presence or absence of mental illness, or the drop in rates of a particular indicator. Another participant offered the practical approach of triangulating technical assistance, dissemination, and feedback based on changes that result from technical assistance.
One example of this approach is to determine the availability or lack of supported housing units in a community over the course of a year following technical assistance. Another possible approach is to measure knowledge dissemination and improvement in people's understanding of certain topics. In short, small, discrete measures, rather than global outcomes of systems change, appear to be a realistic approach to evaluating the effectiveness of TA Center programs.
One participant noted that, although it may not be recognized, TA Centers have positive effects on the system. Certain legislative advances, for example, illustrate the success of TA Centers in pushing the limits for systems change.
One participant cited the importance of understanding the logic behind the program or services when evaluating a TA Center. This requires looking at what variables affect the success or failure of the program. For example, political realities must be made explicit if the goal of the TA Centers is systems change.
The discussion again shifted to the difficulty of choosing appropriate criteria for measuring outcomes. To suggest using surveys and evaluation forms is an obvious approach that quickly becomes a conundrum when looked at closely. Including goals that are mandatory in Federal grant applications, such as fostering collaboration between consumers and providers, touches only the overarching mission.
The overall goal of the evaluation will determine how to proceed. For example, if internal quality improvement is the goal, the design may not need to be rigorous. Rather, the key questions may be responsiveness and timeliness.
One participant commented that, if outcomes are to be the essential evaluation measure, the TA Centers have a great deal of work to do quickly. When evaluation addresses the efficacy of a specific TA Center, the TA Center, by default, is then placed in the position of defending its funding.
If the evaluation's intent is to determine whether certain programs work in the abstract, then the focus shifts to systems. For example, the focus of such an evaluation would not be on the work of individual organizations. It would be on looking at small towns in general to determine whether they approach certain issues, such as stigma, in a uniform way and whether the approach has value.
The participants agreed that the TA Centers lack the level of development to measure outcomes. It was suggested that the TA Centers can move in that direction by incorporating evaluation into their project contracts.
Certain project aspects are easier to evaluate. For example, evaluating a dissemination project may simply require creating a database of users and periodically sending them a feedback form. It was asked whether it is sufficient to find out if people merely received the material and found it useful, or if it is necessary to determine whether the information resulted in certain outcomes.
The response to this question was that telephone follow-up helps to determine how the information was incorporated, as well as how the information may have changed a practice or a program. Telephone follow-up offers an opportunity to determine what people really want and to obtain suggestions for improving the program. However, unless the TA Center represents a behavioral health system or county mental health authority, change is unlikely to occur at the systems level.
One participant remarked that it is an unfair obligation to be responsible for ensuring that a program reads the TA Center's material and uses it appropriately or effectively. In addition, most program requests are for on-site technical assistance and intensive training, which the TA Center cannot afford to provide.
It was suggested that on-line training via the World Wide Web may be a better alternative to on-site training. One participant suggested evaluating what effect on-line courses have made so far on mental health systems.
It was proposed that the TA Centers share the evaluations done in the last funding cycle and plan to share the evaluations from this cycle. However, if evaluation is defined as outcomes data, some TA Centers lack the means to gather and analyze such data.
Following an invitation to continue the discussion informally, the session concluded.
Knowledge Application (1:50 p.m. - 2:40 p.m.)
The critical issue for the TA Centers regarding knowledge application is how to get people to use existing knowledge about what works. Reports simply end up being used as paper weights or collecting dust on bookshelves. Literature suggests that clinicians are notorious for resisting journals as a source for new ideas. Videos or regional demonstrations may be a more effective means of showing what works.
It may be useful to target audiences with strong linkages to planners, policy makers, advisory boards, and the provider community, so systems change will actually occur.
One participant pointed out that there are two sides to the equation of getting people to use knowledge. If you are on the receiving end of information, you are unlikely to adopt or use it unless you feel a sense of buy-in or ownership. For example, if you do a regional demonstration that simply presents a list of 10 steps for accomplishing something better, the information is likely to be rejected. But if you work with the targeted audience to apply the information to their needs, then you're building a long-term relationship and satisfying both sides of the equation.
The composition of the audience is another factor that affects whether the knowledge shared is actually put into practice. Simply put, the person representing an organization at a presentation must have the authority to make decisions. This is especially important given the collaborative nature of many services. An audience that claims only to want answers may be the least likely candidates for systems change and will probably reject what's offered.
Although thousands of communities may be ripe for change, money is simply not available for sending experts to every community that is ready. This problem could be solved partly by capitalizing on the networking potential of national conferences in a deliberate, yet informal, manner.
Discussion returned to the topic of readiness. Several questions were raised: Is it better to allocate limited resources to those ready for systems change or to invest efforts in getting people ready? How is the appropriate level of readiness to ensure systems change determined? Should the target audience define the appropriate level of readiness?
One participant asked whether it is a good idea to "take a community's pulse." The goal would be to identify the communities ready for systems change, to provide an array of technical assistance to the ready communities that have the linkages to change, and to assist communities that are ready but do not have the linkages to change.
One respondent said that such an approach clearly would work initially but would spark a demand for similar technical assistance. The participant questioned whether it is acceptable to tell communities that lack the necessary linkages for successful systems change that, despite their readiness, resources to assist them do not exist. It was further questioned whether it is appropriate to then suggest that the communities seek more money from their elected officials. Another participant expressed discomfort with the potential for creating a system where the most advanced communities gain the most, leaving behind those with fewer resources and capacity.
One participant talked about a positive experience working with targeted audiences, such as the State mental health associations, who are ready for systems change and are set up for effective advocacy. Other associations in this category include the National Association of State Mental Health Program Directors, the American Medical Association, and the American Psychiatric Association.
It was suggested that core capacities, such as family and consumer involvement, are one way to bring about systems change. Investing in family and consumer involvement creates demand, and if it is informed demand, the right things will be done.
The session concluded with a summary of the major points of the knowledge application discussion:
- Although journal articles are seldom read and are not a particularly useful way to disseminate knowledge, they are necessary to maintain credibility.
- The "sender/receiver" model of knowledge exchange allows for buy-in.
- Technical assistance may be viewed as a learning or skill development process rather than as a knowledge exchange process.
- Using the sites as models and ambassadors may be especially fruitful.
- Conferences can provide invaluable opportunities for networking.
- Investing in communities/programs that are ready for systems change and have the linkages to do so may be the best use of scarce resources.
- CMHS and the TA Centers may make better use of membership organizations as "communities" that are ready since they already have buy-in from their members.
The session was adjourned.
Knowledge Application (2:40 p.m. - 3:20 p.m.)
Michael English, J.D.,Director, CMHS Division of Knowledge Development and Systems Change, opened the session by stating that the goal was to explore how to bring about real knowledge development and application, also known as KDA. The discussion began with a brief description of KDA. KDA involves learning about what works, getting the information out to those who need it, and ensuring that what works is implemented. In other words, KDA creates systems change.
One participant suggested that one way to push systems change with those TA Centers that focus on researching and writing articles is to package the information that appears in academic journals in "user-friendly" formats. In addition, frameworks or logic models are necessary to help people use the information. For example, a framework for recovery would present all of its elements, such as understanding the theory, application, potential outcomes, and the different constituencies that would be interested in this topic such as consumers, providers, and State and local administrators.
One participant mentioned the way he was introduced to the Program for Assertive Community Treatment (PACT) model. Using cost effectiveness data and information about the model's fidelity and disseminating it in various settings, PACT was presented in terms of systems change. However, despite fairly convincing data to back up its effectiveness, the participant noted, in general people will continue doing what they've done in the past based on the consensus-building process in their area.
The PACT anecdote sparked a question about whether or not CMHS and the TA Centers have an obligation to market knowledge that works, or whether marketing is beyond their scope. This led to a discussion about what needs to be done to bring about knowledge application and systems change. The key steps are to identify exemplary practices, to find the appropriate vehicles (publications or on-site technical assistance) for sharing that information, and, most importantly, to translate the information into a usable format. These steps must be repeated over time and in ways that constantly re-evaluate success and promote an ongoing relationship between the TA Center and the community.
It was suggested that targeted technical assistance is one way to build a relationship. About 13 people at the TA Center are assigned to develop long-term personal relationships with individuals in the States. This enables knowledge to flow from the TA Center to the State, and the TA Center also stays in contact with and receives information from the field.
Discussion turned to the possibility of funding meetings comprised of key decision-makers to focus on an agenda of systems change. One participant suggested creating a new "broker or match-maker" position. This individual would have responsibility for staying informed about what is happening in the key communities of five States and would help to identify decision-makers.
The notion of taking a proactive networking role at conferences was discussed. One participant noted that the best conferences are those where attendees come as interagency teams. These teams may be formed solely for the conference and are therefore required to work together beforehand. At the conference, the team works on decision-making processes and strategic planning. This helps to ensure that the interagency teams return home with a plan and at least one or two action steps. For some communities, such interagency conference teams may represent the first step toward integrated systems of care.
Another way to increase the value of conferences is to insist that speakers focus on practice rather than theory. Helping conference participants to build skills allows them to return home with tools for systems change. It was suggested that when scholarships are provided for attendees through a TA Center, the recipient should be chosen by an advocacy group with the understanding that he or she will go back and share the information gleaned from the conference. It would be wise to give scholarship recipients specific suggestions or strategies about how to effectively share the information learned.
The session concluded with a summary of the major points of discussion:
- Theoretical or formal knowledge must be translated into practical knowledge.
- Useful knowledge must be defined by the constituents.
- Producers of knowledge development must also think in terms of usefulness and practicality.
- Information materials must be translated into skills-oriented materials.
- Systems change occurs in steps over time.
- Relationships between providers and users of information must be long-term.
- Systems change rests on relationships with individuals who have the authority to advance change.
- Match-making/linking/networking must be a part of TA Center and KDA activity.
- Receivers of information bear responsibility for sharing the skills gained from meetings and conferences.
A final comment addressed the need to have the Centers for Medicare and Medicaid Services sit at the table and write regulations that reimburse best practices if systems change is to occur.
Community Action Grants for Systems Change
The session began with a discussion about what constitutes an exemplary practice in the eyes of the grant review community. In the absence of defined common denominators for an exemplary practice, one participant suggested developing a skeletal framework for exemplary practices with key elements at the Federal level. Key elements may include theory, practice, activities, outcomes, and how these factors relate to one another. A consistent framework for all programs would facilitate communication, evaluation, and synthesis of information.
Discussion turned to the role TA Centers can play in helping community stakeholders reach consensus. Participants advised that a "consensus" curriculum would help stakeholders build concrete skills. Another alternative would be to provide on-site consultants who are trained in mediation and dispute resolution. The Georgetown TA Center, which has been operating a mediation and dispute resolution program for about a year, could serve as an appropriate source of neutral, non-biased facilitators.
TA Centers can also help community stakeholders reach consensus by providing information about exemplary practices and how others have solved problems that can plague collaborative projects, such as personality clashes, shared responsibility, and shared resources. One participant mentioned the need to be sensitive to specific issues, such as long-standing histories of administrative issues, funding streams, and differences in theories and beliefs, within each community. An understanding of and respect for how the "culture" evolved over time is essential.
Returning to the topic of exemplary practices, one participant recommended a collaborative process among peers to identify what constitutes an exemplary practice, which again raised the difficulty with identifying the elements of an exemplary practice.
Another participant raised the issue of how to communicate easily and exchange information among the TA Centers and also connect with grantees who can provide cutting-edge information about what works in the field. A listserv among TA Centers and grantees was suggested as a technological pipeline for facilitating such communication.
TA Centers clearly can help potential grantees in several ways: developing grant application skills, building consensus, helping with negotiation and mediation, identifying exemplary practices and programs, and evaluating the consensus-building process. Evaluating the consensus-building process is necessary to learn what works and what can be replicated in other parts of the country. One participant cautioned that such help is possible only if the TA Center has the necessary expertise and capacity.
The initial community action grants represent opportunities to define models of successful consensus building and to determine for whom these models work. These initial grants will also provide an understanding of how long consensus building takes and how much it costs. This information can be obtained partly through each grant's process evaluation component.
After a brief recess, the discussion turned to the topic of community information needs.
Paolo del Vecchio, CMHS Consumer Affairs Specialist, noted that the goal was to focus the discussion on the information needs of consumers/survivors and family members. Clinical trials show that people who are well-informed about their treatment tend to have better outcomes and that their treatment tends to be less costly in the long run.
Given these benefits, Mr. del Vecchio, highlighted the primary ways in which CMHS identifies consumers' and family members' information needs: focus group meetings, regional meetings with consumers and family members, and conference caucuses. He asked the TA Centers for suggestions about strategies and pitfalls related to how CMHS can determine and address information needs. TA Centers were asked what they can do to assist in providing such information to those who need it.
Several participants noted the value of telephone calls from individuals requesting specific information and the importance of toll-free 800 numbers. Access to toll-free numbers is critical, since many consumers and family members are low-income. Knowing what information requests a TA Center receives most frequently, as well as what information requests it is unable to fulfill can help to identify information needs. TA Centers can use surveys to acquire such information.
One participant noted the importance of telling people that information sources other than the TA Center are available. The Internet was suggested as a potential tool for identifying and collecting information to meet the most pressing information needs.
Placing consumer advisory groups and consumer representatives on boards are another way to ensure that consumers' and family members' information needs will be met. However, these representatives must be trained in leadership and advocacy skills to be effective. Those willing to serve as board members may need training to be able to "hear" consumers' and family members' views.
It is important to go outside the usual spheres of influence and input for fresh ideas that "resupply the system." Places where "real" people congregate, such as clinics, hospitals, and self-help programs, were suggested as possibilities.
The discussion moved from identifying information needs to meeting those needs. The information may need to be delivered by an independent agent not vested in the system to ensure that all information needs are met. For example, at one time hospitals hired patient representatives, also known as patient advocates. If they had been hired by an independent agency, the program might have been truly successful.
Providing information/education must be an ongoing process. Going beyond the usual mental health circles and humanizing mental illness were emphasized as two critical needs. Video conferencing was suggested as a way to reach broader audiences and put a human face on mental illness.
Discussion returned to how the TA Centers can better meet consumers' and family members' information needs. One participant suggested mailing and posting flyers within the community to plug a TA Center's free materials. It was noted that outreach and marketing in a non-stigmatizing, non-threatening manner promotes a sense of hope and gets out the word about TA Center resources. Conducting training activities, sharing mailing lists, tapping into the faith community, and creating a centralized directory of information were the other suggestions given for ways to better meet consumers' and family members' information needs.
The session was adjourned.
Innovation Packages
Jim Mora, of the CMHS Division of Knowledge Development and Systems Change, facilitated the session. He pointed out that innovation packages are an important part of KDA because they translate the information developed by the Division into a format that can be used in different venues.
One Division frustration is that, although State planners have access to and receive much of the knowledge generated, they tend not to use it. In their defense, it was noted that State planners cite lack of money to meet basic needs, never mind money for new programs. Mr. Mora asked the participants for ideas about how to get new knowledge embedded into State plans to prompt systems change. He also asked for suggestions for a new name for the information packets; it should be a name that State planners will not associate with added costs and work.
One participant said that State planners appear to lack receptivity and that understanding State planners' views, needs, and wants may bolster their interest in innovation packages. It was suggested that State planners may be the wrong audience to advance systems change. In that case, who should receive innovation packages and what they should look like are questions that need to be answered.
The issue of readiness was raised. One participant who has done a clinical training on preparing people for treatment suggested that the concepts used in the training are also applicable to systems change. Three concepts were cited. First, an individual (or, in the case of systems change, the State planner) needs to feel a need for change. Second, the individual must think change is possible, that change is positive, and that he or she can be an agent of change. Third, the individual should have access to a permanent and supportive person or structure.
One participant suggested bringing those newly hired to write State plans together for a 2 to 3-day training, since almost one-third of these individuals change every year. Training would provide newcomers with skills they need to perform their jobs and would create an opportunity to capitalize on their initial eagerness to improve the system.
One participant suggested asking State mental health planning councils to endorse the innovation packages. Since all councils have consumer and advocacy representatives, they may be able to advocate effectively for systems change using innovation packages.
It was suggested that State planners might be reluctant to adopt innovation packages because of their uncertainty about funding and the question of whether their programs will be rolled into managed care. It was suggested that CMHS give State planners recommendations about how innovation packages can fit with their existing plans. Breaking the information into manageable pieces may help State planners see how they can integrate parts of innovation packages into their overall strategy.
One participant said that presenting innovation packages in a step-by-step format may prove to be effective, because people want to know what steps are necessary to implement changes. People also want to know what problems and obstacles they may encounter as they make changes, as well as possible solutions.
A manual that includes a conceptual framework for understanding knowledge application may be a useful tool for State planners. The manual would provide a road map that State planners could review, use to identify where they are, and figure out what information they need. Peers who have already written State plans and have incorporated some of the approaches in innovation package could be used as trainers.
The need to provide follow-up assistance to State planners was raised. Discussion focused on who should provide the assistance.
The session was adjourned.
|
