Yet, the majority of participants in formal collaborative processes know that with this great promise comes great challenges. Success depends on many factors--some common sense, others less obvious, and many not always practiced or universally understood. For this reason, the New National Opportunities Task Force decided to examine some of the lessons that could be learned from a sampling of formal collaborative efforts now underway or recently completed. Specifically, the Task Force wanted to (1) document lessons learned from a diverse sample of collaborative processes; (2) identify when and under what circumstances collaborative approaches are useful and effective; (3) identify characteristics that are essential to successful collaborations and, conversely, the characteristics that cause collaborative processes to falter and fail; and (4) recommend next steps for evaluating collaborations.

The working group reviewed existing academic literature on collaborations, a variety of written project evaluations where they existed, and other published and unpublished background materials for a number of high profile projects. The findings and recommendations presented below are based on these materials and the expertise of the PCSD working group.

Cases Studies Evaluated

EPA's Common Sense Initiative (CSI)
EPA's Project XL
EPA's National Environmental Performance Partnership System
Great Printers Project
Pollution Prevention Pilot Project (4P)
PCSD I - Eco-Efficiency Auto Project
Regulatory Negotiation (compendium of 8 cases)
South Florida Ecosystem Restoration

2. Collaboration is a process that is helping us learn how to solve society's complex problems, and evaluation is a key to learning. Learning can and should take place during and after a collaborative process. During a collaboration, the parties can learn in real-time, track short-term milestones, and take proactive steps to ensure the success of the project or program. After a collaboration, the participants or others can look back to learn whether the overall program or subparts of it met their objectives, and how to improve future efforts. Both forms of evaluation are important, although we need to recognize the difference between proactive, real-time learning and retroactive, after-the-fact learning. In either case, stakeholders should be involved in the evaluation. In some instances, it may be useful to engage an outside expert or organization to help guide, direct and implement the data collection and analysis. If the parties to a collaboration have reached an impasse that causes a breakdown in the collaborative process, it may also be useful to seek the assistance of outside stakeholders to conduct the evaluation.

3. Evaluations of collaborative efforts are rarely conducted either during or after a project. When they have been conducted, they are not usually designed at the outset of the project, with adequate involvement of the diverse stakeholders who have an interest in the project's success or failure, or, as may be useful in some instances, with outside professional expertise to guide and assist participants in creating the evaluation. These shortcomings often lead to recommendations with little, if any, stakeholder buy-in. Because many of the collaborations reviewed are "works in progress" and the available evaluation materials were limited, it is not possible to determine whether the efforts had succeeded or failed. Despite these constraints, the existing evaluation materials are useful for identifying specific features of collaborations that can strengthen the process and results (See Finding 5).

4. Collaborations are becoming more complex and evaluation processes must adjust accordingly. In the past, a partnership was often viewed as a one-time collaboration such as a regulatory negotiation ("reg neg"), where a single party governed the process, many players participated, and the process had a discreet endpoint (e.g., writing a rule). Now references to collaborations often mean something different. The Environmental Protection Agency's (EPA's) Project XL, Common Sense Initiative (CSI), and National Environmental Performance Partnership System (NEPPS) are themselves long term collaborative efforts to create institutional frameworks that foster many simultaneous mini-collaborations and sequential negotiations or projects. For example, the EPA set up the XL framework to foster facility-specific agreements that would deliver more innovative and protective environmental solutions than would be expected under existing regulatory requirements. These new approaches indicate a fundamental change in how society solves its problems and requires evaluations at two levels--the overall framework and individual projects.

5. Trust is essential and ownership of the process and outcomes fosters trust. Stakeholders' trust in each other and in the "process" are the most important determinants of success for most projects. The most important determinant of trust is stakeholder ownership of the process, outcomes, and metrics of success. However, even when trust among stakeholders is difficult to obtain, collaboration can still be successful if the stakeholders trust the process. Although it is difficult to predict how much of an impact a particular problem will have on trust, it is certain that routine breakdowns in trust make it difficult to achieve the original goals of a project.

There are specific "characteristics" of a formal collaboration's objectives, process, and participants that will engender stakeholder trust and ownership. When present, they significantly increase the chances of success. A common theme underlying many of the characteristics is the importance of participants maintaining shared or equal power and influence. Not surprisingly, when collaborations first begin, influence and power are often unevenly distributed among the partners, with some partners having greater control over the process, resources and information. Success in collaboration seems closely linked to the ability of the participants to resolve or overcome these inequities.

1. Characteristics of the Vision and Objectives

Measurable Outcomes. The case studies indicate a general consensus that (1) projects should focus on quantifiable environmental and economic results, (2) agreed upon measures can strengthen ownership, and (3) a monitoring system should allow stakeholders to easily track results. Yet, in many of the case studies, performance measures do not exist. Where they exist, it is uncertain whether they are being used. The lack of specific and agreed-upon measures in Project XL, the CSI and NEPPS undoubtedly is responsible for some of the disagreements that have arisen in these projects. There are some notable exceptions, including the South Florida Ecosystem Restoration project and the CSI Metal Finishing sector's strategic goals project. Note also that measures may be specific to an overall project--the NEPPS system as a whole--or a specific negotiated agreement--between EPA and a state environmental protection agency. Both are important. Despite the general lack of environmental and economic measures, all case studies examined (with the exception of the CSI and Project XL which cannot yet be fully evaluated) indicate a belief that they have had some successes. One of the lesser known benefits of stakeholder's focus on outcomes has been discussions about developing baselines, tracking results, and collecting and reporting data.

2. Characteristics of the Process

Shared and Defined Decision Making Process. Clearly defining both the decision rules and stakeholders' roles in decision making early in the process is another critical feature for engendering ownership. When implemented correctly, a "consensus" decision rule is perhaps the most effective way to create ownership because it allows participants to present their views honestly while maintaining sufficient power to protect their interests. Whatever the decision rules, participants should have a role in developing them and share the power of decision making. The more participants have at stake, the more critical it is to define clearly the decision-making processes.

Up-Front Planning. Organizations launching new collaborative efforts need advance plans for budgets and resources. Often, however, there is no such plan. The lack of planning, the reality that collaborative processes require more financial and human resources than many realize, and the routine delays in government procurement mean that it can take months or years to fund critical activities such as facilitation, research, travel, and consultants. The delays can deflate expectations and leave public interest organizations and small businesses without adequate resources to participate in the early stages of the process. The result may jeopardize stakeholder ownership. Public agencies often face these problems when they lead collaborative processes because new initiatives are typically launched quickly and with great fanfare, well after the agency's formal budget planning process for that year has concluded. All of the case studies with direct and important federal participation faced these dilemmas. Budget planning has also affected the foundation community, as it has scrambled to keep up with requests from non-governmental organizations for resources to participate in collaborative efforts.

Conflict Resolution. Conflict resolution has two important features that should also be clearly defined at the beginning of the collaborative process. First, a facilitated process with a professional facilitator or co-management team of stakeholders can avoid or resolve most conflicts before they escalate. Second, because all conflicts cannot be avoided, clearly articulated and collectively developed conflict management ground rules should be established. The GPP used small ad-hoc groups of representative stakeholders to resolve differences. A tiered approach that escalates issues to a higher level group or stakeholder-chosen decision making body may be a useful approach. A tiering approach will only work well, however, if decisions are dispatched quickly. The principal lesson is that when power is skewed, the process may collapse. When power is shared, the potential exists for outstanding results.

Open Communications Among Participants. Open and transparent formal and informal communication is critical for trust and ownership. Information that is withheld, or that is suspected of being withheld, threatens trust. Yet, overloading stakeholders with written material is not helpful and can shut down communication. Participants should periodically identify what information is available and/or needed by each of the participants. The importance of informal communications--by phone, in person--or the exchange of draft documents can be even more important in building trust than formal communication.

3. Characteristics of the Participants

Strong Leadership. Strong leadership is also a key to a successful collaborative process. For an organization such as the EPA that launches a new framework for cleaner, cheaper and smarter environmental protection such as the Common Sense Initiative, it is important to designate a leader or champion who provides strategic direction and moral support, secures or helps others secure financial support, and is willing to take risks to resolve an impasse in negotiations quickly and definitively. In South Florida the Governor's Commission and the Interagency Task Force provided this leadership. In launching the NEPPS system, several state environmental commissioners played an important role in overcoming concerns that the NEPPS policy was developed by a small group of state and federal agency personnel. In this situation, the leaders of the effort emerged during the process.

Create Capacity for Stakeholders to Understand Information. Because not all stakeholders come to a collaborative effort with the same knowledge or experience, it is typical for some stakeholders to feel left out of the process. This issue most often arises for citizens, public interest organizations and small businesses. It is important for all stakeholders who need them to have access to adequate information and financial and human resources to help them fully participate. Without this access, stakeholders may conclude they are being mislead or excluded by the other stakeholders. Whether they are actually being misled is largely irrelevant. Their belief usually cannot be overcome by either assertions to the contrary or piles of paper with detailed technical information.

Facilitators May Help and Should Apply Similar Tools. In situations where stakeholders have significantly different perspectives, and do not begin a collaboration trusting one another or the process, a "third party" facilitator without a stake in the issue or debate can make a significant contribution to helping parties communicate and build trust. The key to a facilitator's success is independence and objectivity. Beyond independence and objectivity, successful facilitation involves helping participants establish a common vision and objectives, and develop clear decision making rules and conflict resolution processes at the beginning of the process. The facilitator must also be able to anticipate conflict, help the group to resolve conflict and bring them back to the over-arching vision and objectives. Among a variety of challenges facing a facilitator is the potential for partiality that may arise because he or she has been hired by one of the stakeholders, and may, in turn, consult more frequently with that stakeholder. To overcome this possible impression of impartiality, a facilitator must be sure to consult jointly and equally with all stakeholders.

Step 2. Existing and future collaborative projects should establish ongoing processes for multi-stakeholder evaluations. The processes should include opportunities to make mid-course corrections and evaluate a project at its completion. The PCSD's search of the literature suggests that there is a shortage of comprehensive evaluations of specific collaborative processes conducted either during or after the collaboration. Even when they do exist, they have generally not been designed at the outset of the collaboration with input from all the stakeholders; nor have they clearly articulated interim milestones, final objectives, or feedback mechanisms to improve the process as it moves forward.

As originally envisioned, the working group's analysis was to include systematic evaluations, including interviews with stakeholders, of a select number of collaborative efforts. Given time and resource constraints, the working group raised concerns that adequate evaluations could not be done properly or in the allotted time. Based on this concern, the working group agreed to search the literature for evaluations of recent collaborative efforts. The literature review focused on recent unpublished or soon to be published studies rather than on the academic literature because most of the collaborative efforts of interest have not yet been completed or have been completed only recently. It will be some time before peer reviewed academic studies are completed. Simultaneously, the working group developed a protocol for conducting interviews with stakeholders and reviewing case materials to the extent that was possible. Both published and unpublished academic literature were used to develop the protocol.

To conduct the review of existing evaluations, working group and task force members nominated dozens of case studies of collaborative processes. Working group members then searched for available written material, developed a set of case selection criteria, screened the cases against those criteria, and developed a shorter list of case studies to examine.

The process of selecting case studies revealed that very few systematic studies of collaborative processes have been conducted. Two exceptions were the "Report of the Interagency Ecosystem Management Task Force" and the "Great Printer's Project" recently completed mid-term evaluation. There was no multi-stakeholder study of the U.S. Environmental Protection Agency's Project XL or Common Sense Initiative, although the Global Environmental Management Initiative had just completed its review of both programs. The materials reviewed varied in quality, comprehensiveness, purpose and possible biases. From these limitations arose the Task Force's initial recommendations to ensure that rigorous evaluations are undertaken in the future and that they are conducted with appropriate stakeholder input throughout a project. Despite the limitations of the available materials, there was information useful to the working group.

Time constraints, the volume of material, and the working group's intent to proceed by consensus raised several challenges in drawing useful and valid conclusions from the materials. The working group agreed to assign each member of the working group with responsibility for reviewing several case studies in depth and for becoming familiar with as many other case studies as possible. Working group members used the draft research protocol as a template for their reviews.

During several day-long, facilitated meetings, the working group systematically reviewed each case study identifying how the case materials addressed the following issues: (1) stakeholder participation, (2) clarity of objectives, (3) decision making processes, (4) conflict resolution processes, (5) communications among participants, (6) available information and resources (technical and financial), (7) outcomes (environmental results and cost-effectiveness) and (8) measurability of the outcomes, (9) indirect benefits, and (10) miscellaneous issues not captured in the other categories. In some cases, working group participants had personal experiences with the case studies and their personal experiences were noted. After lengthy discussions of the case studies the working group was able to reach concensus on overal conclusions about the case studies. These conclusions were presented to and approved by the full Task Force.

1. EPA's Common Sense Initiative (CSI)

2. EPA's Project XL

3. EPA's National Environmental Performance Partnership System (NEPPS)

4. Great Printers Project

5. Pollution Prevention Pilot Project (4P)

6. PCSD I - Eco-Efficiency Auto Project

7. Regulatory Negotiation (compendium of 8 cases)

8. South Florida Ecosystem Restoration

Evaluation Protocol:

Environmental Defense Fund, Environmental Sustainability Kit (1996).

Global Environmental Management Initiative, Research proposal to study "new or existing, positive, flexible, mandatory or voluntary business/government initiatives in the US at the federal level" (1996).

Long, Frederick & Mathew Arnold, The Power of Environmental Partnerships (1995).

Meidinger, Errol, Roger Clarke, and Margaret Shannon, Research protocol from study of "Science and Policy in Natural Resources" (1996).

Case Study Materials:

"The Ecosystem Approach: Healthy Ecosystems and Sustainable Economies," Vol. I - Overview, Report of the Interagency Ecosystem Management Task Force (June 1995).

"The Ecosystem Approach: Healthy Ecosystems and Sustainable Economies," Vol III - Case Studies, Chapter 7 - South Florida (March, 1996) p. 159. Evaluation of the Great Printers Project, Phase I and II (October 1996), Northwood Consultants.

Global Environmental Management Initiative, "Industry Incentives for Environmental Improvement: Evaluation of U.S. Federal Initiatives" (September 1996), (covers EPA's Project XL and Common Sense Initiative).

Hawkins, David, G. and Chris van Loben Sels, Memoranda to Fred Hansen Deputy Administrator, U.S. Environmental Protection Agency and Keith Laughlin, Associate Director for Sustainable Development, Council on Environmental Quality, (July 1 and July 3, 1996).

Pollution Prevention Pilot Project (4P), Advisory Council Meeting (February 28, 1996)

Pollution Prevention Pilot Project, Project Summary (June 6, 1994).

President's Council on Sustainable Development Eco-efficiency Task Force, Auto Team Report (March 1, 1996).

Steinzor, Rena I. "Regulatory Reinvention and Project X: Does the Emperor Have Any Clothes?" 26 Environmental Law Reporter 10527 (October 1996).

U.S. Environmental Protection Agency, Draft Final Report "Negotiated Rulemaking on Off-Road Vehicle use at Cape Cod National Seashore" (June 1996), Resolve, Inc.

U.S. Environmental Protection Agency, "A Review of the National Environmental Performance Partnership System Activity," (unpublished) U.S. EPA, Region IX.

U.S. Environmental Protection Agency, May 17, 1995, National Environmental Performance Partnership System.

U.S. Environmental Protection Agency, Project XL, 60 Federal Register 27282 (May 23, 1995).

U.S. Environmental Protection Agency, "Project XL Stakeholder Processes - Draft Interim Report of Observation Data" (August, 1996) (unpublished) Resolve, Inc.