Evaluating Coalition Progress and Impacts

Building Coalitions Series
CDFS-14
Community Development
Date: 
10/15/2014
Carol Smathers, Field Specialist, Youth Nutrition and Wellness, Family and Consumer Sciences, Ohio State University Extension
Jennifer Lobb, Graduate Research Associate, Ohio State University Extension

Evaluation and monitoring of coalition activities is essential for successful coordination of community programs and sustained coalition efforts. There are three general levels of coalition evaluation (Butterfoss and Francisco, 2004):

  1. Evaluation of processes that sustain coalition infrastructure and function (process evaluation)
  2. Evaluation of programs designed to achieve coalition goals (impact evaluation)
  3. Evaluation of changes in the health status of the community (outcome evaluation)

The first level of evaluation typically considers what has been done by the coalition, how many people have been reached, and whether the coalition itself is functioning optimally and as intended (Butterfoss and Francisco, 2004). There are a variety of coalition assessment tools that can be used for this type of evaluation available at coalitionswork.com/resources/tools.

The most common methods for evaluating coalition processes are surveys, questionnaires and the review of coalition records and reports. Additionally, an event log can be an especially useful way to monitor a coalition's success. An event log is a running record of coalition activities, including the people who are involved and the number of people reached. Data from an event log may help a coalition acquire funding and community support, as well as make future plans and decisions (Chalmers et al., 2003).

Questions that may be asked in the first stage of evaluation include the following:

  • How long has the coalition been together?
  • How often does the coalition meet?
  • What type of structure has been developed for the coalition? (officers, membership guidelines, etc.)
  • What types of individuals and organizations are represented?
    • Are there certain groups that are not represented by the coalition?
    • Are these the right types of individuals to accomplish the work of the coalition?
  • What has the group done in training coalition members and other professionals in the community?
  • Is the coalition serving as an advocate for issues in the community?
  • Do all members of the community have equal access to the coalition's efforts?
  • Has duplication of services been reduced/eliminated?
  • Have existing funds been used effectively? How?
  • Has the group been able to access new funds because of working together? How?
  • What is the coalition doing that is really working well?
  • What are the major problems faced by the coalition?

The second level of evaluation asks whether specific program objectives were met by the coalition and whether the programs were carried out as intended. A coalition logic model (discussed in the eighth fact sheet in this series titled Coalition Goal Setting) can help a group discern whether they have achieved success in carrying out planned activities.

Questions to ask in the second stage of evaluation will be specific to the coalition logic model, but general questions to consider include the following:

  • Did the coalition achieve its goals?
  • Did the coalition use its resources in the most efficient way possible to achieve its goals?
  • Is the community aware of the coalition's efforts? Are they supportive of its efforts?
  • Did elected officials support legislation proposed by the coalition?

The most common methods used for this type of evaluation include surveys, interviews, focus groups and structured observation. Evaluators typically need to gather information from community members who were affected by the coalition's activities to assess whether and to what extent the coalition achieved its objectives.

The third level of evaluation is a big-picture evaluation of what the coalition has accomplished. This evaluation takes time. Its ultimate purpose is to detect any changes to the health of a community that can be attributed at least in part to the coalition's efforts. Like the second level of evaluation, questions to ask in this stage of the evaluation will be specific to the coalition's mission.

Common methods used in the third stage of evaluation include surveys, interviews and document review. Evaluators can gather and aggregate information from community members to detect a change in health status. They can also obtain statistics from existing reports and compare data over time to look for changes in the health of the community.

There are many benefits to coalition evaluation. When the right questions are asked, the data collected from an evaluation can be used to do the following:

  • Determine whether coalition objectives were met
  • Improve program implementation
  • Increase community awareness and support of the coalition
  • Provide accountability to the community, to stakeholders and to funding agencies
  • Inform policy decisions

In addition, evaluation can help a coalition identify quick wins and successes that help to increase member commitment and build coalition credibility when recognized.

Despite its many benefits, evaluation is often overlooked because coalition funds are more likely to be spent on intervention than on evaluation. To ensure adequate resources for evaluation, it is wise to build evaluation activities into the coalition's program planning process. Technical assistance with evaluation is often available from local Extension offices, local health departments and other local nonprofit organizations.

References

Butterfoss, F.D. and Francisco, V.T. (2004). "Evaluating Community Partnerships and Coalitions with Practitioners in Mind." Health Promotion Practice, 5, 108-114.

Chalmers, M.L., Housemann, R.A., Wiggs, I., Newcomb-Hagood, L., Malone, B. and Brownson, R.C. (2003). "Process Evaluation of a Monitoring Log System for Community Coalition Activities: Five Year Results and Lessons Learned." American Journal of Health Promotion, 17(3), 190-196.

McKenzie, J.F., Neiger, B.L. and Thackeray, R. (2013). Planning, Implementing and Evaluating Health Promotion Programs: A Primer, 6th Edition. Pearson.

Ohio Center for Action on Coalition Development (1992). Building Coalitions: Coalition Formation and Maintenance. Darby, PA: DIANE Publishing Company.

Additional Resources

The Asset-Based Community Development Institute. "Downloadable Resources." Publications on community assessment and community mobilization. abcdinstitute.org/publications/downloadable

Coalitions Work. "Tools and Resources." Resources for a variety of coalition processes and coalition evaluation. coalitionswork.com/resources/tools

University of Kansas. "Community Tool Box." Toolkits on a variety of topics related to partnership building and community change. ctb.ku.edu/en/table-of-contents

University of Wisconsin-Extension. "Program Development and Evaluation." Logic Model templates and examples. uwex.edu/ces/pdande/evaluation/evallogicmodel.html

Iowa State University, North Central Regional Center for Rural Development. Vision to Action: Take Charge Too. Publication about community assessment, vision development, action planning and evaluation. www.soc.iastate.edu/extension/ncrcrd/ncrcrd-rrd182-print.pdf

Program Area(s): 
Ohioline http://ohioline.osu.edu