Guide to Program Evaluation - Getting Started
Evaluation is a systematic, objective process for determining the success of a policy or program. It addresses questions about whether and to what extent the program is achieving its objectives.
Program Monitoring
Program monitoring involves the ongoing collection of information to determine if programs are operating according to plan. Monitoring provides ongoing information on program implementation and functioning.
Performance Measurement/Assessment
Program measurement or assessment involves the ongoing collection of information on whether a program is meeting its goals and objectives. Performance measures can address project activities, services delivered, and the products of those services.
Process or Implementation Evaluation
Process evaluation focuses on program implementation and operation. A process evaluation can answer questions regarding program effort; identify processes or procedures used to carry out the functions of the program; and address program operation and performance.
Outcome or Impact Evaluation
This type of evaluation focuses on program success and accomplishments. These evaluations answer questions regarding program effectiveness; address whether a program is achieving its goals and objectives; and examine unintended consequences, both positive and negative.
Cost-Effectiveness and Cost-Benefit Assessment
These assessments focus on using the results from a sound program evaluation to assess how effective the program is relative to other program alternatives in terms of cost. Cost-benefit analysis does not answer the question of whether the program works; instead, it uses the results of evaluations to compare the economic value of the outcomes and costs of one program with another.
Programs that participate in evaluations will obtain objective information about their performance and how it can be improved. Evaluation can provide objective evidence that a program is effective, demonstrating positive outcomes to funding sources and the community. It can help improve program effectiveness and can create opportunities for programs to share information with other similar programs and agencies.
Programs can use evaluation findings in a number of ways. For example, the program, to make a case for continued funding and to attract new funding sources, can use evidence of program success. A well-executed evaluation will point out areas in which the program can improve its operations. Also, sharing the results of evaluation has benefits to others outside of the program seeking to replicate justice interventions that work.
Program managers and staff can sometimes be reluctant participants in the evaluation process. Below are some frequently expressed concerns about program evaluation and responses to those concerns.
| Concern: | Evaluation draws resources away from program services. |
| Response: | Without evaluation, how do you know that the services being provided are effective? Program managers can explore options for obtaining evaluation services inexpensively. |
| Concern: | Evaluation increases the burden on program staff. |
| Response: | Evaluators can often implement changes to current client data collection procedures, resulting in little additional effort on the part of program staff. To reduce the burden and increase "buy-in," program staff should be involved in designing evaluation instruments and interpreting evaluation findings. |
| Concern: | Evaluation is too complicated for program managers and staff to understand. |
| Response: | An evaluation does not need to have the most rigorous scientific method, design, and analysis to be considered useful and valuable. Evaluation findings should be expressed in a manner that can be readily understood and used by program managers, staff, and other stakeholders. |
| Concern: | Evaluation may produce negative results that will harm the program. |
| Response: | A good evaluation will point out both program strengths and weaknesses. No reputable evaluator will willingly participate in an evaluation designed to harm a program. |
Every evaluation is carried out under certain constraints or limitations. These constraints should be identified as part of the planning process for the evaluation. Two major evaluation constraints are time and cost. Evaluation results that are not timely are not useful to program managers and funding agencies. When evaluation information is needed quickly, the evaluation must address fewer questions. Similarly, the financial resources available for the evaluation help to determine its scope. The strengths and weaknesses of various evaluation approaches should be considered while keeping in mind the level of resources available.