PART ONE
CHAPTER 13. RATING SYSTEM
DEFINITION
A rating system is an instrument for forming and validating a judgement on the relevance, per-formance and success of a programme or project through the use of a scale with numeric, alpha-betical and/or descriptive codes. As part of the broader monitoring and evaluation framework, it has the potential to provide critical inputs to decision-making at various management levels, support accountability, and enhance learning.
ESTABLISHING A RATING SYSTEM
The following factors need to be considered in establishing a rating system.
Mechanisms are needed to ensure the reliability of the rating system. The system must be continuously tested and refined within the relevance-performance-success framework for monitoring and evaluation. The questions need to be reviewed so that the specific criteria of relevance, performance and success are captured. To the extent feasible, the key stakeholders (i.e., target groups, programme or project managers, Government and UNDP) must rate the programme or project as a way of obtaining their views and as part of a validation process that can help to ensure the reliability of the rating system.
Rating System for UNDP-assisted Programmes and Projects
Rating Categories
The following categories will be used to rate UNDP-assisted programmes and projects through monitoring and evaluations.
USES OF THE RATING SYSTEM
The rating system must be used for the monitoring and evaluation of both programmes and projects. It should thus be treated as an essential component of monitoring and evaluation reports (see chapters 14 and 21). For monitoring, the rating system should be built into field visit reports and annual and terminal reports. For evaluation purposes, the rating system will continue to be incorporated in the programme or project evaluation information sheet (PEIS), which evaluators are required to submit along with their evaluation reports. Data from the PEIS will continue to be entered in CEDAB.
PART ONE
CHAPTER 14. REPORTS
IMPORTANCE OF REPORTING
Monitoring and evaluation reports are valuable sources of information that can form the basis for decision-making and learning at the programme or project level. They constitute part of the institutional memory on programmes and projects that can be easily retrieved and used by managers and development partners. This is especially true when the basic information on relevance, per-formance and success is extracted from the reports and entered into a computerized database, such as CEDAB, to facilitate the retrieval of information and contribute to trends analysis. In addition, recommendations from evaluations initiated by headquarters can also serve as a basis for policy directives to improve performance at the corporate level.
MONITORING REPORTS
Reports must be prepared for all monitoring actions: field visits and stakeholder meetings, including bipartite and tripartite reviews. In addition, periodic and terminal reports must be prepared for all programmes and projects regardless of budget and duration. These reports should serve as inputs to CCF reviews as well as to any evaluation exercises that will be conducted.
Monitoring reports should include an assessment of the relevance and performance of programmes and projects. They should identify early signs of potential problems or success. Based on such an assessment, monitoring reports must contain practical recommendations on how to resolve problems or optimize initial gains. A rating system should be a useful tool for the key stakeholders in forming and/or validating a judgement about a development intervention.
The form for preparing the annual programme/project report that replaces the project performance evaluation report (PPER) is presented in chapter 21. It incorporates the principles discussed in chapters eight through 13.
The report form has two parts. Part one consists of a series of questions for use by the key programme or project stakeholders (beneficiaries, managers, government and UNDP) in numerically rating the relevance and performance of a programme or project. The sample form contains space for the inclusion of the specific performance indicators that have been selected for the programme or project that is being monitored. Part two contains questions for which written answers must be provided. Its purpose is to elicit in-depth responses to the issues addressed in part one.
EVALUATIONS REPORTS
Evaluation reports should focus on how issues pertaining to relevance, performance and success were, or continue to be, addressed as substantive concerns during the formulation, implementation and post-implementation stages. Evaluation reports should contain the following core elements: findings, conclusions, recommendations and lessons learned.
Findings
A finding is a factual statement about the pro-gramme or project based on empirical evidence. Findings should enable a critical assessment of the programme or project in terms of:
Conclusions
A conclusion is a reasoned judgement based on a synthesis of empirical findings or factual statements that correspond to a specific circumstance. Conclusions serve as the basis for recommendations.
Recommendations
A recommendation is a proposal for action to be taken under a specific circumstance, including the parties responsible for that action. Recommendations should focus on improving the relevance, performance and success of the programme or project being evaluated as well as similar interventions in the future. They should also include suggestions for improving existing indicators or introducing new ones as needed.
When formulating the recommendations, the evaluators should ensure that:
Lessons Learned
A lesson learned is learning from experience that may be applicable to a generic situation rather than to a specific circumstance. The lessons learned should be structured along the same lines as the findings, conclusions and recommendations, i.e., with a focus on relevance, performance and success. They should include both positive and negative lessons - the best and worst practices - that have a bearing beyond the programme or project at hand. The main factor that must be considered when drawing positive lessons from experience is their applicability. Thus, cluster evaluations (more than individual programme or project evaluations) offer the advantage of providing a broader basis for generic lessons that can be applied to a particular thematic, sectoral or socio-cultural context.
Within the overall relevance-performance-success framework, the reports should be structured in such a way that best and worst practices are adequately and consistently captured in the findings, conclusions, recommendations and lessons learned. Such practices may concern various categories such as programme or project management, participation of target groups, and the contribution of development partners.
FOLLOWING UP THE IMPLEMENTATION OF MONITORING AND EVALUATION RECOMMENDATIONS
The preparation of monitoring and evaluation reports is not an end in itself. Recommendations contained in these reports must be carefully reviewed and acted upon, as deemed appropriate, in a timely manner. In some cases, programme or project management can easily take action on monitoring and evaluation recommendations. In other instances, it may be necessary for the main stakeholders to reach a consensus before action on the recommendations can be taken.
The executing agency bears the primary responsibility for ensuring that recommendations are implemented. It must duly inform UNDP (and the Government if the Government is not the executing agency) of the implementation status of major recommendations, i.e., those that are crucial to the success of the programme or project.
Tracking the implementation status of major recommendations will be undertaken through the annual reporting required for all UNDP-assisted programmes and projects and the annual reviews of the CCF.
PART ONE
CHAPTER 15. FEEDBACK
DEFINITION
Within the context of monitoring and evaluation, feedback is both a product and a process.
As a product, feedback refers to information generated through monitoring and evaluation and transmitted to parties for whom it will be relevant and useful. It includes findings, conclusions, recommendations and lessons drawn from the programme or project experience.
As a process, feedback involves the organization and packaging of relevant information in appropriate form, the dissemination of that information to the target users and, most important, the use of that information as a basis for decision-making and the promotion of learning in the organization.
PURPOSES OF FEEDBACK
Feedback from monitoring can be distinguished from feedback from evaluation in terms of immediate purpose. Feedback from monitoring actions should provide managers and other stakeholders with a basis for making decisions or taking actions relating to the ongoing programme or project. In this context, feedback can reveal a problem that needs to be addressed before it becomes more serious. It can also indicate areas where progress is being made and that might benefit from additional support (a mid-term evaluation of an ongoing programme or project can also provide this type of information). Hence, feedback from monitoring supports immediate decision-making needs more than it contributes to long-term knowledge-building.
On the other hand, feedback from evaluation exercises (particularly ex-post evaluations) supports the learning function more than it assists in immediate decision-making. This type of feedback takes the form of lessons learned about what works or does not work under certain conditions.
FEEDBACK AS A PROCESS
As stated earlier, feedback can contribute to organizational learning, which may be defined as follows:
The key elements of such learning are experience, knowledge, access and relevance. Testing implies that learning is a dynamic process, that knowledge must be used in order to determine whether the lessons drawn from experience are valid. These elements are emphasized in the following discussion of the feedback process.
Extracting Lessons from Experience
Providing feedback from monitoring and evaluation is a means of sharing lessons that can be applied to current and future programming, implementation, monitoring and evaluation of UNDP programmes and projects. By capturing problems and likely areas of success, monitoring can provide initial lessons specific to the pro-gramme or project. On the other hand, evaluation must be able to draw or extract lessons from experience in such a way that generic issues are identified and alternative solutions are developed.
The underlying issue is how to capture lessons from experience that are transferable, that is, those lessons that can have a broader application as against those that are relevant only to a single programme or project. One of the most efficient ways of arriving at these transferable lessons is through the use of cluster evaluations - e.g., sectoral, thematic, or strategic - that can facilitate learning across countries.
Transforming Lessons from Experience into Knowledge
The significance of lessons from experience increases when the lessons are shared with others through such support mechanisms as training and workshops. Strategies must therefore be adopted by UNDP to ensure that experience becomes an effective teacher or source of knowledge. Thus, evaluation feedback in the form of best and worst practices must be systematically organized and disseminated to the target users by the producers of that feedback.
Providing Access to Feedback
Feedback is a two-way process that should involve an interactive relationship between producers and users. It extends beyond the physical distribution of feedback products, requiring feedback about feedback in order to ensure the high quality of the knowledge that is shared within and outside the organization.
As a two-way process, feedback should be able to accommodate both the information needs that are identified by the users (demand-driven) and those that the producers identify (supply-driven). By the same token, accessing feedback (user-initiated requests for information) should receive as much emphasis as disseminating that information (producer-initiated action).
Making CEDAB available within the entire organization represents a significant step in improving access to the lessons that have been gained from experience. It will also provide OESP with the opportunity to obtain feedback on feedback from staff at headquarters and in country offices, who will be able to comment on the quality and usefulness of the information provided by the database given their respective requirements. This constitutes a participatory approach in determining what feedback would be relevant and useful to specific target users.
Training and workshops for the staff of UNDP can be effective means of disseminating feedback in the form of substantive lessons from experience that can be useful in various stages of programme or project management, including evaluation. Training must focus on such areas as how to improve the quality of UNDP programmes and projects. In addition, it must develop skills in methodological innovations such as participatory evaluation, the selection of indicators, etc. Training in the use of CEDAB is also necessary.
Publication of evaluation results must be complemented by an improved system of dissemination to ensure that the target recipients receive the evaluation feedback that is relevant to their specific needs.
Ensuring the Relevance of Feedback
It is important to identify which feedback products are useful for the specific information needs of particular groups at various levels within UNDP. In keeping with the national capacity development objective of UNDP programmes and projects, the feedback requirements of governments must also be identified. Therefore, UNDP must initiate a participatory process through which target users can identify relevant feedback products (demand-driven) and provide inputs concerning their design and packaging. Target users include UNDP staff at headquarters and in the country offices, governments, and other United Nations agencies.
Using Feedback
The use of feedback depends on the action-orientation and timeliness of the information as well as on its relevance. Feedback must be action-oriented, that is, designed so that it can aid decision-making in the overall programme or project management cycle.
Relevant lessons must be incorporated in new programmes or projects. No programme or project should be considered for approval unless there is evidence that a comprehensive search for relevant lessons has been carried out and that the pertinent lessons have already been applied in designing the programme or project.
Feedback must be provided on a timely basis. Feedback from monitoring and mid-term evaluations must be available immediately if it is to be used as a basis for decision-making to improve implementation. This also applies to feedback from terminal evaluations of pro-grammes or projects for which a second phase is being considered. In general, lessons from evaluations must be available when new projects or programmes are being identified, designed and appraised prior to approval.
FEEDBACK PRODUCTS
A wealth of feedback products available within UNDP and the larger international donor com-munity contain lessons on the design, implemen-tation, monitoring and evaluation of technical cooperation programmes and projects in developing countries. Some of the most im-portant products are listed below.
OESP PUBLICATIONS ON LESSONS LEARNED
Lessons Learned - Main Series
Lessons Learned - Companion Series
Lessons Learned - Practitioners Handbook Series