Annex II.

A SAMPLING OF TOOLS FOR PARTICIPATORY EVALUATIONS
Over the years, many tools and manuals have emerged that have contributed to the growing recognition of participatory evaluation as an evolution in, and a valuable alternative to, conventional evaluation methods. While the stages in undertaking a participatory evaluation can be clearly delineated, there is no one model, recipe or tool for such an evaluation. This flexibility has allowed the approach to grow and evolve.

The tools used depend, of course, on the nature of the project, the context and the stakeholders. Since each project setting is unique, different tools will be required, based on the particular political, cultural, economic and social characteristics of the project. Such tools can be used independently or in combination.

It should also be noted that the use of participatory evaluation methods should not preclude the use of quantitative methods. In fact, depending on the initiative, it might be beneficial to combine various methods and approaches. It is the responsibility of the experienced participatory evaluation facilitator to determine the most appropriate tools, depending on the context and nature of the evaluation.

Some of the most common tools are described in this section. For further information on how to apply them, a list of key manuals is also provided.

Beneficiary assessment
A beneficiary assessment involves the participation of beneficiaries in evaluating a planned or ongoing development activity and builds on the experience of participant observation. Assessing the value of an activity as it is perceived by its principal users, this tool seeks to provide a context for quantitative data by letting beneficiaries' voices, values and beliefs be expressed. Methods include direct observation, conversational interviews, and participant observation, which involves the protracted residence of an outsider in a community for a period ranging from several weeks to two or three months. These methods should be used by an experienced observer and inquirer.

Notes from a Participant Observer
After more than two years of existence, a fishing cooperative in the state of Rio Grande do Norte had attracted only about 10% of the fishermen for whom it was intended. To attempt to determine why, World Bank project officers and local management agreed to try out the participant observer evaluation method (which has since become known as beneficiary assessment), using host-country observers. Project officials selected a young man in his twenties who had recently received a university degree. He was to live in two fishing communities for several weeks each and spend an additional few weeks in and around the central cooperative in the state capital of Natal. After almost three months in these areas living with and talking to fishermen, the participant observer made the following observations about the role of local fish buyers ("intermediaries") in the project:

"The fishermen are less exploited if they deal with the cooperative, yet none of them are conscious of this, owing to the anticooperative campaign carried out by the intermediaries (many of whom are also fishermen). The majority of the active (cooperative) members are not individuals who are conscious of the advantages of cooperativism but fishermen who do not get along well with the intermediaries. On the other hand, many nonmembers give preference to the intermediaries in order to maintain ties of family or friendship.

There exists misinformation about the actual prices (offered by the cooperative for fish) such that none of the nonmember fishermen can say exactly what this price is. The notion that the fishermen have... is that the price is lower than what the intermediary pays, whereas the price of the cooperative is 20% higher than the price of intermediaries. This error is daily reinforced by the intermediaries. The anticooperative mentality is such that the fishermen do not believe that the cooperative is paying a higher price when they are informed that this is indeed the case".

As a result of the information from the participant observation, the local manager of this Bank-supported fishing cooperative instituted a comprehensive new promotional campaign to educate the fishermen about cooperativism, took steps to replace a cooperative administrator seen by the fishermen as cold and uncommunicative, and redistributed profits to the fishermen, thus redressing one of their major grievances.

Source: Lawrence Salmen, Listen to the People, 1987:93-94.

Focus group
A focus group brings together a representative group of 10 to 15 people, who are asked a series of questions related to the task at hand. A facilitator guides discussion. Focus groups, which draw from local experience and traditions and provide local insight, are useful in project design and in assessing the impact of a project on a given set of stakeholders. While focus groups are commonplace among North American advertising agencies, they are being used increasingly in the field to validate project designs or help to assess project performance.

Seasonal calendar
This calendar uses visual representation to help to identify events that occur seasonally. It is useful for a wide range of projects and activities (planting, harvesting, identifying seasonal health risks, seasonal transmission of HIV). It can also highlight such constraints as drinking water availability, drainage blocks, labour availability, food intake, etc. A seasonal calendar can be created on the ground using stones, sticks or leaves or on paper.

Logical framework analysis (LFA)
The framework presents, in a nutshell, the core activities and outcomes of the project as well as the key indicators for monitoring and measuring its results. Customarily, this exercise is undertaken by the person responsible for the project design with some type of input from the field. However, it is possible for stakeholders to be the authors of the LFA by holding special workshops. Once the objectives have been agreed upon, stakeholders can work at defining the respective activities, results and performance indicators. Not only do stakeholders provide valuable input into the project design, but they also have a much better understanding of the project.

LFA is used by many donors as a planning tool to summarize the activities, results and verifiable indicators related to each objective of a project. For project evaluations, the LFA will usually be a key source of information referred to by the stakeholders since it contains a summary of the purpose of the project and its expected results. The verifiable indicators will also help stakeholders to measure the degree to which project results have been achieved. It is important to review the LFA with project stakeholders to ensure that the indicators are still valid.

LOGICAL FRAMEWORK ANALYSIS

Semi-structured interview
A semi-structured interview, which is less formal than a structured interview, allows for conversation and the reciprocal transmission of information. Preparation usually involves outlining the broad areas of inquiry, leaving specific questions to be formulated during the interview itself.

In her book, Participatory Program Evaluation, Judi Aubel highlights a series of questions that were part of an interview guide for community health nurses in Gambia and Sierra Leone. As she points out, the questions should be sequenced with the easier questions coming first and less intimate questions coming before more personal ones. The questions are open-ended and seek to collect in-depth information on people's attitudes, opinions and knowledge. This allows the interviewer the time to gain the confidence of the person being interviewed. The questions should also be kept simple.

The following are some of the questions that were used in an interview guide for Community Health Nurses:

1. What was your role in the Nutrition Education Pilot Campaign (NEPC)?
2. What was the role of the Mothers' Committee in the program?
3. To what extent did they assume that role?
4. Did you receive your fuel subsidy?
5. Was the fuel given adequate for carrying out your NEPC activities?
6. What was your role in monitoring the NEPC activities?
7. Were you trained on how to monitor the NEPC activities?
8. What information did you collect in the monitoring?
9. How frequently did you monitor the activities?
10. Did you encounter any obstacles in monitoring the activities?
11. What did you do with the information provided?

Source: Judi Aubel, Participatory Program Evaluation: A Manual for Involving Program Stakeholdersin the Evaluation Process, CatholicRelief Services, 1993:38.

Social mapping
This tool, which can be used at various stages of a project, involves community members in drawing maps of community structures, institutions, associations and resources on the floor, ground or paper. Mapping can provide insight into the interactions or lack thereof within the community, the resources that are available and access to them.

Social Mapping: The Importance of Having a Good Cross-Section of Participants
The diagram shown below shows the importance of ensuring a good cross section of participants in a social mapping exercise and different gender interpretations of one's community.

When men were asked to map their village, they showed their village as a network of roads and services used by them. They showed the official residences of the chief and chairman, the cotton trees where the different clans met. As for the location of the school and hospital its proximity to the village did not seem to be of concern. It was suggested that wells be located near each of the village clans.

When women were asked to map their village, they saw it through their own lenses. Women pointed to the areas where they collected water and fuel and worked in the fields. Their attention was focused more on the village per se than the outlying regions. The women specified specific houses inhabited by village leaders. The women suggested that the hospital and school be located near the village with the well at the center of the community for the greatest benefit of all.

Social mapping can highlight different perceptions of one's social environment. What is important to one group may be less important to another. For that reason, it is critical that stakeholders are well represented and come from a cross-section of the community.

Testimonial
A testimonial records a person's thoughts, feelings and experiences in the first person narrative style. It is a way of learning about a project or its impact through the voices of participants and stakeholders. Testimonials can help to reveal the degree of empowerment, the way in which income is used, how decisions are made or issues tackled. They can also help to corroborate other sources of data and information and provide a more personal insight into a project's achievements. Usually testimonials are taped and played back to the participant.

Sample Testimonial:
Ms. Mosammat Jainab Bibi, the Manager of the Shahapur Bittaheen Women's Cooperative Society in the district of Jamalpur, Bangladesh, joined the Cooperative Society in 1984 as a Manager. She is involved in paddy-husking and poultry-rearing activities. She received training on Members Education, cow rearing and poultry. She is also attending the Manager's Training regularly.

"I joined the Society in 1984. Mr. Tara, the local upazila official had distributed 15 wheat feeding cards among 15 vulnerable female villagers. One day he told to mobilize another 15 women to form a society. We did it and he helped us to form a BRDB society. We deposited Tk.1 per week as savings. We were not united then. We did not know each other. When the other 15 women joined us we held a weekly meeting. We continued it and Mr. Tara would also attend. We generated a little fund and Mr. Tara and we deposited it at BRDB office. With our consent he formed a BRDB society for us. We deposited Tk.1 or 2 as savings in 1984. We did everything by ourselves like raising savings, depositing them at the bank, issuing verity vouchers, taking receipts from bank etc. We registered our society on 30.3.85. It is nearly 7 or 8 years that we have been running our society....

We take a loan every year and husk paddy which provided us with some profit. We spend a little of that for education of our children. Previously, the Railway School was completely reluctant to admit our children. The directors of BRDB asked us once: "How many are you?" We answered we were 46. They replied that means at least 46 children and advised us to go and admit our children in the Railway school and gave us hope that they would help us. We went there, the teachers were in panic. We asked them: "Why do you not want to admit our children-because we are poor? Since we have no clean clothes? Why do you admit rich children?" Then the teachers agreed to admit our children.

They told us to pay Tk.10 for each boy or girl as an admission fee. We had protested earlier but realized very soon that we had to pay Tk.10 because it was compulsory for everyone. They gave us 7 days to collect the money. We collected the money and admitted our children in the school. That's how we overcame the problem...We cultivate fish collectively.. We have no pond but we requested one old man to provide us with his pond for fish culture. "We will cultivate fingerlings or young fish in your pond. We will sell the fish after 2-3 months regularly and the rest will be yours", we said. We took the pond under this condition and we earned Tk.880 in 2 months. We have been cultivating fish for 5-6 years and earn Tk.500-600 each 2-3 months. We maintain the pond and take care of it, catch the fish and sell them. We do not get the help of any men. We have utilized our training fully."

The above excerpts of a dialogue were part of a broader evaluation exercise of a training component of a Bangladeshi project. The dialogue method was used to complement and enrich the quantified data obtained through the interview questionnaire, case studies and file review.

Source: Yusuf Kassam, 1995:6-7.

A Transect Walk Through Mbusyani, Kenya

Venn diagram
A Venn diagram, of usually circular areas, can be used to look at relationships within institutions or relationships between the community and other organizations. It illustrates different participant perceptions of access to resources or of social restrictions, for example. Circles of various sizes are cut out of paper and given to participants, who are then asked to allocate the circles to different institutions, groups or departments. The larger the circle the more important it is. The circles may overlap, showing the degree of contact between institutions or groups.

A Potato Project in Pakistan:

Participants from headquarters and the regions were asked to identify the different institutions, associations and target groups related to their project. Participants were asked to select circles of different sizes to represent the importance of an institution or group. It was noted that the headquarters staff located in Islamabad mapped a wide range of institutions with which they maintained contact: donors, regional research agencies, private businesses. Provincial staff closest to headquarters had less knowledge of international linkages yet knew of many linkages, such as the research agencies, farmers and different private-sector groups. Those most remote showed even fewer linkages and less interaction with other institutions.

In barrack, one of the villages, the following indicators were used to distinguish levels of wealth within the village: