4. Monitoring / Evaluation

The following is a list of ideas about monitoring and evaluation for the film-production and distribution phase of the project.

a. Contracts, script adaptation and pre-testing

Contracts: Have all negotiations and documents monitored by a specialized lawyer.
Script adaptation: You can conduct interviews with the young authors and the artists with regard to the personal significance and value of their involvement.
Pre-testing: Pre-testing is, of course, a thorough evaluation of the script.

The pre-testing phase, from an organizational and methodological point of view, can be monitored and evaluated through discussions with individuals involved in one capacity or another. This might be of particular benefit to those who have recently received training in pre-testing techniques and are keen to know how they are doing.

b. Shooting

C Here too, you might consider conducting interviews with the young contest winners regarding the personal significance and value of their involvement on set.
C You could carry out interviews or group discussions with those involved in activities designed to inform onlookers, asking them about the value of the experience for them personally and their assessment of the impact it had on onlookers. You can also ask them if the onlookers made any observations that might be taken into account by those who draft the Users' Guide.
C You can assess changes in information levels and attitude in the film-production team itself.
C Be sure to monitor media coverage of the shoot.

c. Post-production

C Pre-test proposed scripts for language dubbing exhaustively.
C Monitor/verify the content and technical quality of the filmmakers' product at intermediate stages.
C Monitor/verify the content and technical quality of the final original-language version and each additional language version before starting duplication; monitor duplication.

d. Distribution; the impact of the films

_Showing these on TV means that young kids will grow up with a good attitude about AIDS._

Ousmane, aged 24, student at the University of Ouagadougou, Burkina Faso and focus group participant

_ Because of taboos it_s not easy for us to buy condoms, especially us women. I have no doubt that an awful lot of people have been helped to overcome once and for all their shame over buying condoms thanks to the film _The Shop__ (a film by Idrissa Ouédraogo based on an idea by Olga Ouédraogo).

Elizabeth, aged 21, focus group participant, student at the University of Ouagadougou, Burkina Faso

(This section includes ideas on monitoring and evaluation of both the distribution process and of the impact of the films themselves.)
Some general comments
C Remember that any evaluation of the impact of the films is in essence an evaluation of the success of all project phases, from initial planning to distribution.
C Pre-testing should have been so exhaustive that there is practically no chance that you will encounter any major negative surprises on a substantive level. It should also have almost totally precluded the possibility of causing any harm with the films. Pre-testing might also have revealed some interesting approaches to specific elements of the evaluation process and helped you to discover some intriguing questions to ask at this time.
C Be realistic when formulating your indicators. Different channels of distribution will have potential for different kinds of impact. Mass-media broadcasts are likely to result in the greatest numerical reach. Distribution channels that have potential for community-level reinforcement of messages are likely to have far greater depth of impact.

Consider, for example, whether it is realistic to expect television broadcasts to bring about dramatic changes in behavior. A televised anti-smoking campaign is highly unlikely to bring about a dramatic change in people_s behavior, such as a reduction in smoking that would register in a nationwide survey. What a survey might detect is an intermediary step towards behavior change, for example, an increased awareness across the population exposed to the campaign of the risks smoking poses to one's health. On a societal level, this increased awareness might make the general social environment less tolerant of smoking. While this may represent an added incentive to smokers to quit, it is likely to take many years to take maximum effect. On an individual level, awareness of risks is only one step in a whole series that may lead the smoker to quit. You might want to consult some documents on theories of behavior change when formulating your evaluation strategy.

To sum up, it is generally accepted that mass media alone can raise awareness, generate discussion and increase knowledge. It can also lead to some shifts in attitudes and practice. It is, however, face-to-face communication (e.g., the use of the compilation tape by local organizations to provoke debate, providing an opportunity to respond to individual questions) that is likely to lead to the greatest changes in these areas.

C Be realistic about what can be attributed to your project. Are the changes you are trying to monitor really attributable to the Scenarios films, or are you momentarily "forgetting" the impact of all the other HIV-related activities that have been carried out in the project zone?
A few suggestions for evaluation strategies
Monitor the success of the distribution strategy on a quantitative level by observing:
C the number, timing and scheduling of TV broadcasts and projected number of viewers. In some countries, broadcasters may have access to electronic ratings data. Alternatively, you may want to explore the numerical reach of your distribution strategy by means of surveys.
C the number of showings at cinemas, timing and billing, projected number of viewers;
C the number of compilation cassettes distributed, to whom, frequency of use, estimate of number of people reached.

You may want to map the distribution network, and share this information with others who produce health-related audio-visual materials.

Evaluate the impact of films from a qualitative perspective or combine quantitative and qualitative approaches:
C You may choose to conduct surveys in selected sites to assess the reach (numerical, geographical, sociological) of the films, and audience recall, approval and appropriate interpretation of messages. Respondents could also be asked if the films generated discussion (and with whom) or reflection on HIV/AIDS; if the films made them feel more positive towards HIV/AIDS-prevention strategies or more understanding of those living with HIV/AIDS; if they did anything different as a consequence of seeing the films. You may wish to use stills from the films to jog people_s memory.

C Focus-group discussions could be conducted in a variety of milieux to allow in-depth investigation of the impact of the films and to permit exploration of the findings of the surveys (above). They might explore, for example: under what viewing conditions the films have greatest impact; the nature and depth of discussion generated by the films (for example, with children, with sexual partners); perceptions of the broadcast schedules, etc.
C It is through the use of the compilation cassette that we expect the Scenarios from the Sahel audio-visual resources to have their greatest and most sustainable impact _ not least in rural areas, where access to television is limited. The compilation video could be accompanied by an evaluation-oriented questionnaire, along with the request that it be completed and returned to the project team. NGO and CBO partners will be asked in a survey to assess the value of the compilation tape to their own work. They will also be asked to indicate how often they are using the tape and the number of people they are reaching with it. A log of demand for and distribution of the tape in its various language versions will be kept.

The effectiveness of the compilation tape as a teaching resource for use in small-group sessions can also be assessed experimentally by means of the pre- and post-intervention KAP questionnaires routinely used by the larger agencies. This should allow for direct comparison with resources previously in use.

C If you chose to use a time series survey, this too will give you feedback on the impact of broadcasts of the films and use of the compilation cassette.
C Once distribution is well underway, you might want to consider bringing in an outside evaluator to examine not only the impact of the films, but also to pursue additional objectives, such as measuring the synergies and partnerships created during the project, evaluating the medium-term impact of the contest, and studying the significance of the entire project for the core structures.