e. The selection process: model methodologies

The selection methodology we are proposing here is for a hypothetical group of 1,000 scenarios contributed by young people in the course of the contest.
In that imaginary contest, young people were invited to write on any of 9 suggested topics. Numbers 1-3 had to do with condom use; 4-6 had to do with abstinence; 7-9 had to do with socio-economic consequences of the epidemic. Young people were not required to write on one of those topics; they could write about anything at all related to HIV/AIDS.
In this model, we are assuming that the selection team aims to choose ten winning scenarios that will subsequently be turned into short films on HIV/AIDS.
The team will follow a two-phase procedure:
1) Pre-selection, during which a team of 20 jurors will select 60 semi-finalists out of the 1,000 contributions. (Duration: Once the jury convenes, this phase would take a total of five days, two of which would involve jurors reading scenarios independently and three in discussion/debate in the meeting room.)

2) Final selection, during which 8 jurors choose the 10 winners from among the 60 semi-finalists. (Duration: Once the jury convenes in the meeting room, final selection would last a minimum of four days, ideally more.)
The methodologies proposed for the two phases differ from one another. By presenting two different models, we hope to provide you plenty of ideas to draw on when developing your own selection strategy.

i. Pre-selection

STEP 1: Draft all documents you might need for the pre-selection jury (as discussed above) and make the requisite number of photocopies.
STEP 2: Sort and number the contributions so as to facilitate both the selection and archiving processes.
This step should be carried out by a team of project partners before the jury itself convenes.
In the contest, young people were given nine selected topics to choose from plus the freedom to write about any other pertinent topic of their choice. The scenarios are sorted into ten piles: one pile for each of the suggested topics, plus pile number ten for the free-choice scenarios. The sorting team writes two numbers (clearly and legibly) in the upper right-hand corner of each scenario. These will be the identification numbers for those scenarios during the selection process and in the archives.
The first number, written in red, corresponds to the number of the topic selected (or "10" for free-choice). Then, the scenarios within each topic are given a number in black, starting with the number one.

Imagine that 87 scenarios were written on topic 2, which has to do with negotiations within couples on condom use. Each of them will have a red number 2 in the upper right, and they will be numbered in black from 1 to 87.
STEP 3: Arrange the scenarios into (nearly) equal-sized stacks
This step also comes before the jury actually convenes.
You have determined that the pre-selection team will be made up of a total of 20 people and have carefully selected those individuals.

Early on in the grading process, each juror will be given a stack of approximately 50 scenarios to read and grade. Those stacks must be prepared now.
Try to keep the scenarios on a given topic in the same stack as much as possible. That way, jurors will not be asked to compare apples and oranges, but rather scenarios on the same subject.
For example, imagine that exactly fifty scenarios were written on topic number one. Keep those fifty together in one stack, put them in a large folder, and label it "STACK A".
There were 87 scenarios written on topic 2. Put 44 of them in a large folder and label it "STACK B". Put the remaining 43 into another folder labeled "STACK C". Of course, this does mean that at least one of the remaining stacks will have over 50 scenarios in it.
This task of sorting becomes slightly more complicated if, for example, there are only 14 scenarios on topic 3. In such an event, you will have to create a mixed stack and will have to bear this in mind later, when you ask the graders to decide which of the scenarios in that stack will be selected as semi-finalists. More on that in just a minute.
STEP 4: Pair jurors with one another
Before the jury convenes, you need to sit down and figure out the best possible juror pairings.
Every stack of scenarios is read by two people. Near the end of the selection process, those two people will meet to discuss that stack of scenarios and determine which ones will be designated semi-finalists.
Early in the pre-selection process, each juror will have a stack of roughly fifty scenarios to grade on his or her own. Then, at _half-time_, all the jurors come together, the stacks are redistributed (and not simply exchanged between two people), and then the jurors break up for another round of individual grading. This means that each juror will actually be a part of two pairings. For example:

STACK A is read during ROUND 1 by KRISTEN and during ROUND 2 by BOBBY.

STACK B is read during ROUND 1 by BOBBY and during ROUND 2 by MAURA.

STACK C is read during ROUND 1 by MINA and during ROUND 2 by KRISTEN.

After both rounds of individual grading are completed, the pairs get together to compare notes, debate and reach decisions as to semi-finalists. That means that, in Kristen_s case, things would look like this:

SESSION 1: KRISTEN and BOBBY discuss stack A, and later:

SESSION 2: KRISTEN and MINA discuss stack C, whilst BOBBY and MAURA discuss stack B.

During SESSION 1, MAURA and MINA each discuss another stack of scenarios with the second juror they have been paired with.

The slightly complicated (but most interesting!) task that you face before selection begins is pairing people in an ideal manner. The two people in a pairing should complement each other well. For example, do not put two audio-visual specialists together in a pairing, but rather have them work together with people whose expertise is in HIV/AIDS.
DANGER: In planning your pairings, try to avoid _triangles_ that will complicate things later on by forcing you actually to hold three sessions for discussions in pairs rather than two. A time-wasting _triangle_ would look like this:
STACK A: round 1: BOBBY round 2: KRISTEN
STACK B: round 1: KRISTEN round 2: MAURA
STACK C: round 1: MAURA round 2: BOBBY
During the first round of discussions, Bobby and Kristen discuss stack A, and what does Maura do? Nothing. You would have created a situation in which the entire jury would have to wait around while this triangle finished its three discussions rather than two. (You might have guessed by now that we did indeed create this problem for ourselves.)
STEP 5: The jury convenes: Conduct the introductory session
Do not forget to have an outside rapporteur present at all deliberations.
This introductory session includes:

C A general welcome and introduction of the chairperson
C Introductions of the jurors
C Discussion and adoption of the day_s agenda and the timetable for the entire session
C Overview/update of the project

C A presentation of the documents you have prepared as the basis for collecting juror feedback in the course of the selection process.
C An overview of the methodology to be used for selecting the winners

STEP 6: Determination of selection criteria
We would recommend devoting at least two hours to this task. It is absolutely essential that everyone understand in exactly the same fashion what the criteria for selection actually are.
Furthermore, we would recommend that the criteria be determined through consensus-oriented dialogue among the pre-selection jurors. That consensus might be very difficult to come by, as different people have a different sense of what characterizes a _good_ short film on HIV/AIDS.
In the course of Scenarios from the Sahel, juror debate on this topic was impassioned, sometimes heated, and absolutely fascinating. The breadth and dynamism created by vibrant exchanges between filmmakers and HIV specialists, between individuals inclined to emphasize the emotional and those naturally leaning toward approaches based on maximum information delivery, was exhilarating. The debate allowed jurors _ with their highly diverse professional backgrounds and radically different characters _ to develop their own, unique style of consensus-building, something that would serve them well on countless occasions during the intense days to come.
Each of the Scenarios from the Sahel juries did manage to come to an agreement on this matter, but they felt that the term _criteria_ sounded too restrictive. They preferred _factors to bear in mind during the selection process_. You will find the relevant framework document developed by the Scenarios from the Sahel juries in Appendix Three. Key _factors to bear in mind_ included: impact (personal identification, emotiveness, ability to trigger behavior change), creativity, constructiveness (i.e., ability to foster solidarity for people living with HIV and to safeguard cohesion among all members of the AIDS-prevention community), educational qualities, and potential.
The concept of potential was the subject of much debate. It was emphasized that the _script-doctors_ and production teams would be able to create a film based on certain, selected ideas contained within a scenario (as opposed to the entire scenario) and that minor errors in the scenario could be edited out (such as inappropriate vocabulary, small misinterpretations of scientific facts associated with HIV, etc.).
At the final, regional selection of Scenarios from the Sahel, we prefaced the debate on criteria and gave the discussion a realistic framework by showing the jurors several short films from the series 3,000 Scenarios Against a Virus. Before watching and discussing the films, we explained that we had a certain number of objectives in that exercise:

C Highlight the variety of topics and approaches;
C Illustrate the technical possibilities available to professional script-doctors and filmmakers;
C Show examples of different ways to reach out to people on an emotional level and to discuss the importance of this in the area of HIV prevention;
C Emphasize to the jury that a scenario is actually a point of departure and not an end in itself; underscore the role of a professional script-doctor in reworking the scenario in collaboration with the young author;
C Motivate the jurors by showing them what is possible;
C Expose the jurors to the range of styles and emotional tools available: irony,
humor, song, dance, empathy, uncertainty...

After each film, the tape was stopped, and the jurors were invited to respond to three questions:

C What effect does this film have?
C What is the film_s source of impact?
C What makes the film so good?

Those discussions allowed the jurors to center their thoughts on a very practical level and expanded their horizons before starting actual deliberations.

In the model methodology we are presenting here, the criteria adopted by the pre-selection jury are to be presented to the final jury, which will debate them and adopt them by consensus.
STEP 7: Determination of a grading system (point system)
However uncomfortable you may feel with the idea of allotting quantitative scores to creative works, there is really no way around it in a large-scale contest. It alone makes it logistically possible for jurors to compare their impressions of around one hundred scenarios, providing a starting point for the discussions at which the most difficult selection decisions will be made. But don_t forget this is only a point of departure, and one essential element of a rich and multifaceted selection strategy.

The Scenarios from the Sahel juries came to the conclusion that it is not at all helpful to establish and follow a rigid grading barometer, such as:
_10 points possible for impact
15 points possible for creativity
.... ...= 50 points total._

Rather, the juries felt strongly that a scenario should be looked at in a global, integral fashion, and all the while a juror should recall _ and frequently refer to _ the _factors to bear in mind_.
In the end, jurors graded scenarios on a scale of 0 to 50, with 50 being the best possible score. Some said they felt uncomfortable for a moment due to the absence of a set, detailed barometer, but everyone soon realized that the diversity of the scenarios (length, type of presentation, style...) was such that a rigid barometer would have complicated the grading process and would surely have led to situations whereby some highly appreciated scenarios would regrettably have been discarded. It would have been like a boxer who dominates an entire bout with flair, panache and courage, but ends up losing on points.
STEP 8: Jurors test-grade a few scenarios and discuss their experiences
Once the jurors have agreed on criteria and a grading system, have them test-grade a few scenarios in plenary.
Give each person a copy of the same scenario, and ask them to take ten minutes on their own to read and grade it. Then, ask each person to say what their grade is and how they arrived at that particular score. Ask them to talk about specific difficulties encountered in the grading process and how one might overcome those difficulties. You can complement that discussion by recounting the difficulties experienced by the group that pre-tested your selection methodology, as well as those of the Scenarios from the Sahel jurors.
During Scenarios from the Sahel, jurors cited the following challenges:
C Many of the scenarios were hard to understand due to: poor handwriting, awkward use of language, or ideas started but not finished (so the jurors had to guess at what was meant).
C A few of the scenarios stopped abruptly because the participant had not managed to bring the story to a close within the 10-page limit. Again, jurors had to try to guess at where the story might be going.

C Obvious plagiarism.
C The difficulty of reconciling one_s desire to reward effort and the imperative of providing the AIDS-prevention community with the most useful films.
C Maintaining the same spirit, focus and level of objectivity throughout the entire process.
C Not having a clear sense of what grades to assign to the first scenarios one reads; having to go back and re-grade the first several scenarios.

For this last problem, juror Fatim Dia of Senegal formulated a strategy that received high marks itself. She said that she pre-tested her own grading strategy on a group of 10 scenarios, then reevaluated her strategy and went back to start over again. She said that that pre-testing procedure took her about an hour and a half.

STEP 9: Jurors read, grade and rank a first stack of scenarios
The jurors receive a stack of scenarios as well as individual grading sheets. The grading sheets should include the following elements:

C A space for the name of the juror
C A space for the letter of the stack being corrected
C A table with approximately 60 rows (one row for each scenario; some stacks might have up to 60 scenarios depending on how the sorting process turned out) and four columns: one for the identification number of the scenario (made up of the red and the black numbers in the upper right-hand corner of the scenario), a large column for the juror_s comments, a column for the juror_s grade, and the ranking of that scenario relative to the others in the stack (first, second..... forty-ninth...).

The ranking column will, of course, be completed once the entire grading process is complete. If two scenarios are tied for, say, first place, then there will be no scenario ranked second: the scenario with the next highest score will be ranked third.
The jurors are given at least one full day (or the equivalent, spread out over three or four evenings _ the jurors could do the grading after their regular workday) to read and grade their stack. Reading and grading 50 scenarios is at least one full day_s work, if it is done conscientiously.

Remember to emphasize that the jurors must take special care of the scenarios. It would be tragic if any were to go astray and a betrayal of the jurors_ _contract of conscience_ with the participants.
STEP 10: The jurors meet in plenary to exchange stacks of scenarios and to share their observations
This two-hour session, sandwiched between the two rounds of individual reading/grading, is a chance for the jurors to talk about their general observations on the scenarios they read, the difficulties they encountered and the solutions they discovered. At this stage, jurors should avoid saying anything about specific scenarios that might influence the thinking of the person who is about to read that same block of scenarios.
STEP 11: The jurors, working independently, read, grade and rank their second stack of scenarios
STEP 12: The jury reconvenes; discussions in pairs are conducted
We suggest that the tasks set out in STEP 12 be carried out in a 4-hour session in the morning, followed by lunch and the activities of STEP 13 in another lengthy session in the afternoon.
At this point, all of the scenarios have been graded by two different people. Now is the time for those two jurors to sit down together and come to a decision about which of the scenarios in their stack will be designated semi-finalists and sent on to the final jury.
Three semi-finalists will emerge from each stack. As there are twenty stacks, there will be a total of 60 semi-finalists.
After listening to an explanation of the procedure and receiving the necessary documents, the two jurors use a combination of calculation and discussion to select the three semi-finalists. The procedure used in Scenarios from the Sahel, presented in the following paragraphs, was much appreciated by the jurors. Remember that the procedure will be carried out twice, as each juror is a member of two different pairings.
The jurors fill out the _Comparison Table_ (see Appendix Four). This is where the jurors_ rankings come into their own. This table will reveal if, by very good fortune, the two jurors have independently ranked the same three scenarios in the first three places. If they have (in any order), their work is done. If they have not (which is almost always the case), let the debate begin!

The ranking column has another crucial advantage. It counteracts imbalances which might result from jurors_ varying generosity in their grading systems: one juror might be unwilling to go higher than 40 for a favorite scenario, whilst another might happily give a good scenario 50 points.
The _Comparison Table_ will show which three scenarios are top-ranked based on the average of the two jurors_ scores. Next to that ranking by average, the table shows how each scenario was ranked by each juror. This could reveal a situation whereby, for example, the top-ranked scenario by average received a super-high score by the first juror (who ranked it first by far), and merely relatively good score by the second juror (who ranked it seventh and is far keener on his or her own favorites). In this situation, it is up to the two jurors to discuss the scenario(s) in question, argue its strengths and weaknesses, and urge the other to alter his or her position. The two discuss, argue, shake their fists and jump up and down until they come to an agreement on which three scenarios they would like to designate as semi-finalists.
Several of the members of the pre-selection committee in Senegal said that these debates were the richest, most profound, most thought-provoking and eye-opening exchanges they had ever had on the subject of HIV/AIDS. Some of those debates resembled the sparring of a ferociously determined prosecutor and a clever, counter-attacking defense lawyer. Others were more like lively squabbles between brother and sister. None were anywhere close to boring.
When the two jurors have reached a decision, they inform the rapporteur, who records results as they come in.
Selection of the 60 semi-finalists is completed once all 20 pairings have selected their 3 winning scenarios and informed the rapporteur.
(A technical detail: If the stack is of mixed composition, e.g., if the stack includes all 14 of the scenarios written on topic 3 plus 30 of the scenarios written on topic 4, the pair of jurors who graded that stack should complete two separate comparison tables, one for each topic, and be instructed to select one semi-finalist from among the scenarios written on topic 3, and two semi-finalists from among those on topic 4. In this way, proportionality by theme is maintained, and the final jury is assured to have the opportunity to consider a scenario on topic 3.)
STEP 13: The jurors present their observations and recommendations and discuss evaluation of the contest and of the selection process, documents relating to subsequent project phases, and any other documents you distributed with a view to collecting juror input

We discovered that this forum at the end of a selection process was one of the most fruitful and fulfilling experiences of the entire project. The atmosphere, especially in light of the morning_s debates, was electrifying. Useful ideas and insights flew in all directions for hours on end.
Take this opportunity to collect the input-seeking documents you distributed to the jurors at the beginning of the process, and debate the subject matter of those documents with a view to fostering an exchange among the jurors that will surely take the discussion to ever more profound levels. If the jurors have no objection, record this session.
STEP 14: Thank the jurors and close the session, ideally with a dinner
STEP 15: In collaboration with the rapporteur, draft the final report of the meeting and summary reports on each of the input-seeking documents you had distributed to and discussed with the jurors. Circulate these to the jurors.

There are two parts of the final report that will be of great importance to the final jury, so you might want to give them top priority:

1) The list of the semi-finalist scenarios, along with
2) the pre-selection jurors_ comments on each of those contributions, and
3) a detailed document on the selection criteria established by the pre-selection committee and to be debated and adopted by the final jury.

Once a draft pre-selection report has been completed, submit it to the pre-selection jurors themselves for comment before drafting the final version.