B2. RESPONSIVE EVALUATION

Responsive Evaluation is an approach to the evaluation of educational and other programs. Compared to most other approaches it is oriented more to the activity, the uniqueness and the social plurality of the program.

The essential feature of the approach is a responsiveness to key issues or problems especially those recognized by people at the site. Its design is slowly developed, with continuing adaptation of evaluation goal-setting and data-gathering while the people responsible for the evaluation become acquainted with the program and the evaluation context.

Issues are suggested as "conceptual organizers" for the evaluation study, rather than hypotheses, objectives, or regression equations. Issues are organizational perplexities or problems. The term "issues" draws thinking toward the complexity, particularity, and subjective valuing already felt by persons associated with the program. (Examples of issue questions: Are the admission criteria appropriate? Do these simulation exercises confuse the students about authoritative sources of information?) People are concerned about one thing or another (or likely to become concerned). The evaluator inquires, negotiates, and selects a few issues around which to organize the study.

To become acquainted with a program's issues the evaluator usually observes its activities, interviews those who have some stake in the program, and examines relevant documents. These are not necessarily the data-gathering methods for informing the interpretation of issues; but are needed for the initial planning and progressive focusing of the study. And even later, management of the study as a whole remains flexible--whether quantitative or qualitative data are gathered.

OBSERVATIONS AND JUDGMENTS

A responsive evaluation study is, of course, directed toward the discovery of merit and shortcoming in the program. It is attentive to multiple and sometimes contradictory standards held by different groups.

Ultimately the evaluators should make summary statements of the program's worth. But first they may provide descriptive data and the judgments of others so that report readers can make up their own minds about program worth.

There is a common misunderstanding that responsive evaluation requires naturalistic inquiry or qualitative research. Not so. The evaluators and program staff and evaluation sponsors discuss alternative methods. They negotiate. Knowing more about what different methods can accomplish, and what methods this evaluation "team" can do well, and being the ones to carry them ouL the evaluators ultimately determine what the methods will be. Preliminary emphasis often is on becoming acquainted with the history and social interactions of the program. It may be phenomenological, other times quantitative, possibly goal-oriented. Method depends on the situation. For it to be a good responsive evaluation the methods must fit the "here and now," having potential for serving the evaluation needs of the various parties concerned.

Even so, it has been uncommon for a responsive evaluation study to emphasize the testing of students or other indicators of successful attainment of objectives. This is because such instrumentation has so often been found simplistic and inattentive to local circumstances. Available tests are often not good approximations of the several outcomes intended. And even when possible, developing new tests is very expensive. Test results have too often been disappointing, with educators probably justifiably believing that more was learned than showed up on the tests. With the responsive approach, tests may be used, but usually are kept in a subordinate role. They are needed when it is clear that they actually can serve to inform about the quality of the program.

People are used more as sociological informants than as subjects here. They are questioned not so much to see how they have changed but to indicate the changes they see.

SUBJECTIVITY AND PLURALISM

Those who object to the responsive approach often do so on the ground that too much attention is given to subjective data, e.g., the testimony of participants. For description of what is happening the evaluation researchers try (through triangulation) to show the reliability of observations. Part of the description, of course, especially that about the worth of the program, is revealed in how people subjectively perceive what is going on. Placing value on the program is not seen as separate from perceiving it.

The researchers' own perceptions too are recognized as subjective, in choosing what to observe, in observing, and in reporting the observations. One tries in responsive evaluation to make those value commitments more recognizable. Issues, e.g., the importance of discovery learning, are not avoided because they are inextricably subjective. When reporting, care is taken to illuminate the subjectivity of data and interpretations.

Objection to a responsive approach is also expressed in the belief that the program staff, the funding agency or the research community should specify the key questions. Their questions often are worthy of study, but in program evaluation for public use, never exclusively. There is general expectation that if a program is evaluated, a wide array of important concerns will be considered. Embezzlement, racial discrimination, inconsistency in philosophy, and thwarting of creativity may be unmentioned in the contract, and barely in the evaluation specialist's range of view, but all such shortcomings belong to the evaluation expectation, and the responsive evaluator at least tries not to be blind to them.

Further, it is recognized that evaluation studies are administratively prescribed not only to gain understanding and inform decision-making but to legitimatize and protect administrative and program operations from criticism, especially during the evaluation period. And still further, ..that evaluation requirements are sometimes made for the purpose of promulgating hoped-for standards.

By seeking out stakeholder issues, the responsive evaluator tries to see that these efforts at extending control over education are not undermining legitimate interests. Responsive evaluation is not intended as an instrument of reform, though reformists might find it useful. It is intended to serve the diverse people most affected personally and educationally by the program at had - though it is bound to produce some findings they will not like.

ORGANIZING AND REPORTING

The feedback from responsive evaluation studies is expected to be in forms and language attractive and comprehensible to the various groups. Thus, even at the risk of catering, different reports may be prepared for different groups. Portrayals and verbatim testimony will be appropriate for some, data banks and regression analyses for others. Obviously a budget will not allow everything, so these different communications need to be considered early in the work.

It is not uncommon for responsive evaluation feedback to occur early and throughout the evaluation period, particularly as a part of refining the list of issues to be pursued. The evaluator may ask, "Is this interesting?" and might, based on the answer, change priorities of inquiry. As analyzed by Ernest House (1980, p. 60) responsive evaluation can be considered "intuitive" or indeed subjective, closer sometimes to literary criticism, Elliot Eisner's connoisseurship, or Michael Scriven's Modus operandi evaluation than to the more traditional social science designs. But it differs from them in the most essential feature, that of emphasizing the issues, language, contexts and standards of stakeholders.

When I proposed this "responsive evaluation" approach at a conference at the Pedagogical Institute in Göteborg, Sweden, in 1974, I drew particularly upon the writings of Barry MacDonald, Malcolm Parlett and David Hamilton, all stressing the necessity of organizing the evaluation of programs around what was happening in classrooms and boardrooms, drawing more attention to what educators were doing and less attention to what students were doing.

It is difficult to tell from an evaluation report whether or not the study itself was "responsive." A final report seldom reveals how issues were negotiated and how audiences were served. Three examples of studies which were clearly intentionally responsive were those of Stake and Easley, MacDonald, and Murray indicated in the references below.

Prepared for Tübingen Workshop, 1996, Stake

REFERENCES

House E. 1980. Evaluating with validity. Sage Press, Beverly Hills.

MacDonald B. et. al. 1982. Bread and dreams, CARE, University of East Anglia, Norwich.

Stake R. E. 1980. "Program evaluation, particularly responsive evaluation." In Dockrell W. B., Hamilton D., Rethinking educational research. Hodder and Stoughton, London.

Stake R. E., Easley J. et. al. 1978. Case studies in science education. University of Illinois. Urbana. 16 volumes.

Shadish W. R., Cook T. D., Leviton L.C. 1991. Foundations of program evaluation. Sage Press.