B22. August 11, 1993

Dear Bob,

Your mention yesterday of your text and your upcoming course in case study research methods got me thinking again about the process which has evolved in an evaluation I'm working on. The State of Washington office of the Superintendent of Public Instruction (OSPI) awarded a grant to Cap Peck and I to evaluate a grant program by which the state promoted the inclusion of preschoolers and kindergartners eligible for special services in regular classrooms. I thought your readers or students might find the issues of this study a useful.

Cap and our research assistant, Janet Curley, have had no formal training in qualitative methodology. They were eager to learn my methods, as I was theirs. But we struggled at virtually every turn to get past our habits and inclinations. At the outset, Cap's design before I arrived at WSU called for telephone interviews with grantees and one day site visits at 20 of the 150-or-so grant sites, these to gather data from which to construct a questionnaire which would be statistically analyzed and reported. The qualitative data was intended to improve the quantitative instrument. At our first meeting, I convinced them to expand the design, including case studies and a review of the data by an expert panel (which included Lizanne DeStefano). I saw the survey as a means of discovering interesting case study sites and as needing qualitative amplification and interpretation. Cap saw the case studies as another step building toward a better survey. Discussions about chronology were our fora for trying to persuade each other of the relative importance of our methods--Which comes first: the survey or the case study? I was so convinced the case studies would provide the richest and most useful information that my beloved colleagues began a running gag in which they gigglingly reduced my persistent arguments to: "It's just a survey."

Another running gag mimicked my continuing attention to timelines (I know you won't believe that): "Don't whine, just do it." I swear I never said that, but I did worry about preserving adequate time in the field and at the computer. In the end, time was in fact a serious enemy: 14 of the 20 site visits were completed, too many foisted upon Janet, I thought; three of the four case studies were done, Cap allocating only four days to write his; the survey was so poorly pilot-tested, it's an exaggeration to use that term; and over my objections, Cap assigned the major writing responsibility of the report to Janet.

We were all baffled by another non-meeting of the minds, this one about transcriptions of interviews. Cap hired transcribers to type up every conversation from every site visit, believing this was "rich data" which could be mined over and over again. I doubted we'd have/take time to pore over these transcripts. And I was sure the case studies would be richer. Cap and Janet couldn't believe I, the "qualitative one," could fail to be enamoured of transcriptions. And, given that I wasn't for the site visits, they were stunned when I wanted transcriptions from the case studies so badly I spent 9 consecutive 12-hour days typing them up myself. (The transcriber couldn't have gotten to them until after the cases were to be sent to the panel because she was still inundated with tapes from the site visits.) Nor did they later want transcripts of the panel's discussion; I did.

Analysis also gave rise to friendly conflict. Their plan: reread every transcript and every note, identifying and categorizing themes. The themes would become findings. The more frequently the theme appeared in the data, the more robust. It was a coding and counting strategy. My plan: interpretive discussion of data from case studies, site visits and telephone interviews (and survey responses and correlations, had they arrived in time) identifying issues to focus our analysis. I offered as guiding questions: What is important to understand? Why? How important is it? What are we neglecting? Are we overemphasizing? It was a holistic, interpretive strategy. I worried that, if we followed their plan, our thinking would never rise above the lowest level of our data. And I disagreed that the frequency of our recognition of a theme in the data was a measure of robustness. What of important, unstated themes? What of the constraints of early coding categories which fossilized and narrowed our perceptiveness? What of the limits of expertise in recognizing and coding, especially since this task fell primarily to Janet, who unfailingly came to meetings with piles of cards in rubber bands? Cap and Janet worried that, if we followed my plan, we'd lose pieces of data or themes and stray too far from the data, become ungrounded, over interpret.

Another conundrum: reporting. Cap had used Bronfenbrenner's 1979 levels of ecological analysis (macrosystem, exosystem, mesosystem, and microsystem) productively for developing and presenting findings, and he wanted to do so again. I agreed this was a useful analytic tool for our study. Early, I suggested structural functionalism, conflict theory, and symbolic interactionism in addition, but I didn't push these conceptual frameworks. I concentrated my efforts on developing and structuring the reports around the issues. As you have had trouble in explaining what you mean by issues in your case study courses, I failed to make the distinction between themes and issues comprehensible to Cap and Janet. We tried to maximize the benefits of our two styles in a tacit compromise: Cap and Janet continued to try to identify issues as a kind of culminating academic step; I was willing to use the Bronfenbrenner framework for structuring a presentation to the Early Childhood Development Association of Washington-OSPI conference and for the report to the state. But an interesting thing happened at the meeting where we parceled out writing tasks among ourselves. (You'll like this part because it involves a Venn-type diagram.) We talked about issues and about how to locate them on our concentric circles of visualization of Bronfenbrenner's levels. The plan was to have a section of the body of the report for each ring of the concentric circles. The issues cut across various levels. We visualized the issues as radii dividing the levels' ring into pie-shaped pieces. The issues would be discussed as parts of each level-ring section. As we were agreeing to his plan, Janet saw that the body of the report might be sectioned into the pie-shaped pieces, rather than the rings, with the levels being discussed as parts of each pie-shaped price. When she asked why I was smiling, I said, "You've just described an issues structure for the report." Cap then grasped my argument that structuring around the issues was an emic strategy, whereas structuring around Bronfenbrenner's levels was an etic strategy. In the end, we married the two. How important was the change--issues and Bronfenbrenner rather than Bronfenbrenner and issues? I thought it a good choice--but of course, I would.

We're meeting next week to discuss our (Janet's) first draft. I'm not sure I've contributed usefully to this project. I believe the case studies provided the most illuminating data, but perhaps we would have gotten more information from the survey had we concentrated out efforts there instead. (Of course, I don't believe a better survey would have enhanced our understanding as these or better case studies). I think it was helpful to identify issues, but perhaps themes would have served. Our analysis discussion carried us into fundamental concerns about ideology, ontology, values, conceptualizations, power. Stimulating and provocative, but every discussion with Cap Peck could be so described. I can say this project has contributed usefully to me. I know more about special education and early childhood, about inclusion, about the state context--and about the appeal and obviousness of certain methodological choices for me and about their lack of appeal and obviousness to others. Like so much knowledge, this is sobering.

Linda (Mabry)