Multiple methodologies for evaluating computer uses in education

• progress reports due today
• digital video tutorial at 11am in Room 28 Ed Bldg

.

.

.

.

.

.

Multiple methodologies

.

.

.

.

The Range of Methodologies

• Naturalistic observation
• Participant observation
• Interview
• Survey/Questionaire
• Quasi-experimental
• Experimental

.

.

.

.

Kinds of evaluation

• Summative evaluation
• Formative evaluation

.

.

.

.

Choosing a methodology

• What are your goals?
• How much do you know about what is happening?

.

.

.

.

Examples of effective observation

• The "space bar" example
• The "DoIt" example

.

.

.

.

The "space bar" example

• Push the space bar to continue.
• The students expected...

.

.

.

.

The "DoIt" example

.

.

.

.

.

.

.

.

And the user read "DoIt" (DO IT) as "Dolt" (DOLT)

dolt: a stupid person

.

.

.

.

How to run your own web server

• Turning this computer into a web server
  • Personal web serving
  • Heavier duty web serving: "Recipes" from Web66

Process data

• Web logs:
  03/27/00 12:49:49 OK 128.174.91.148 :default.html 34
• LRS web logs
• EPS 304 White Paper Nov 99 log

Some methodological suggestions

• survey or questionaires
  • rank order and then reduce the number of questions
  • "debug" your questions - test them on a number of people before using them
  • end with some open ended questions
    • What is the best thing about this?
    • What is the worst thing?
    • What improvements would you suggest?
• observations
  • don't make suggestions, only respond to help questions, not volunteer help
  • whenever the user asks for help, note that down after giving help
  • write notes, both during and after
  • can followup observations with a survey at the end
  • for preliminary studies, use audio tape or video tape as a backup source of data to written notes