Robert L. Morgan
Department of Special Education & Rehabilitation
Utah State University
Logan, UT  84322-2865
801-797-3251
bmorgan@cc.usu.edu

I. Development, Evaluation, and Dissemination of a Motion Video, CD ROM-Based Consumer-Driven Job Preference Program

Model Demonstration Project H158V70037

Final Report

September 7, 2001

II. Project Summary

This final performance report is submitted upon completion of a grant award provided by Model Demonstration Projects to Improve the Delivery and Outcomes of Secondary Education Services for Students with Disabilities (84.158V). Starting date for the project was October 1, 1997. With approval from the Office of Special Education Programs, Secondary Educational Institute, the project received a no-cost extension through September 2001 (see Appendix A).

This project proposed to develop, evaluate, and disseminate a program that provides youth with a realistic representation of various jobs using motion video CD ROM. Consistent with the initial grant proposal, specific goals were to (I) identify the critical attributes of jobs commonly held or potentially attainable by youth with disabilities, (II) develop a prototype video CD ROM based on critical attributes of jobs identified in GOAL I research and conduct preliminary field testing, (III) revise the program and conduct the main field test, (IV) revise the program and conduct the operational field test, and (V) disseminate the program to secondary special education, rehabilitation, and supported employment programs nationwide.

The original project was proposed for a timeframe from September 1997 through August 2000. Appendix A presents the Grant Award Notification which adjusted the project period one month forward, i. e., from October 1997 through September 2000. An approved no-cost extension (see Appenddix A) provided for additional project activity through September 2001. Project activities were concluded in January 2001. Timelines shown in parentheses below reflect this adjustment of one month.

III. Project Status

This section describes project activities, accomplishments, and outcomes related to goals and objectives. Table 1 presents a summary of project activities. Significant changes from the original project timeline, and explanations of the changes, appear in italics below.

GOAL I. Identify the Critical Attributes of Jobs Commonly Held or

Potentially Attainable by Youth with Disabilities

GOAL I - Objective 1: Survey Supported Employment Program Providers and Transition Programs to Identify Jobs Held by Youth with Disabilities and Those Jobs Potentially Attainable

Background literature on job placements. Prior to conducting the job placement survey, project staff reviewed numerous studies on community job placements of youth and adults with disabilities (e.g., Affleck, Edgar, Levine, & Kortering, 1990; D’Amico & Blackorby 1992; Colley & Jamison, 1998; Dunn & Shumaker, 1997; Hoish, Karen, & Franzini, 1992; Johnson, McGrew, Bloomberg, Bruininks, & Lin, 1997; McNeil, 1993; Sitlington, Frank, & Carson, 1991). These studies organized job placements based on the Dictionary of Occupational Titles (DOT: U.S. Department of Labor, 1991) or the Occupational Outlook Handbook (U.S. Bureau of Labor Statistics, 1998). For example, in the National Longitudinal Transition Study, D’Amico and Blackorby (1992) assessed hours worked, wages and benefits, workers’ satisfaction, length of time out of high school, and type of employment placement. They discovered that, of employed youth with disabilities out of school less than 2 years (N = 1,087), 24.3% worked as general laborers, 16.7% in food service, 15.6% in clerical positions, and 125 in operative occupations. These data reflected competitive employment acquired by the youth, family members, or others; the authors did not consider supported employment (SE) placements. Results of the D’Amico and Blackorby (1992) survey served as an empirical basis for organizing job placements.

Johnson et al. (1997) examined types of placements of about 400 youth and found that 50% worked during high school. About one-half of these youth worked in food service, laundry, childcare, or janitorial jobs. Fewer youth worked in clerical and sales positions or benchwork jobs. Categories of occupations used by Johnson et al. (1997) differed from those used by D’Amico & Blackorby (1992). However, types of employment held by youth during high school in the Johnson et al (1997) survey resembled jobs held by youth up to two years post-high school in the D’Amico and Blackorby (1992) survey.

McNeil (1993) studied a database with over 3,000,000 rehabilitation job placements of individuals with severe disabilities (e.g., mental retardation, and cerebral palsy). Unlike the previous surveys of youth during/following high school, McNeil’s research involved adults with disabilities. McNeil found that the most common type of job placement was food preparation and service, followed by cleaning and building maintenance, machine operators, executive and administrative functions, and sales-related services.

Common themes regarding types of community employment placement begin to emerge from these and other survey studies. Employment positions in food service, cleaning, laundry, and general labor were common across a diverse population ranging from youth with disabilities recently out of high school to adults participating in vocational rehabilitation services.

Survey procedures. The Principal Investigator (PI), Project Coordinator (PC), and Project Assistant (PA) compiled lists of adult supported employment providers from 1,200 in the Council for Accreditation of Rehabilitation Facilities (CARF) Directory and 200 youth transition programs from listings currently held by the PI. Transition programs were identified in directories of eight states (Connecticut, Florida, Kansas, Maine, New Jersey, North Dakota, West Virginia, and Utah). Adult programs from 40 states were represented. Staff used a random numbers table to select 203 CARF programs from these state directories to receive the survey. Staff randomly selected 186 transition programs to receive the survey. The three-page survey questionnaire addressed demographic information, frequency and types of job placements made by the program, and emerging markets. Job categories listed on the survey included clerical and sales, service, agricultural, fishery, forestry, processing, machine trade, benchwork, structural work, and miscellaneous occupations. The survey provided several examples of jobs within each category. We asked that survey respondents review all job categories and list the total number of individuals placed in the last 12 months. In March 1998, the PA mailed the survey questionnaire. Jobs were grouped in a manner similar to those listed in the DOT (U. S. Department of Labor, 1991). Follow-up letters/surveys were sent in early April 1998, and phone calls were placed 2 weeks later. In April 1998, the Project Assistant compiled data on (a) demographic characteristics of youth and adult participants, (b) jobs identified most frequently for youth in transition; (c) potentially attainable (emerging) job areas with competitive wages, and (d) jobs most frequently associated with promotions, salary increases, and longevity. These data, shown in Table 2, guided staff, the project consultants, and task force in identifying jobs to be shown on the CD ROMs.

Survey results. The survey yielded results on 7,553 job placements of youth and adults with developmental disabilities. The most frequent job placement category was food and beverage preparation services, which accounted for almost 15% of all placements. For both youth and adult programs, placements in this job category were over 5% higher than the nearest category (i. e., building and related services occupations). Food and beverage services, building and related service, miscellaneous work, sales, production and stock clerks, and domestic service occupations accounted for over half of total job placements. In contrast, Agricultural, Fishery, Forestry, and Related Jobs, Benchwork, Structural Work, and Machine Trade Occupations collectively accounted for less than 10% of placements.

Project staff listed frequencies of specific job placements within each general category as one guide for identifying jobs for the program. Job placements from the survey accounted for 102 of 120 jobs (85%) identified in the program. For example, the program included jobs for which youth and adults had been placed, such as Baker, Copy Center Worker, Fast Food Cook, Food Preparation Worker, etc.

Table 1

Project Timeline: Accomplishments October 1997 - September 2001

Projected Actual

Responsible Completion Completion

Objective Person(s) Date Date

GOAL I - Objective 1 (Oct. - Dec. 1997):

Survey supported employment program

providers and school-to-work transition

programs to identify jobs held by youth

with disabilities and those jobs

potentially attainable. PI, PC, PA Dec. 1997 April 1998

GOAL I - Objective 2 (Oct. - Nov. 1997):

Review the rehabilitation literature to

identify critical attributes of targeted jobs. PA Nov. 1997 January 1998

GOAL I - Objective 3 (Jan. 1998):

Survey advisory board members, the

project consultant, and local employers

to validate critical attributes of jobs. PI, PC January 1998 April 1998

GOAL II - Objective 1 (Jan. - Aug.1998):

Develop a prototype program which can

be field tested. Proj. Staff August 1998 August 1998

GOAL II - Objective 2 (Aug. - Sept. 1998):

Conduct preliminary field testing in two

districts (Ogden City and Granite

Districts, Utah). Project Staff August 1998 Jan. 1999

GOAL III - Objective 2 (March - June

1999): Conduct the main field test. Project Staff June 1999 June 1999

GOAL IV - Objective 1 (Oct. 1999 -

April 2000): Conduct the operational

field test and make final revisions in the

program and facilitator’s manual. Project Staff April 2000 April 2000

GOAL IV - Objective 2 (April.-Sept. 2000):

Determine the psychometric characteristics

of the program. PI, PC Sept. 2000 Dec. 2000

GOAL V: Disseminate

the program to secondary, rehabilitation,

and supported employment programs

nationwide. Project Staff Ongoing Dec. 2000

Table 2

Highest Frequencies of Rural and Urban Transition and SE Program Placements by

Specific Job Category

Transition SE

Rural Urban Rural Urban Total

Job category n n n n N %

Food and beverage

preparation and services

(e.g., kitchen help) 306 369 106 332 1,113 14.74

Building and related service

occupations (e.g., janitors) 154 215 160 196 713 9.60

Miscellaneous work

occupations (e.g., grocery baggers,

auto detailers) 98 240 125 145 608 8.04

Miscellaneous sales occupations

(e.g., grocery clerks) 174 120 89 151 534 7.07

Sales: consumable commodities (e.g.,

department store clerks) 99 145 53 228 525 6.95

Production and stock clerks

and related occupations

(e.g., shipping, receiving) 99 164 26 216 505 6.69

Domestic service occupations

(e.g., housekeepers, launderers) 85 126 66 179 456 6.04

Miscellaneous personal service

occupations (e.g., child care workers,

nurse assistants) 115 169 23 86 393 5.20

For a given job, specific tasks were originally identified using job analyses in The DOT. To verify the accuracy of job tasks, staff made telephone calls to employers across the U.S performing each job included in the program. We used a random numbers table to select a telephone area code and prefix, then located the job type in a yellow pages list on the internet. Staff called the first employer listed in the yellow pages. When the accuracy of tasks were questioned by an employer, our staff called additional employers for clarification. In all telephone contacts with employers, staff determined that 98% of job tasks were accurate as described in  The DOT.

Decision rules for including/excluding jobs. Project staff, consultants, and the Utah task force formulated decision rules for jobs. They decided to include a job in the program if it was represented in the job placement survey data as one in which (a) youth/adults with disabilities had previously been placed, or (b) youth/adults may be placed in the future because the job was identified as representing an "emerging market." A job was excluded from consideration if:

1. The job was similar to others in the same domain to the extent that distinctions would be difficult for youth to make, i. e., one job represented most of the tasks in the other job.

2. The job required a bachelor’s degree at its entry level such that at least four years of post-secondary education would be necessary, at minimum. Though excluded, information in an appendix of the Facilitator’s Manual describes how other jobs may lead to the same career, such as a Teacher’s Assistant position (i.e., Paraeducator) leading to certified teacher.

3. The job carried liabilities involving frequent and exclusive responsibility for the lives of others (such as Emergency Medical Technician, Life Guard, Air Traffic Controller, etc.).

4. The job was only available on a local or regional basis or was not continuously available to youth during many times of the year (such as Snow Removal Worker, Swimming Pool Attendant, etc.).

Fastest growing occupations: extent to which expanding job markets were included in program job list. In addition to the decision rules described above, project staff ensured that most jobs included in the program were continuously available to youth and represented rapidly growing jobs. The Occupational Outlook Handbook (JIST Publishing, 1998) listed fastest growing occupations and those with the largest projected increases in employment between 1998 and 2008.

These occupations were arranged according to education and training required. Based on decision rule #2 for exclusion, project staff limited analysis to jobs requiring an Associates Degree or less education. The Occupational Outlook Handbook listed 35 fastest growing occupations requiring an Associates Degree or less. As shown in Table 3, the CD ROM program included 22 of these jobs (63%). The remaining 37% were excluded based on one or more decision rules above.

The instructional design specialist (IDS) used the GOAL I.3 information to develop shot lists for video recording at job sites.

GOAL II. Develop a Prototype Video CD ROM Based on Critical Attributes of Jobs

Identified in GOAL I Research and Conduct Preliminary Field Testing

GOAL II - Objective 1: Develop a Prototype, Video CD ROM Program which can be Field Tested

The prototype was developed in August 1998 and field tested at school district sites in Utah (Ogden, UT., Granite, Salt Lake City, UT.) to determine whether it functioned as anticipated and whether high school participants and facilitators could operate the prototype and derive information from the video. Problems were encountered in compatibility of the video drivers in some computers relative to the Macromedia program and QuickTime Movie files. These compatibility problems were subsequently resolved by the Instructional Design Specialist (IDS).

GOAL II - Objective 2: Conduct Preliminary Field Testing in Two Nearby Districts (Ogden City and Granite Districts, Utah)

Project staff conducted the preliminary field test beginning in September 1998 to ensure the program was functional and to determine if it is addressing the needs identified in GOAL I. A sample of 20 youth from the Ogden and Granite Districts participated.

Table 3

Fastest Growing Occupations and Those with Largest Projected Increases in Employment with Reference to Jobs Included in CD ROM Program

Occupations Requiring:

Associates Degree ( x = included in CD ROM program)

Computer support specialist

Dental hygienist

x Electrical and electronic technician

x Health information technician

Paralegal and legal assistant

x Physical therapy assistant

Registered nurse

Respiratory therapist

Postsecondary Vocational Training (x = included in CD ROM program)

x Automotive mechanic

x Data processing equipment repairer (Computer repairer)

x Hairstylist and cosmetologist

x Licensed practical nurse

x Manicurist

Surgical technologist

Work Experience in a Related Occupation (x = included in CD ROM program)

Detectives and criminal investigator

x Food service (Fast food cook, Food preparation worker)

Instructor: adult nonvocational education

Instructor, vocational education and training

x Lawn service manager

x Lodging management (Hotel desk clerk)

Marketing and sales worker supervisor

x Office and administrative support (Secretary, Receptionist)

Private detectives and investigator

Long-Term On-The-Job Training (more than 12 months) (x = included in CD ROM program)

x Carpenter

Correction officer

Desktop publishing specialist

x Maintenance repairer, general utility

Police patrol officer

x Restaurant cook

Sheriffs and deputy sheriff

x Telephone and cable TV line installer

Moderate-Term On-The-Job Training (1 to 12 months) (x = included in CD ROM program)

x Dental assistant

x Electronic semiconductor processor

x Instructors and coaches, sports and physical training (Recreation Assistant)

x Medical assistant

Models, demonstrators, and product promoters

x Packing and filling machine operators

x Social and human service assistant

Short-Term On-The-Job Training (up to 1 month) (x = included in CD ROM program)

x Adjustment clerk

Ambulance driver

Bill and account collector

x Cashier

x Home health aide

Office clerk

x Personal care aide

x Retail sales person

x Teacher assistant

Truck driver, excluding driver/sales worker

GOAL III. Revise the Program and Conduct the Main Field Test

Project staff also contacted a multi-state task force assembled by the New Jersey Center for Occupational Employment Information (NJCOEI). The NJCOEI task force included members in Alabama, Florida, Indiana, Iowa, New York, North Carolina, Washington, and Wisconsin. Some members of this task force reviewed career video sources and determined that there was a need for quality video footage that reflected current jobs in a non-stereotyped manner (L. Siedel, personal communication, August 10, 2000). This group was charged with responsibilities of selecting relevant occupations for videotaping, arranging videotaping, and assessing the accuracy of recorded video in relation to job tasks. The task force incorporated the videos into the One-Stop Career Center System used by schools to assist typical youth in career exploration. Videos may be viewed at America’s Career InfoNet (2000). The NJCOEI task force assisted project staff by providing copyright-free footage of career videos and by consulting on various project activities.

Project staff contacted consultants in December 1998 to present the revised program and gather recommendations. Staff compiled a list of modifications needed in video content, branching methods, facilitator’s manual, and other aspects of the program in preparation for GOAL III (Main Field Test). In October 1998, the project purchased public domain video material of 68 jobs from MIDI, Inc., Princeton, NJ. With permission from the NJOICC (see letter acknowledging purchase in Appendix B), this public domain video material was edited for inclusion in the CD ROM program. Project staff and the local video subcontractor, K-SAR Productions, recorded supplemental video in September 1998 and November 1998. From September 1998 - January 1999, the PI and PC revised the manual and arranged implementation of the main field test.

In preparation for the Operational Field Test, new video material was recorded in July-August 1999, editing was completed at K-SAR in October 1999, and digitizing/authoring was completed in November 1999. The PI and PC made arrangements for implementing the operational field tests at national sites (California, Colorado, Illinois, Montana, Maryland, Pennsylvania, and Utah) beginning January 2000. In addition to these sites, the Utah districts (Ogden and Granite) continued participation.

GOAL IV. Revise the program and conduct the operational field test

Table 4

Field Sites Across Preliminary, Main, and/or Operational Field Tests

High School Transition Program Sites (n=10) and Numbers of Participants (n=159)

Site Name of District Contact Staff Person

Ogden, UT. (n=81) Ben Lomond and Ogden High Schools, Gary Fritz,

Post High School Program, Transition Services

Ogden City School District Coordinator

Salt Lake City (n=7) The Jones Center for Transition from Jeff Rydalch, Director

UT. School to Work, Granite District

Kaysville, UT. (n=22) Davis School District High School Kitty DeJarnett, Tran-

Transition Program

Englewood, CO. (n=10) Sheridan High School, Colorado Lisa Sauers, Teacher

School District

Lewistown, MT. (n=10) Fergus School District Dale Lambert, Teacher

Baltimore, MD. (n=10) Towson University Outreach Prog. Sandy Fisher

Holton, MI. (n=10) Holton School District Jane Behm

Chicago, IL. (n=1) Aero Cooperative Cindy Mayer

San Pablo, CA.. (n=1) West Contra Costa U.S.D. Laraine Domenico

Logan, UT. (n=7) Cache Employment and Training Center Kaelyn Beecher

Adult Supported Employment Program Sites (n=2) and Number of Participants (n=23)

Logan, UT. (n=8) Northeastern Services Rebecca Blair

Pittsburgh, PA. (n=15) Arc Allegheny Tom Geyer

TOTAL SITES = 12

Table 5

Characteristics of Youth/Adult Participants in Preliminary, Main, and Operational Field Tests

N %

Male 109 59.89

Female 73 40.11

TOTAL 182

N %

Ethnicity

Caucasian 126 69.23

Hispanic American 36 19.78

African American 17 9.34

Pacific American 1 0.55

Asian American 1 0.55

Not Reported 0 0.0

Disability

Autism 4 2.20

Behavior Disorders 8 4.40

Communication Disorders 5 2.75

Deafness/Hearing Impairment 2 1.10

Traumatic Brain Injury 7 3.85

Intellectual Disability 108 59.34

Multiple Disabilities 9 3.30

Orthopedic Impairment 0 0.0

Table 5, continued

Characteristics of Youth/Adult Participants in Preliminary, Main, and Operational Field Tests

Other Health Impairment 1 0.55

Specific Learning Disability 29 15.93

Blindness/Visual Impairment 1 0.55

Not Listed 8 4.40

Mean %

Overall mean age 20.14

Mean age: youth programs 17.33 (N = 159)

Mean age: adult programs 38.70 (N = 23)

Measured Intelligence Level (Mean) 59.60

Estimated/Measured Reading Level 2.41

Participants with Previous Work Experience 83 45.60

Demographic data regarding youth and adult participants are shown in Table 5. The majority of participants were male, Caucasian, and diagnosed with Intellectual Disabilities. However, participants with a diversity of characteristics assisted in evaluating the program and selecting job preferences.

A total of 29 facilitators including. high school teachers, transition specialists, paraprofessionals, and supported employment program staff evaluated the program.

Results from Preliminary, Main, and Operational Field Tests. Generally, facilitators provided positive feedback regarding organization, clarity, relevance, ethnic and gender diversity, and usefulness of the program. They identified some problems with clarity of job tasks (e.g., a sequence of job tasks shown for a punch press operator). They questioned the inclusion of some jobs on video because of relatively sophisticated skills, such as Corrections Officer, Dental Hygienist, and Paralegal. Their input matched information received from the task force, and compelled project staff to eliminate these jobs. Other jobs were maintained despite questions from facilitators about skill levels. In these cases, project staff emphasized the importance of modifying jobs, if necessary, to match the skills of youth. Staff included examples in the manual and a video clip on the CD-ROM showing how to modify jobs. For example, an auto mechanic job may require skills beyond the abilities of some youth. However, we described how the job may be modified (in this case, scaled down) to create jobs related to auto lubrication, tire replacement, muffler replacement, auto detailing, etc.

Although facilitators were satisfied with ethnic and gender diversity, they found limited video depicting employees with disabilities and also recommended younger employees in some video scenes to better match the age levels of participants. Project staff used this feedback to guide subsequent videotaping of jobs. Of 120 total jobs, the final product portrays employees with disabilities in 17 jobs (14.1%), females in 81 jobs (67.5%), only females in 12 jobs (10%), employees of color in 70 jobs (58%), and youth 25 years or younger in 110 jobs (92%).

Facilitators indicated the program was useful for IEP planning, career education activities, referral to other services or assessments, disclosure of new information about preferences, confirmation of previous assumptions about preferences, and communication with parents/family, and service providers. Most agreed that the program was not useful as a means of arranging job placement.

Some facilitators provided case examples of program use. For example, a male youth with intellectual disability had shown inconsistent attendance in previous food service placements and sometimes dressed inappropriately for work. He selected the hotel housekeeper job, was placed on a housekeeping job by a job coach, and has performed well in that capacity for 2 years. He was promoted and now supervises a crew of housekeepers with typical characteristics. One young adult with cerebral palsy had been difficult to place on a job due to motor impairment. She selected clerical and hair styling jobs shown in the program, but her service providers doubted her capacity to perform the tasks. Nevertheless, they acknowledged her social skills, money management, and attention to detail. The support team designed a web page from which she could sell cosmetics. She mails cosmetics to her customers.

Program description. Recorded on seven CD-ROMs, the program contains two to four min of video information on 120 jobs. The entire program contains about 6 hours of motion video. As video plays, a narrator describes the critical features of each job. Individual youth work at a computer terminal with the assistance of a facilitator, who may be a teacher, rehabilitation counselor, transition specialist, advocate, or parent. Not all 120 jobs are shown to a youth; an initial set of decisions (e.g., selecting work conditions such as inside versus outside work, working with public versus working in private) allow the youth to target a "domain" of 20 jobs matching interest areas. Youth make selections after watching video on pairs of jobs by using a computer mouse to "click" on the preferred job. The outcome is a list of 5-10 preferred jobs. These job selections may then be used to guide transition planning and job placement efforts.

Materials. The program consists of the following:

1. Seven CD-ROMs. The first CD-ROM introduces the program and presents the "Preview" and "Work Conditions" phases described below. Each of the remaining CD-ROMs shows video of one domain of 20 jobs with common characteristics.

2. A facilitator’s manual. The manual describes how to use the program and how to assist youth in making choices. It includes an appendix with additional information on each job, such as typical wages, working conditions, job outlook, training and educational requirements, qualifica-tions, advancement opportunities, and tasks that comprise the job (see excerpt, Appendix C).

Facilitator responsibilities. Administration responsibilities of the facilitator include establishing rapport, familiarizing the youth with the computer, prompting the youth to respond to initial screening items and job domain selections, answering questions, obtaining a printout of the youth’s selections. These activities are described in the Facilitator’s Manual. A summary form allows the facilitator to compile data on selections (see Appendix D).

Program Procedures

A youth and a facilitator conduct the program in three phases: (a) Preview, (b) Work Conditions, and (c) Job Choices. These phases are described below.

Preview. The optional Preview Phase determines whether youth with cognitive or functional limitations can discriminate objects, actions, and work vs. leisure actions. This phase establishes whether youth have the prerequisite skills to participate in subsequent phases of the program. Initially, the Preview Phase assesses whether youth can identify objects commonly found in work environments (e. g., telephone, pencil). Youth are first presented with two still pictures of objects on the screen as a narrator asks, for example, "Which one is a telephone? Click on the telephone." The computer records the response and presents the second pair of objects. Figure 1 shows one screen presented to youth in the Preview Phase. Five pairs of objects are shown. Next, youth watch 5 seconds of motion video showing an action (e.g., walking) followed by 5 seconds of another action (e.g., climbing a ladder). After each pair of videos showing actions, a still frame is shown as the narrator asks, for example, "Which one shows walking? Click on walking." Five pairs of actions are shown. Third, youth are shown pairs of videos related to either employment or leisure. For example, a video shows a female vacuuming a vehicle in an auto detailing facility, followed by a segment of the same female listening to an audio cassette playing music in the vehicle’s cassette deck. Five pairs of work/leisure actions are shown. At the end of the Preview Phase, the computer presents the youth’s percentage of correct responses. The Facilitator’s Manual suggests criteria for deciding whether to continue with the program based on the accuracy of the participant’s responses.

Work conditions. As shown in Figure 2, in the Work Conditions Phase, youth determine general preferences for employment environment. For example, the program may show video from a variety of jobs primarily performed outdoors, followed by indoor jobs. At the conclusion of this pair of 18-second videos, the narrator asks, "Which do you like: Working outdoors or working indoors?" Additional discriminations include heavy lifting/high exertion vs. light lifting/light exertion, working with people vs. working alone, and working with the public vs. working with co-workers. After making these decisions, the program identifies a domain of 20 jobs matching the preferred work conditions. The Work Conditions decision chart is shown in Figure 3. The youth and facilitator are encouraged to explore multiple domains, or to return later to the Work Conditions Phase for additional job exploration.

Job choices. In this phase, the youth and facilitator watch 20 jobs in 10 pairs that match the interests in work conditions. For example, if responses indicate interest in inside work, light lifting, and interaction with the public, the program directs the youth and facilitator to a CD-ROM that includes Receptionist, Manicurist, Retail Salesperson, Dental Assistant, Hotel Desk Clerk, etc. Each job is presented in 2-4 min of video showing the critical tasks. The narrator describes the tasks shown on the screen. Customers, employees, supervisors, noise levels, work pace, and other attributes of each job are presented in a way that matches information collected by project staff from The DOT (U.S. Department of Labor, 1991), Occupational Outlook Handbook (JIST Publishing, 1998), and interviews with employees. After the youth and facilitator watch two jobs, the original frame of each video condition is presented on a split screen, as shown in Figure 4. For example, the Receptionist is shown in a still frame on the left half of the screen, juxtaposed with a still of the Manicurist on the right. A narrator asks, "Which job do you like? Click on the job you like." If the youth requests more information, the facilitator refers to the manual, which includes two pages on each job (wages, benefits, qualifications, training requirements, etc). If the youth requests to watch the video again, a "rewind" button replays the video. If a youth wants to skip a video, a "fast forward" button presents the next one. The youth then selects one of the two jobs, or clicks a button indicating preference for neither job. After the computer records the selection, two different jobs appear. When the youth and facilitator have viewed 10 pairs of jobs and made selections, the same 20 jobs are presented again in new pairings. Two sets of job pairings per domain allow the

Figure 2. One Work Conditions Screen in which Youth Determine General Preferences for Employment Environment.

youth to consider a job against two alternatives. Project staff established pairings using a random assignment process. Job pairings for all six CD ROMs are shown in Appendix E.

Project staff selected the paired choice format after reviewing research literature on choice responses of people with disabilities (Fisher et al., 1992; Pace, Ivancic, Edwards, Iwata, & Page, 1985) and examining three other formats. A single presentation format involved youth making "yes/no" decisions about each job. This format was rejected because of high probabilities of false positives (Fisher et al. 1992). A concurrent presentation format with "replacement" required viewing two jobs, making a selection, and retaining that selection while subsequent jobs were shown in sequence. This format was rejected because it was time consuming and repetitive. The paired choice format was found to be most efficient and user friendly.

Video recording and editing. Project staff and K-SAR Video Productions recorded video of 52 jobs. The remaining 68 jobs were purchased in raw footage from MIDI, Inc. Staff edited video using a Media 100 desktop video editing system purchased by project funds. The use of a nonlinear Media 100 4.0 system allowed the IDS to monitor job editing, verify the edited video or request changes, and view the changes "on the fly." Staff edited jobs to illustrate highest priority tasks and to portray typical environments, noise levels, lumination, pace, numbers of co-workers and members of the public, and other critical variables. Staff used The DOT job analyses (Farr & Ludden, 1998) and job verification contacts with employees to allocate video to particular job tasks, i.e., high priority tasks were shown for longer periods of time than low priority tasks. When a specific job involved a clear outcome or objective, such as auto detailing, the edited video showed a sequence of tasks from beginning to end (e.g., washing the vehicle, applying wax, removing wax, vacuuming interior, etc.). When jobs involved multiple tasks or actions without clear sequences, such as butcher or retail sales, the edited video showed workers performing several actions in small segments.

Programming. Project staff and subcontractors recorded the source footage using Beta SP videotape. Staff edited video using the nonlinear system described above. The IDS performed primary compression with the Media 100 Codec set at 15 frames per second. For QuickTime video clips (Apple Computer, Inc., 2000), the IDS produced secondary compression using Media Cleaner 4.0 (Terran Interactive, 1999) at 30 frames per second, 200 kbps transfer rate, 360 x 240 pixels, for video only. The IDS programmed the forced-choice presentation of two jobs using Director 7.0 (Macromedia, Inc., 2000). Color depth was 800 x 600 at 16-bit color.

Project staff also computed the degree to which actual jobs selected in re-administration were identical to jobs selected in the first administration. If a participant selected work conditions leading to a different disk than the disk selected in the original administration, a score of 0 was recorded. If the same disk was selected and a participant selected the same job, a score of 1 was recorded. If 5 identical jobs were selected but two were different, 5 "hits" and 2 "misses" were

Table 6

Reliability Data by Field Site.

Work Condition Decisions Job Selections

Site Hits Total Hits Total

Ogden 41 43 64 103

Lewistown 31 34 40 49

Baltimore 32 32 49 60

TOTAL 104 109 (.969) 153 212 (.722)

recorded. Using this index for specific job choices, 153 "hits" were recorded and 69 "misses," for a reliability index of 0.722. That is, about 72% of job choices selected in retest were identical to jobs selected in the initial test.

Validity measures. Preliminary validity research is reported here; more extensive psychometric analysis is ongoing. Two validity questions were investigated: (a) To what degree do program choices correlate with choices from the Reading-Free Vocational Interest Inventory - 2nd Edition (RFVII:2) (Becker, 2000)? (b) To what degree do program choices correlate with community employment selections?

CD ROM Program and RFVII:2. The RFVII:2 is an assessment of vocational interest for individuals who have no reading skills. This instrument has undergone extensive psychometric assessment in the new, revised version as well as the original one (Becker, 2000). Participants view sets of three line drawings. Drawings depict different vocational activities, such as using a hammer to frame a house, waiting tables at a restaurant, and providing personal care to persons with disabilities. A participant draws a circle around the picture he/she likes best. The RFVII: 2 consists of 55 sets of drawings. Results are expressed as raw scores, T scores, and percentiles in 11 Interest Areas, including Automotive, Building Trades, Clerical, Animal Care, Food Service, Patient Care, Horticulture, Housekeeping, Personal Service, Laundry Service, and Materials Handling.

Two undergraduate students independent of the project reviewed the RFVII:2 and CD ROM program. The purpose was to develop an index for measuring the relationship between selections from the two programs. To do this, the undergraduate students identified all line drawings from the RFVII:2 that corresponded with a CD ROM job. For example, students identified four RFVII:2 line drawings (drawing nos. 6-g, 18-k, 34-b, and 13-e) that matched the CD ROM job of "Gardener." One student found 129 drawings that corresponded with CD ROM jobs. The second student found 126 of the same drawings and three additional ones. Inter-observer agreement was calculated by dividing 126/129 x 100 = 97.7%. Both students found that 45 of 120 CD ROM jobs corresponded with RFVII: 2 line drawings (37.5%).

Using the student's index of 126 items of correspondence, we arranged for 10 youth with specific learning disabilities and 10 youth with intellectual disability, specific learning disabilities, or multiple disabilities to participate in both the CD ROM program and RFVII:2. Two facilitators, unaware of the purpose of the research, administered the two programs. We compared jobs selected from the CD ROM program with potentially matching line drawings in the RFVII:2. For example, if a participant selected the Carpenter job from the CD ROM program, we checked V line drawings 6-g, 18-k, 34-b, and 13-e to determine if they were also selected (i.e., "potential matches"). Of 229 potential matches in RFVII: 2 line drawings, participants selected 92. Given that RFVII: 2 procedures require that a participant select from groups of three line drawings, 76 of 229 potential matches, or one-third, would have been expected by chance. Chi-square analysis (x2: Ferguson, 1981) revealed that this difference between obtained and expected matches was not significant (x2 = 3.08, df = 19, effect size = .162). A Contingency Coefficient (C: Ferguson, 1981) was computed to determine the correlation between these data. The Contingency Coefficient is a nonparametric measure of correlation based on the c2 statistic. Values vary from 0 to 1. In this analysis, C = .365.

Low correlation between CD ROM selections and RFVII:2 should be interpreted with caution for at least three reasons. First, the RFVII:2 was designed to yield standard scores in Interest Areas based on patterns of line drawing selections. Comparing CD ROM selections with single RFVII:2 line drawings, while necessary as an index for comparison, is inconsistent with the purpose of RFVII:2. Second, only 45 CD ROM jobs corresponded with RFVII:2 line drawings. Undergraduate students found no relationship between the remaining 75 CD ROM jobs (62.5%) and RFVII:2 line drawings, so no matches existed for the majority of CD ROM jobs. Third, data from only 20 participants were analyzed.

For 20 participants, project staff compared the two most highly preferred CD ROM jobs with the two highest Interest Areas from RFVII:2. Lists of CD ROM jobs were categorized according to RFVII:2 Interest Area (e.g., RFVII:2 Clerical Interest Area included CD ROM jobs of Data Entry Worker, Health Information Technician, Library Worker, Receptionist, Secretary, etc.). After administering both programs, matches and mismatches were compiled. Of 40 possible matches (20 participants x 2 highly preferred CD ROM jobs), 13 matches with RFVII:2 Interest Areas were found. The most frequently matched Interest Area was Food Service. However, because the probability of a match was only one in 11, or 9%, chi-square analysis revealed that this difference between obtained and expected matches was statistically significant (x2 = 24.54, df = 10, p < .01, effect size = .613, C = .617).

CD ROM program selections and community employment selections. Nine males and 11 females were selected from 75 participants in a high school transition program conducted by the Davis County School District in Kaysville, Utah. Participants were selected based on seven criteria: (a) proportions of gender found in the general population, (b) age range from 16 to 22, (c) severe intellectual disability based on recent psychometric data, (d) previous independent work experience of less than one year, (e) enrollment in a school transition program, (f) current unemployment, and (g) expressed interest in participation. In addition, participants demonstrated the ability to express a preference through verbalization, vocalization, sign, or gesture. Each youth first participated in the CD ROM program administered by a co-author. Based on work conditions selections, each youth viewed jobs on one of the CD ROM Job Choices disks. Youth responded to both sets of job pairings on the selected disk. For each youth, the co-author identified 4 or 5 jobs selected in both the first and second pairings. We assumed these jobs were "highly preferred." Two of these highly preferred jobs were randomly selected and scheduled for a brief community employment tour. For each youth, the co-author identified 4 or 5 jobs that were not selected in either the first or second pairing. We assumed these jobs were "not preferred." Two of these jobs were randomly selected and scheduled for a brief community employment visits. Four community employment visits were scheduled within two weeks of CD ROM program administration. The order of preferred and non-preferred job visits was randomized, except that one preferred and one non-preferred job was visited in each of two pairs. For example, Pair 1 may have involved visits to a preferred job followed by a non-preferred job. Pair 2 may have involved visits to a nonpreferred job followed by a preferred job. Initial contacts with employers were made by the co-author to schedule the tour. Community employment visits were conducted by a Davis School District employee (the "facilitator") who remained unfamiliar with purposes of the study. The facilitator conducted all community employment visits with individual participants; no visits were conducted with multiple participants. The facilitator told each participant, "We're going to visit some work sites. After we're done, I'll ask which ones you liked." Tours lasted a maximum of 15 minutes. Facilitators carried forms listing each job's critical tasks as described in the Occupational Outlook Handbook. During the tour, the facilitator and tour guide from the employment site reviewed the list of critical tasks and briefly showed as many tasks as possible to the participant. Instructions to the facilitator called for viewing individual job tasks for no more than one minute. Participant questions were answered either by the facilitator or tour guide. The facilitator was instructed to avoid allowing the participant to perform the tasks or receive gifts, food, or refreshments from the tour guide or other employees. These activities, the researchers reasoned, might have biased the participant's selections of preferred jobs. At the conclusion of the 15-minute tour, the facilitator took a snapshot of the participant positioned in the foreground of the photo with the work environment in the background. Instructions called for the snapshot to "contain enough information so that the participant can identify the work environment later." The facilitator conducted four community employment visits with a participant in the same day. After four visits, the facilitator and participant returned to the transition program. Snapshots from the first pair of jobs were presented to the participant as the facilitator asked, "Which one did you like? Point to the job you liked better." This process was repeated with the second pair of jobs. Participants made selections from snapshots immediately upon return to the transition the same day as the tours. The participant's selections of two preferred jobs were recorded on a Job Record Sheet and returned to the co-author for comparison to CD ROM program selections.

We compared data from CD ROM program selections with community employment selections for 20 participants. Data indicated that, after community employment visits, participants selected as highly preferred those jobs previously identified as highly preferred from the CD ROM program in 33 of 40 cases. In only seven cases were highly preferred jobs from the CD ROM program not identified as preferred after community selection. In only 7 cases were non-preferred jobs from the CD ROM program identified as preferred in community selection. Chi-square analysis (Ferguson, 1981) revealed that this difference between obtained and expected community selections was significant (x2 = 8.45, df = 19, p < .01, effect size = .422). A Contingency Coefficient (C: Ferguson, 1981) was computed to determine the correlation between these data. The Contingency Coefficient is a nonparametric measure of correlation based on the x2 statistic. Values vary from 0 to 1. In this analysis, C = .545.

The CD ROM job program includes an option allowing the participant and facilitator to repeat identical pairings for three pairs of jobs.The only difference in the second presentation is that the position of jobs is reversed on the computer screen. Selections in these "repeat pairings" provide the participant and facilitator a measure of preference consistency. Project staff assessed consistency of selections among 29 participants at the Ogden site. Data indicated that 54 initially

selected jobs were re-selected when paired with the same job and seven were not selected (54/61 = .885).

GOAL V. Disseminate the Program to Secondary Special Education, Rehabilitation, and Supported Employment Programs Nationwide

GOAL V: Disseminate the Program to Secondary, Rehabilitation, and Supported Employment Programs Nationwide

Dissemination activities have consisted of (a) describing job placement survey research and the CD ROM program in the transition literature, (b) presenting information regarding the program to state and national conferences, (c) disseminating a brochure and newsletter to transition and supported employment programs nationwide, (d) expanding and updating a mailing list of interested professionals who may use and/or market the program, (e) developing and maintaining a web site, and (f) preparing to distribute the program at the conclusion of the project.

Publications and presentations. As shown in Appendix F, project staff have submitted four refereed manuscripts (Ellerd & Morgan, under review; Morgan, et al., 2000; Morgan, Ellerd, Gerity, & Blair, 2000; Morgan, Gerity, & Ellerd, 2000) and three nonrefereed manuscripts (Ellerd & Morgan, 1998; Ellerd & Morgan, 1999; Morgan, Ellerd, & Gerity, 2000) for publication. Seven state and national conference presentations have been conducted, as listed in Appendix F. Additional manuscripts on reliability and validity analyses, and national presentations describing the product are also planned.

Refereed manuscripts include (a) an article describing results of the national survey on job placement (Morgan, Ellerd, Jenson, & Taylor, 2000), (b) a report detailing the functions and operations of the program to an audience primarily consisting of teachers, transition specialists, and general educators (Morgan, Ellerd, Gerity, & Blair, 2000), and (c) an article providing technical information to an audience of assistive technologists and special educations interested in technology (Morgan, Gerity, & Ellerd, 2000). All three manuscripts were published in journals of the Council for Exceptional Children, including Career Development for Exceptional Individuals (Morgan, Ellerd, Jenson, & Taylor, 2000), Teaching Exceptional Children (Morgan, Ellerd, Gerity, & Blair, 2000), and The Journal of Special Education Technology (Morgan, Gerity, & Ellerd, 2000). The manuscript published in Teaching Exceptional Children was a feature article and consisted of cover and other photographs taken by the third author and the project IDS, Brent P. Gerity. A fourth manuscript was submitted to Education and Training in Mental Retardation and Developmental Disabilities and remains under review.

Brochure and newsletter mailings. In February 1999, 2000, and 2001, annual newsletters were mailed to about 40 professionals interested in the project and task force members. In February 2001, a brochure was developed. In March 2001, the brochure was mailed to 11,000 school districts, transition programs, and supported employment programs throughout the U.S. This mailing list was developed over the course of the three-year project. Additional brochures will be mailed in 2001 to University Affiliated Programs, Higher Education Programs, The Transition Institute, and other transition-related programs.

Development of web site. In January 2000, a web site was developed for the project (www.trisped.org/yes). In February 2001, the site was updated to describe the final, field-tested product.

IV. Budget Report

No grant funds are expected to be unobligated at the end of the current budget period (September 30, 2001). If unobligated funds are identified at the end of the current budget period, the principal investigator will promptly contact the U.S. Department of Education.

Changes in Year 3 Budget (see attached line budget). No major changes occurred in Year 3 expenditures or in the no-cost extension period. All no-cost extension funds were expended during the no-cost extension period which commenced October 1, 2000 and ended September 7, 2001.

V. Supplemental Information/Changes:

Adjustments and changes encountered during the course of the project include the following:

• Personnel: Dr. Robert Morgan, the project’s Principal Investigator, reduced his time in Year 3 from .25 FTE to .20 FTE to provide necessary monies for travel to disseminate information about the program.

• Adjustments in the project timeline:Preliminary field testing in Ogden and Granite districts of Utah exceeded the anticipated August 1998 deadline by 5 months (January 1999). However, project staff were able to start main field test activities (GOAL III, Obj. 2) in March 1999 in anticipation of reaching timely conclusion in June 1999 as planned.

• Internet web site: The Instructional Design Specialist and Assistant revised the project web site in October 1998 (http://sped.usu.edu/tri-sped.html/future.html) to include information about the proposed CD ROM program.

• Survey of computer capabilities, CD ROM capabilities, and internet access of schools: Beginning in March 1998, project staff began a national survey of 400 school district media specialists to determine their technology capabilities and internet access. This survey will firmly establish the extent to which schools can use the CD ROM product and access it via the internet (see below).

References

Affleck, J. Q., Edgar, E., Levine, P. & Kortering, L. (1990). Postschool status of students classified as mildly mentally retarded, learning disabled, or nonhandicapped: Does it get better with time? Education and Training in Mental Retardation, 25, 315-324.

Becker, R. L. (2000). Reading-free vocational interest inventory. Renton, WA: Edmark.

Farr, J. M., & Ludden, L. (1998). The O*NET dictionary of occupational titles. Indianapolis, IN: JIST Works.

Ferguson, G. A. (1981). Statistical analysis in psychology and education. New York: McGraw-Hill.

Fisher, W. W., Piazza, C. C., Bowman, L. G., Hagopian, L. H., Owens, J. C., & Slavin, I. (1992). A comparison of two approaches for identifying reinforcers for persons with severe and profound disabilities. Journal of Applied Behavior Analysis, 25, 491-498.

Hoish, S. A., Karen, R. L., & Franzini, L. R. (1992). Two-year follow-up of the competitive employment status of graduates with developmental disabilities. Career Development for Exceptional Individuals, 15, 149-156.

JIST Resource Directory. (1998). Resource Directory. Indianapolis, IN: JIST Works, Inc.

Johnson, D. R., McGrew, K. S., Bloomberg, L., Bruininks, R. H., & Lin, H. C. (1997). Results of a national follow-up study of young adults with severe disabilities. Journal of Vocational Rehabilitation, 8, 119-133.

McNeil, J. (1993). Americans with disabilities: 1991-1992. Data from the survey of income and program participation. Washington, D.C: U.S. Department of Commerce, Economics and Statistics Administration.

Morgan, R. L., Ellerd, D. A., Gerity, B. P., & Blair, R. J. (2000). That’s the job I want! How technology helps young people in transition. Teaching Exceptional Children, 32 (4), 44-49.

Morgan, R. L., Ellerd, D. A., Jensen, K., & Taylor, M. (2000). A survey of community employment placements: Where are youth and adults with disabilities working? Career Development of Exceptional Individuals, 23 (1), 73-86.

Morgan, R. L., Gerity, B. P., & Ellerd, D. A.. (2000). Using video and CD ROM technology in a job preference inventory for youth with severe disabilities. Journal of Special Education Technology, 15 (3), 25-33.

Pace, G. M., Ivancic, M. T., Edwards, G. L., Iwata, B. A., & Page, T. J. (1985). Assessment of stimulus preference and reinforcer value with profoundly retarded individuals. Journal of Applied Behavior Analysis, 18, 249-255.

Sitlington, P. L., Frank, A. R., & Carson, R. (1991). Iowa’s statewide follow-up study: Adult adjustment of individuals with severe/profound mental disabilities 1 and 3 years after leaving school. Des Moines, IO: Iowa Department of Education.

U.S. Bureau of Labor Statistics. (1998). Occupational Outlook Handbook, 1998-1999. Washington, D.C: Department of Labor.