A. Executive Brief

Two major conclusions resulted from this project:

(1) ECS, including the Workplace Analysis, is a valid and an appropriate assessment and curriculum management model to use for determining the employability skills and postsecondary outcomes of secondary students who have specific learning disabilities, and

(2) Effective classroom and workplace accommodations are best identified on an individual basis and need to be expressed during IEP/ITP meetings for all service representatives on the team.

Following is a summary of outcomes from this three-year project that support the use of ECS for this population:

1) The CASAS Workplace Analysis was validated for identifying community employment and postsecondary outcomes in the transition from school to postschool environments. The validation procedure consisted of three major components:

a) Using the Job Profile from the Analysis to determine "Skills Employers Want" in employment training programs and jobs in the community. A sample of these skills can be found in Appendix A.

b) Using the Individual Profile from the Analysis to determine employment related basic skill competencies needed by the student in order to succeed in these employment training programs and specific jobs/job clusters. The validation of the Workplace Analysis in Appendix B is shown as a sample in three parts: (1) original paper/pencil version, (2) revised paper/pencil version, and (3) computerized version.

c) Using the Analysis as a communication tool for facilitating communication and focusing the transition team on planning postsecondary goals with the student.

2) The Employability Competency System (ECS) was validated for secondary students who have specific learning disabilities in the planning of postsecondary goals. The validation procedure consisted of three major components:

a) Using ECS to obtain assessment information (Appraisal and Pre/Post-tests) in reading, math and communication skills related to employability. Sample test items are in Appendix K. The pre- and post-test results were used for improving special education curriculum and instruction.

b) Using the results from ECS on competency attainment for individual transition planning and for determining appropriate postsecondary outcomes. The Individual and Class Profiles (Appendix K) on individual competency attainment became part of the student's IEP/ITP Planning Documents.

Demographic background (Table 1 in Appendix E) on the students included gender, age, grade level completed, native language and ethnic background. A majority (62.4%) of the students were male, most (84.2%) were 16 to 18 years of age, and most (85.1%) completed the tenth or eleventh grade the previous year. Most of the students (81.6%) reported English as their native language and an additional 12.9 percent reported Spanish. Over half (54.9%) of the students in the two targeted urban areas indicated their ethnic background was Hispanic (44.6%) or Black (10.3%) while only a little over thirty percent (32.2%) indicated Caucasian.

A total of 379 students were assessed in reading and 377 students were assessed in math using the ECS Appraisal test (Tables 2 and 3 in Appendix E). As can be seen in the Site Reports in Appendix C, some school sites only used the Appraisal for assessment. However, the purpose of the Appraisal Test in reading and math is to determine placement into the appropriate pre-test level for beginning instruction.

As can be seen in Tables 2 and 3 students as a whole tend to score a little lower on the pre-test (38.6 percent in the 211 220 scaled score range in reading) compared to the appraisal in which 29.0 percent scored in the 211 -220 range with the highest percentage (39.6 %) in the 221 - 229 scaled score range in reading. The pre-test has more items at one instructional level ranging from A (lowest) through D (highest), whereas the appraisal has an even distribution of test items across all four levels (A to D) .

Only 21% (n=83) of the students who were assessed with the Appraisal (n=379) in reading participated in the pretest for reading, and only 60% (n=50) of these students participated in the post-test. The pre and post-test rates are similar for math. Evaluation focus groups representing some of the teachers revealed that there was a high drop-out and absentee rate among their special education students at the post-test time. Also, some districts chose to participate in the Appraisal test only, even though the purpose of the test is to determine the appropriate level to begin instruction. The districts indicated a desire to use the Appraisal information to justify the need for employment related and transition courses.

The mean reading pre-test score was 216.0 and the mean reading post-test score was 218.2 for the 50 students taking both tests (Table 4), resulting in a mean learning gain of 2.2 points. As shown in the CASAS Level Descriptors for ECS at the beginning of Appendix C, the majority of students post-tested at the higher end of the B level.

Students at this level function at a basic literacy level and may have difficulty pursuing employment above an entry level position. However, they most likely will be able to perform job related tasks such as completing a simple order form and job application, and handle jobs/training that involve following basic oral or written instructions and diagrams if they are clarified orally.

The learning gain of 2.2 points (n=50) in reading on the CASAS scale for the eleventh and twelfth grade students, who had specific learning disabilities and represented urban areas in this study, is in contrast with another CASAS Transition study (Hl58g4OOO4) in which the learning gain was 3.8 points (n=271) in reading for eighth through tenth grade students, who had learning disabilities and represented rural, suburban and urban districts.

These gains in reading can also be compared to other CASAS reading samples such as Adult Basic Education students who gained a mean of 5.5 points (n=943), English as a Second Language students who gained a mean of 5.5 points (n=7,623) and Adult Special Education students (most of whom have developmental disabilities) who gained a mean of 2.5 points (n=1,339) (Student Progress and Goal Attainment for Federally Funded ABE Programs in California 1996-97). CASAS has been collecting mean scores for these populations since 1980.

Progress in math is shown in Table 5 in Appendix E. The mean math pre-test score for the students in this study who had specific learning disabilities was 207.9 and the mean math post-test scaled score was 210.3 resulting in a mean learning gain of 2.4 (n= 53). As can be seen in the individual school Site Reports in Appendix C, the gain in math scores was not always higher than the gain in reading.

Hours of instruction were reported for an extremely small number of cases (9 to 11). For these cases (Tables 7 and 8 in Appendix E), learning gains were greater in reading but lower in math when there were 90 to 139 hours (usually a semester) of instruction between pre- and post-testing, than for 40 to 89 hours of instruction (usually a 9-week unit). The quality and focus on transition during instruction for these cases were not studied.

The mean appraisal score, mean pre- and post-test scores and mean learning gain, in reading and math, for the school sites and states participating in this project are reported in Tables 9, 10, 11 and 12 in Appendix E. The mean learning gains in reading and math for the states can also be found in the State Reports in Appendix D. The State Reports include younger students participating in another related CASAS project (H158940004). The school sites within each state participating in both projects are listed in these state reports.

Special test accommodations used are reported in Table 6. The most frequent accommodations used were extra time (25.4%), frequent breaks (21.8%), and use of the calculator (40.8%). The mean pre-test score, mean post-test score and mean learning gain for both reading and math, when the accommodation was used and when it was not, are reported in Tables 13, 13a, 14 and 14a. Tests of significance were not done for these groups because of their small size, but were done for the combined group in this project with another related CASAS federal transition project (H158Q40004).

The pre-test score means, post-test score means and mean learning gains in reading and math for both CASAS Federal Transition Projects (Projects TAC/OTR) are reported in Tables 13, 13a, 14 and 14a in Appendix F. The ECS tests are not timed, but most students usually complete them in an hour.

When the accommodation of extra time was used, it was stipulated that no more than time-and-a-half be used to take the tests. CASAS tests were developed to be independent of time in order to complete the items. Also CASAS math tests were initially validated without the use of a calculator. It was stipulated that accommodations be used only when the need was indicated on the IEP and when they were also used for instruction.

Table 14 and 14a in Appendix F report mean scores from the two CASAS federal transition projects (Projects TAC/OTR) for math. When the accommodation of a calculator was used, the mean math pre-test score was 210.0, the mean math post-test score was 211.8, resulting in a mean math learning gain of 1.8 points (n=218) As shown in Table 14a, when the accommodation of a calculator was not used, the mean math pre-test score was 206.5, the mean math post-test score was 209.4, resulting in a mean math learning gain of 2.9 points (n= 139).

As shown in Tables 13 through 14a in Appendix F, the observed differences in learning gains between students using and not using the accommodations of extra time for reading and a calculator for math, were not found to be statistically significant (p<.05). As shown in Tables 15, 15a, 16 and 16a in Appendix F, the use of an accommodation results in a consistently higher learning gain only for the lower (Level A) scaled score ranges (181 - 190 and 191 - 200). This is consistent with other CASAS learning gain results, especially for students in English as a Second Language classes.

Results of interviews with a sample of site personnel on the benefits of using ECS are reported in Appendix I. Benefits to students, teachers, parents and transition coordinators are given. As a result of this project, standardization data in reading and math related to employability, are available for the first time for secondary students in grades eleven and twelve

who have specific learning disabilities. CASAS will conduct further studies on accommodations used with this age group in the future. No longer will schools have to rely on statistics from norm-referenced academic tests or those available from vocational rehabilitation in order to receive norm information. Practical and functional basic skill norms in an employability context are available for students who have specific learning disabilities at the secondary school level.

The implementation of ECS and the results from this project meet the requirements of the reauthorization of the Individuals with Disabilities Education Act (IDEA) of 1997. A summary of how CASAS ECS meets these new requirements can be found in Appendix J.

IDEA requires the use of functional assessment such as the CASAS Employability Competency System (ECS). ECS has the added benefit of being standardized, which many functional assessments do not. Many states are using CASAS/ECS for state level performance indicators, and this project provides the additional benefit of normative as well as accommodation data for students who have learning disabilities.

Three problems were encountered in implementation of this project. One was introducing another assessment with students, in addition to the basic skill academic type of assessment required by the school district and/or state for accountability purposes. Even though employability assessment was_a higher priority according to local school personnel, the school had to give precedence to state assessment, which was more academic in nature and not necessarily useful for special education students. This type of academic assessment was especially used for tenth and eleventh grade students.

In some cases special education teachers felt overworked and students "dreaded" having "another" assessment. Most schools offered some employability instruction but formal and matching assessment was not part of the program. This problem reduced the number of students involved in the project in some cases.

At least four new opportunities arose for CASAS and the schools as a result of participating in this project. First, schools have employability assessment information and outcomes that can be included in IEP/ITP's to justify the reason for selected transition goals and the need for transition resources. CASAS has a model that can be replicated by other interested users, including VR counselors.

Second, as a result of this project, CASAS is conducting further studies and development for using the accommodation of a calculator for ECS Math, targeting youth and adult basic education students. In this age of technology, schools will be able to use an additional assessment tool that matches their instruction.

Third, this project has provided more information for the national debate on using accommodations for standardized testing. CASAS is continuing to gather information from the field on policies for meeting the requirements of the Americans with Disabilities Act (ADA) of 1990. This information is being used for a continuously developing CASAS ADA Policy Statement.

Administrators and state agencies often want to have a consistent assessment program in basic academic skills for all students. However, many times students enrolled in special education programs are not included because of their low scores or perceived low scores in reducing school and/or state averages in basic academic skills reported to the public. Even though transition services are required beginning at age 14 and assessment needs to be functional in the reauthorization of the IDEA Act (1997), the connection of formal assessment related to employability and transition is not part of special education curriculum in most schools. Further, VR is focusing on assessment related to work, while most schools are focusing on academic assessment.

The outcomes of this project have demonstrated that standardized employability assessment is valuable in order to determine instructional needs and programming needed for the transition from school-to-work. It behooves school personnel to evaluate the type of inclusiveness of standardized or formal assessment for all students, as the focus of curriculum becomes more focused on school-to-work activities.

Over 350 secondary students in the eleventh and twelfth grades who have specific learning disabilities, participated in the validation of the CASAS Employability Competency System (ECS) model, representing six school districts in the states of Arizona and California. The reported assessment results and curriculum management strategies of the model, including the workplace analysis procedure and training/ technical assistance available, are being disseminated through the CASAS National Dissemination Program, originally funded by the U.S. Department of Education, National Diffusion Network. Replication sites in the future will be able to implement the model using existing personnel with no additional costs, except for nominal costs of materials and training in the future. Staff training is required for implementation of the model.

The goals of this project did not change during its operation. Local school needs and requests changed the direction of some activities in three major ways. First, the paper and pencil version of the Job Profile and Individual Profile became a computerized version, in which it became much more efficient for each local school to document and use the specific skills required for local jobs and produce profiles of job related basic skill competencies for each individual.

Second, some schools that wanted to be part of the project did not implement all parts of the Employability Competency System (ECS), due to factors such as delayed funding start time and length of time needed to implement related transition curriculum (these schools chose to use the ECS Appraisal only for documentation of needed IEP goals).

Third, effective classroom accommodations transferable to the workplace became part of IEP/ITP Planning Meetings with postschool agencies, including VR. The classroom accommodations selected by the schools as the most effective and transferable to the workplace include:

1) Extended Time on Task

2) Quiet Environment without Distractions

3) Task Broken into Small Steps or 10-15 Minute Segments

4) Use of Calculator for Math Tasks

5) Oral Instructions and/or Explanations

6) Written Procedures and/or Instructions

7) Demonstration and/or Modeling of New Tasks

8) Written Sequence of Steps /Timeline /Checklist

9) Mentor for Initial Time Period

10) Picture Charts/ Stop-Watch /Other Cueing Device

The Project Manager and State Transition Specialists for the three states involved in the project, selected and supported school districts that were interested in implementing ECS. The Transition Specialists for the states of Arizona and California as well as the directors of special education for the high schools involved were highly supportive of the purpose and activities for this project. All of the students involved were identified through school district criteria and state/federal guidelines as having specific learning disabilities.

In year two, the project activities centered on using the data collection forms and implementation and field-testing of the model with a total of at least 350 students who had specific learning disabilities. These data collection forms can be found in Appendix N. Each of the two urban areas in the project served at least 100 secondary students who had specific learning disabilities. Letters of support were gathered from the State Transition Coordinators, JTPA State Coordinators, and the Special Education Department Heads /Transition Coordinators for the high schools/ districts involved in the project.

The school districts participating were Tucson Unified School District and a Tucson area Projects With Industry in Arizona and the greater East San Gabriel/Pomona Valley Transition Coalition (Los Angeles area), including Victor Valley, Saddleback and Irvine School Districts in southern California.

There were seven major steps in the implementation of ECS:

1) Establishment of basic employment-related competencies needed for entry-level jobs or job clusters.

2) Pre- and post-assessment of functional reading, math and communication skills needed for employability such as reading a job application, calculating wages on a wage statement, and listening to work-related directions.

3) Pre- and post-assessment of pre-employment and work maturity skills such as completing a job interview and working effectively under pressure.

4) Determination of appropriate curricular resources and instructional interventions for the employment related competencies needed for each individual student, using the Individual Profile, Class Profile and CASAS Instructional Materials Guide.

5) Use of the results of the post-assessment in reading, math and communication to establish postsecondary goals and outcomes for the IEP/ITP.

6) Evaluation of student outcomes in reading, math, and communication skills, including pre-employment and work maturity skills.

7) Documentation of student gains between pre- and postassessment for program improvement.

Sample pages from ECS showing sample assessment items, Individual and Class Profiles can be seen in Appendix K. The individual and class profiles were the most valuable component for the special education teachers in this project, because they could use the profiles to determine which competencies the students needed during instruction.

Some teachers also used the CASAS Pre-Employment and Work Maturity Checklists for curriculum planning and as a post-test measure to document students had acquired these skills (Appendix L). The Assessment Summary for the Checklists became an additional part of the student's Employment Portfolio and/or IEP/ITP.

Sample pages of employment related competencies from the CASAS Competency List (Appendix M) are the base for the ECS system. These competencies were identified by the CASAS National Consortium of field practitioners as being of high priority for the transition from school-to-work. The assessment tests/items, individual and class profiles, work maturity checklists, workplace analysis and the curriculum materials -all match the specific competency statements in Appendix M.

The data collection forms completed by the sites for this project are in Appendix N. The forms include the following: Field-Test Timeline, Instructions for Completing the ECS Appraisal Program Answer Sheet, Sample ECS Appraisal Answer Sheet, Accommodations Form, Instructions for Completing the ECS Pre- and Post-tests, Sample CASAS General Purpose Answer Sheet (for pre- and post-testing), and sample Parent Permission Form. The completion of these forms resulted in the information produced in the tables shown in Appendices C through F. These tables were shared among school districts and discussed at the Evaluation Workshop for the CASAS Summer Institute in June of 1997.

The individual and class profiles resulting from the assessment were the most important component for individual classroom improvement. The fmal data tables were the most valuable component of the project in order to improve the local transition program and to document needed curriculum and resources for the local board of education. Schools collecting only Appraisal results used this information as support for needed transition courses.

The management plan for this project had process evaluation questions which addressed the completion of each activity. A self-evaluation form (Appendix 0) was used as a guide for training, technical assistance, and to help the sites implement ECS during all three years of the project. Focus Groups were used in year three to discuss how to use the data tables and other results of the project for program improvement. These focus groups completed the CASAS Projects Data Evaluation forms (Appendix 0) in preparation for discussion.

Both states (Arizona and California) used the same basic definition of learning disabilities to identify the students for this project, namely "a disorder in one or more of the basic psychological processes involved in understanding or in using language, spoken or written, which may manifest itself in an imperfect ability to listen, think, speak, read, write, spell or do mathematical calculations." Both states had the majority of their students in resource support programs or special service classes.

California had some of their students in an additional specially funded Transition Partnership Program (TPP). Local sites were able to see the more positive results from students who participated in focused transition programs. The additional benefits of a transition focus can be seen in the ECS Site Report for Victor Valley Union High School District in Appendix C.

The final outcome for this project is an Employability Competency System (ECS) validated for secondary students who have specific learning disabilities. The project results were presented at numerous workshops and conferences the last three years. A listing of these presentations and training can be found in Appendix P.

Persons who are interested in the results and further plans related to this project, may contact the Project Director, Patricia Rickard at (619) 292-2900, e-mail address: rickard@casas.org or the Project Manager, Virginia (Ginny) Posey, Ph.D. at (619) 292-2900, extension 363, e-mail address: gposey@casas.org or write to either person at CASAS, 8910 Clairemont Mesa Blvd., San Diego, CA 92123.

Persons interested in the products used in this project, such as the ECS assessment tool, Instructional Materials Guide, QuickSearch, Computerized Workplace Analysis, and training opportunities, can call CASAS Customer Service at (619) 2922900, extension 310.