CASAS Project OUTREACH (Opportunities Utilizing Teaching Resources with an Educational Assessment and
Curriculum Hierarchy) Final Report
1994-1997
Submitted by Virginia Posey, Project Manager
Comprehensive Adult Student Assessment System
8910 Clairemont Mesa Blvd.
San Diego, CA 92123
619-292-2900
Contact Virginia Posey for more information: gposey@casas.org
A. Executive Brief
The purpose of CASAS' Project OUTREACH was to validate and
disseminate the Employability Competency System (ECS), a proven comprehensive employability assessment model, with 689 secondary students primarily in grades eight through ten who had specific learning disabilities. The students represented fourteen school districts in the states of Arizona, California and Idaho.The validation process provided information that resulted in: 1) identification of basic skills required to enter specific jobs/job areas or vocational training in the school or community; 2) determination of basic skill competencies needed by students, that were related to these jobs and training requirements; and 3) targeted curriculum and instructional strategies in order for students to master the skills needed for the transition from secondary school to postsecondary environments.
B. Conclusions
CASAS developed the Employability Competency System (ECS) to serve youth programs and to meet the requirements of the Job Training Partnership Act (JPTA). Some secondary programs found the system useful for youth who had learning disabilities in their transition from school-to-work. However, users wanted verified mean scores, which were comparable for their students, to use as a guide for instructional purposes.
The major conclusion resulting from this outreach project is that ECS is a valid and an appropriate assessment and curriculum management model to use for determining the employability skills of secondary students who have specific learning disabilities. Following is a summary of outcomes from this three-year project that support the use of ECS for this population.
Outcomes
coded is found in Appendix H. A description of how to obtain a copy of this Guide is described in the marketing and dissemination section for this project. A total of more than 1400 commercially available curriculum materials are listed in this Guide.Transition counselors and teachers for the sites used CASAS' Job Profiles for determining "Skills Employers Want" in the community. The CASAS Individual Profile was used to determine competencies needed by students in order to perform the selected jobs/job areas. A sample report on "Skills Employers Want" can be found in Appendix A.
A computerized version of this Job Profile or "job bank" of local job requirements and Individual Profile of employment related basic skill competencies for the student were developed through a related CASAS federal transition project (H078C40024). A sample Job Profile and Individual Profile from the computerized version, that illustrates how this process works can be found in Appendix B.
2) Transition counselors/ teachers used the reports on "Skills Employers Want" and the Job and Individual Profiles to help students select jobs that matched their basic employability skills and progress in these skills. Assessment results using ECS pre-tests in reading, math and communication skills in a functional employability context were used to determine initial skill levels and for planning instruction focusing on transition.
ECS post-tests were used to measure progress in the skills required. The pre- and post-test results were used for improving special education curriculum and instruction as well as for establishing postsecondary goals in Individual Education and Transition Plans (IEP/ITP's).
3) Information collected on employability skills of students included a demographic comparison, initial appraisal scores, reading and math scaled score ranges, mean pre- and posttest scores, mean learning gains, and accommodation information. Tables displaying the results for the fourteen sites in the project can be found in Appendix C. The tables include students in grades eleven and twelve in southern Arizona and California, who were part of another related CASAS federal transition grant (H078C40024).
Included at the beginning of Appendix C is the CASAS Level Descriptors for ECS, which is used to interpret the meaning of the scaled scores. Each school district site and state has a series of tables on demographic and assessment results that can be used for instructional planning and to give various publics accountability information. The State Reports can be found in Appendix D and the final Project OUTREACH Report can be found in Appendix E.
3) A total of 185 titles of competency- based instructional materials related to employability were validated for students who have learning disabilities through this outreach project.
These titles are included in the CASAS Instructional Materials Guide (1997-98). A listing of the titles appropriate for students who have learning disabilities can be found in Appendix G.
Sites are able to easily identify the coded materials and have been able to select materials that link with competencies students need. A sample of how the materials are linked and
4) A computerized version of this guide called Instructional Materials QuickSearch (1997) has been completed through another CASAS project and is also available. School personnel are able to search for and print validated materials by competency area, instructional level (Pre A to D), and appropriateness for learning disabilities. The package is available for Windows, Macintosh. and the Power Mac. Both hard copy and computerized versions have a cost for printing and reproduction.
C. Summary of Assessment Results
Tables illustrating the assessment results for all of the secondary students who had specific learning disabilities participating in this outreach project are found in Appendix E. The overall results of the OUTREACH project are discussed below in terms of demographic background, pre-test performance, progress between pre- and post-test scores, progress by hours of instruction, results for the three states involved in the project and results using special test accommodations.
Demographic Profile
Demographic background (Table 1) on the students included gender, age, grade level completed, native language and ethnic background. The majority (67.6%) of the students were male and over half (53. 1%) were 15 or 16 years of age. The vast majority (87.0%) reported English as their native language and an additional 9.5 percent reported Spanish. Most students indicated Caucasian (46.9%) or Hispanic (29.8) as their ethnic background. An additional 10.3 percent indicated Black.
Pre-Test Performance
A total of 689 students were assessed in reading and 682 students were assessed in math using the ECS Appraisal test (Tables 2 and 3). As can be seen in the Site Reports in Appendix C, some school sites only used the Appraisal for assessment. However, the purpose of the Appraisal Test in reading and math is to determine placement into the appropriate instructional level for beginning instruction.
As can be seen in Tables 2 and 3 (Appendix E), students as a whole tend to score a little lower on the pre-test (40.9 percent in the 211 - 220 scaled score range in reading) compared to the appraisal (41.8 percent in the 221 - 229 scaled score range in reading). The pre-test has more items at one instructional level ranging from A (lowest) through D (highest), whereas the appraisal has an even distribution of test items across all four levels (A to D) -
Progress Between Pre- and Post-Test Scores
The mean reading pre-test score was 213.9 and the mean reading post-test score was 217.7 for the 271 students taking both tests (Table 4), resulting in a mean learning gain of 3.8 points. As shown in the CASAS Level Descriptors for ECS at the beginning of Appendix C, the majority of students post-tested at the higher end of the B level.
Students at this level function at a basic literacy level and may have difficulty pursuing employment above an entry level position. However, they most likely will be able to perform job related tasks such as completing a simple order form and job application, and handle jobs/training that involve following basic oral or written instructions and diagrams if they are clarified orally.
The learning gain of 3.8 points (n=271) in reading on the CASAS scale for the secondary students who had specific learning disabilities in this study. can be compared to other CASAS samples such as Adult Basic Education students who gained a mean of 5.5 points (n=943), English as a Second Language students who gained a mean of 5.5 points (n=7,623) and Adult Special Education students (most of whom have developmental disabilities) who gained a mean of 2.5 points (n=1,339) (Student Progress and Goal Attainment for Federally Funded ABE Programs in California 1996-97). CASAS has been collecting mean scores for these populations since 1980.
Progress in math is shown in Table 5 in Appendix E. The mean math pre-test score for the students who had specific learning disabilities was 208.8 and the mean math post-test scaled score was 211.1 resulting in a mean learning gain of 2.3 (n= 290). As can be seen in the individual school Site Reports in Appendix C, the gain in reading scores is not always higher than the gain in math.
Hours of Instruction
Learning gains were greater when there were 90 to 139 hours (usually a semester) of instruction between pre- and posttesting, than for 40 to 89 hours of instruction (usually a 9-week unit). The limitations to be considered in reporting by hours of instruction include: (1) The number of hours of instruction were reported for only about half of the students participating in pre- and post-testing, and (2) the quality and focus on transition during instruction were not studied.
The mean learning gain in reading and math by hours of instruction are reported in Tables 7 and 8 in Appendix E. The mean learning gain in reading was 3.6 points for 90 to 139 hours of instruction (n=46) as shown in Table 7, compared to 3.0 points for 40 to 89 hours of instruction (n=101). As shown in Table 8, the mean learning gain in math was 5.0 points for 90 to 139 hours of instruction (n=42), compared to 2.2 points for 40 to 80 hours of instruction (n=106).
State Results
The mean appraisal score, mean pre- and post-test scores and mean learning gain. in reading and math, for the school sites and states in this outreach project are reported in Tables 9. 10, 11 and 12 in Appendix E. The mean learning gains in reading and math for the states participating in this project can also be found in the State Reports for Arizona, California, and Idaho in Appendix D. The State Reports include older students from the states participating in another related CASAS project (H078C40024). The school sites within each state participating in both projects are listed in these state reports.
Rural districts represented the state of Idaho while suburban and urban districts represented the states of Arizona and California. The average learning gain in reading was highest for the state of Idaho with a mean of 4.1 (n=42), the next highest gain in reading was for the state of California with a mean of 3.9 (n= 127) and the least gain in reading was for the state of Arizona which had a mean of 3.1 (n=162).
The average learning gain in math was also highest for the state of Idaho with a mean of 4.4 (n=49), the next highest gain in math was again for the state of California with a mean of 2.3 (n= 130) and the least gain in math was for the state of Arizona which had a mean of 1. 6 (n= 178).
Both Arizona and Idaho had approximately forty percent fewer students take the pre- and post-tests as compared to the number who took the appraisal. Only twenty percent of California's students participated in the pre- and post-tests as compared to the number that participated in the Appraisal test. However, some districts chose to participate in the Appraisal test only, even though the purpose of the test is to determine the appropriate level to begin instruction. Evaluation focus groups representing some of the teachers revealed that there was a high drop-out and absentee rate among their special education students at the post-test time.
Test Accommodations
Special test accommodations used are reported in Table 6. The most frequent accommodation used for reading was extra time and the most frequent accommodation for math was the calculator. The mean pre-test score, mean post-test score and mean learning gain for both reading and math, when the accommodation was used and when it was not, are reported in Tables 13, 13a, 14 and 14a. Tests of significance were not done for these groups but were done for the combined group in this outreach project and another related CASAS federal transition project (H078C40024).
The pre-test score means, post-test score means and mean learning gains in reading and math for both CASAS Federal Transition Projects (Projects TAC/OTR) are reported in Tables 13, 13a, 14 and 14a in Appendix F. The ECS tests are not timed, but most students usually complete them in an hour.
When the accommodation of extra time was used, it was stipulated that no more than time-and-a-half be used to take the tests. CASAS tests were developed to be independent of time in order to complete the items. Also CASAS math tests were initially validated without the use of a calculator. It was stipulated that these accommodations be used only when the need was indicated on the IEP and when they were also used for instruction.
Table 13 and 13a in Appendix F report mean scores for reading. When the accommodation of extra time was used, the mean reading pre-test score was 215.5, the mean reading posttest score was 219.3, resulting in a mean reading learning gain of 3.8 points (n=137) As shown in Table 13a, when the accommodation of extra time was not used, the mean reading pre-test score was 213.2, the mean reading post-test score was 216.5. resulting in a mean reading learning gain of 3.3 points (n= 194).
Table 14 and 14a in Appendix F report mean scores from the two CASAS federal transition projects (Projects TAC/OTR) for math. When the accommodation of a calculator was used, the mean math pre-test score was 210.0, the mean math post-test score was 211.8, resulting in a mean math learning gain of 1.8 points (n=218) As shown in Table 14a, when the accommodation of a calculator was not used, the mean math pre-test score was 206.5, the mean math post-test score was 209.4, resulting in a mean math learning gain of 2.9 points (n=139).
As shown in Tables 13 through 14a in Appendix F, the observed differences in learning gains between students using and not using the accommodations of extra time for reading and a calculator for math, were not found to be statistically significant (p<.05). As shown in Tables 15, 15a, 16 and 16a in Appendix F, the use of an accommodation results in a consistently higher learning gain only for the lower (Level A) scaled score ranges (181 - 190 and 191 - 200). This is
consistent with other CASAS learning gain results, especially for students in English as a Second Language classes.
D. Benefits of Using ECS
Results of interviews with a sample of site personnel on the benefits of using ECS are reported in Appendix I. Benefits to students, teachers, parents and transition coordinators are given. As a result of this project, standardization data in reading and math related to employability, are available for the first time, for over 650 secondary students who have specific learning disabilities. No longer will schools have to rely on statistics from norm-referenced academic tests or those available from vocational rehabilitation in order to receive norming information. Practical and functional basic skill norms in an employability context are now available for students who have specific learning disabilities at the secondary school level.
Requirements for IDEA
The implementation of ECS and the results from this project meet the requirements of the reauthorization of the Individuals with Disabilities Education Act (IDEA) of 1997. A summary of how CASAS ECS meets these new requirements can be found in Appendix J.
IDEA requires the use of functional assessment such as ECS. ECS has the added benefit of being standardized, which many functional assessments do not. Many states are using CASAS/ECS for state level performance indicators, and this project provides the additional benefit of normative as well as accommodation data for students who have learning disabilities.
IDEA requires that the IEP include a statement of the student's transition goals by age 14. This project has demonstrated that
ECS provides clear information and documented evidence for establishing goals related to assessment and instruction. This project has also demonstrated that ECS can be used effectively with students at age 14, and can be continued throughout high school to measure continuous progress towards employability and transition outcomes. As legislation changes in the future to meet new demands, CASAS will continue to be in the forefront of meeting these demands and needs of students who have disabilities.
Problems Encountered
The major problem encountered in implementation of this project was introducing another assessment with students, in addition to the basic skill academic type of assessment required by the school district and/or state for accountability purposes. Even though employability assessment was a higher priority according to local school personnel, the school had to give precedence to state assessment, which was more academic in nature and not necessarily useful for special education students.
In some cases special education teachers felt over-worked and students "dreaded" having "another" assessment. Most schools offered some employability instruction but formal and matching assessment was not part of the program. This problem reduced the number of students involved in the project in some cases.
New Opportunities
At least four new opportunities arose for CASAS and the schools as a result of participating in this outreach project. Schools have employability assessment information and outcomes that can be included in IEP/ITP's to justify the reason for selected transition goals and the need for transition resources. CASAS has a model that can be replicated by other interested users.
As a result of this project, CASAS is conducting further studies and development for using the accommodation of a calculator for ECS Math, targeting youth and adult basic education students. In this age of technology, schools will be able to use an additional assessment tool that matches their instruction.
This project has provided more information for the national debate on using accommodations for standardized testing. CASAS is continuing to gather information from the field on policies for meeting the requirements of the Americans with Disabilities Act (ADA) of 1990. This information is being used for a continuously developing CASAS ADA Policy Statement.
Some of the participating schools in this project have expressed a desire to use the employability model with students who have other types of disabilities as well as with all students in the school for school-to-work activities. This outcome may provide direction for future validation and dissemination activities. The Transition Institute at the University of Illinois was invaluable in providing suggested resources and this project would not be possible without OSERS sponsorship.
Recommendations
Administrators and state agencies often want to have a consistent assessment program in basic academic skills for all students. However, many times students enrolled in special education programs are not included because of their low scores or perceived low scores in reducing school and/or state averages in basic academic skills reported to the public. Even though transition services are required beginning at age 14 and assessment needs to be functional in the reauthorization of the IDEA Act (1997), the connection of formal assessment related to
employability and transition is not part of special education curriculum in most schools.
The outcomes of this outreach project have demonstrated that standardized employability assessment is valuable in order to determine instructional needs and programming needed for the transition from school-to-work. It behooves school personnel to evaluate the type of inclusiveness of standardized or formal assessment for all students, as the focus of curriculum becomes more focused on school-to-work activities.
E. Overall Purpose of Project
The major goal of this project over a three-year period was to validate (customize, implement, evaluate and disseminate) a proven comprehensive employability assessment model that would assist secondary students who have specific learning disabilities to transition from secondary school to postschool environments. These environments include community-based work settings, vocational education and training, adult and continuing education, postsecondary education and/or employment.
A total of 689 secondary students who have specific learning disabilities participated in the validation of the CASAS Employability Competency System (ECS) model, representing fourteen school districts in the states of Arizona, California and Idaho. The reported assessment results and curriculum management strategies of the model, including the workplace analysis procedure and training/ technical assistance available, are being disseminated through the CASAS National Dissemination Program, originally funded by the U.S. Department of Education, National Diffusion Network. Replication sites in the future will be able to implement the model using existing personnel with no additional costs, except for nominal costs of materials - in the future. Staff training is required for implementation of the model.
Redirection of Project During Operation
The goals of this project did not change during its operation. Local school requests changed the direction of some activities in two major ways. First, the paper and pencil version of the Job Profile and Individual Profile became a computerized version, in which it became much more efficient for each local school to document and use the specific skills required for local jobs and produce profiles of job related basic skill competencies for each individual (shared with a related CASAS federal transition project serving older students H078C40024).
Second, some schools that wanted to be part of the project did not implement all parts of the Employability Competency System (ECS), due to factors such as delayed funding start time and length of time needed to implement related transition curriculum (these schools chose to use the ECS Appraisal only for documentation of needed IEP goals).
Collaboration/ Context
The Project Manager and State Transition Specialists for the three states involved in the project, selected and supported school districts that were interested in implementing ECS. The Transition Specialists for the states of Arizona, California and Idaho and the directors of special education for the high schools involved were highly supportive of the purpose and activities for this project.
The school districts were also selected because they had collaborative arrangements with local businesses and the local Department of Rehabilitation. All of the students involved were
identified through school district criteria and state/federal guidelines as having specific learning disabilities.
Coordinators were selected from the northern, central and southern regions of each of the three states. The selection process enabled a representation from rural, suburban and urban districts.
F. Project Activities /Procedures
were used for this determination and to establish goals and outcomes for each student's Individual Education and Transition Plan (IEP/ITP).In year one, the project activities centered around training the school site personnel in the Employability Competency System (ECS) and determining the employability skill levels of students. The CASAS Workplace Analysis and ECS assessment
In year two, the project activities centered on using the data collection forms for implementation and field-testing of the model with a total of at least 600 students who had specific learning disabilities. Each of the three states in the project served approximately 200 secondary students who had specific learning disabilities. Letters of support were gathered from the State Transition Coordinators, JTPA State Coordinators, and the Special Education Department Heads /Transition Coordinators for the high schools/ districts involved in the project.
The school districts participating were Tucson Unified School District and a Tucson area Projects With Industry, Paradise Valley High School District (Phoenix area) and Mesa Public Schools (Phoenix area) in Arizona-, the greater East San Gabriel/Pomona Valley Transition Coalition (Los Angeles area), including Victor Valley, Saddleback and Irvine School Districts in southern California; East Side Union High School District in Santa Clara County in northern California; and the Canyon Owyhee School Service Agency (COSSA), Coeur d' Alene High School District and Pocatello Unified School District in Idaho.
In year three the model was evaluated and validated for secondary students who had specific learning disabilities. Information in the 1998-99 CASAS Resource Catalog, Instructional Materials Guide, as well as training and dissemination materials reflect the new results from this project.
Steps in Implementation of ECS
There were seven major steps in the implementation of ECS:
1) Establishment of basic employment- related competencies needed for entry-level jobs or job clusters.
2) Pre- and post-assessment of functional reading, math and communication skills needed for employability such as reading a job application, calculating wages on a wage statement, and listening to work-related directions.
3) Pre- and post-assessment of pre-employment and work maturity skills such as completing a job interview and working effectively under pressure.
4) Determination of appropriate curricular resources and instructional interventions for the employment related competencies needed for each individual student, using the Individual Profile, Class Profile and CASAS Instructional Materials Guide.
5) Use of the results of the post-assessment in reading, math and communication to establish postsecondary goals and outcomes for the IEP/ITP.
6) Evaluation of student outcomes in reading, math. and communication skills. including pre-employment and work maturity skills.
7) Documentation of student gains between pre- and postassessment for program improvement.
Employabilily Curriculum and Instruction
Employability curriculum and instruction focused on the attainment of competencies needed for satisfactory and long-term employment. Minimum scores for attainment of satisfactory employment outcomes were established during the third year of the project. This included correlating the CASAS level descriptors with the Dictionary of Occupational Titles, General Education Development (DOT/GED) levels. It was determined that a CASAS B Level was the minimum scaled score level in order to be prepared for a minimum entry level job. The results of the project revealed that most students were able to attain this minimum proficiency level in reading.
Sample pages from ECS showing sample assessment items, Individual and Class Profiles can be seen in Appendix K. The individual and class profiles were the most valuable component for the special education teachers in this project. because they could use the profiles to determine which competencies the students needed during instruction.
Some teachers also used the CASAS Pre-Employment and Work Maturity Checklists for curriculum planning and as a post-test measure to document students had acquired these skills (Appendix Q. The Assessment Summary for the Checklists became an additional part of the student's Employment Portfolio and/or IEP/ITP.
Sample pages of employment related competencies from the CASAS Competency List (Appendix M) are the base for the ECS system. These competencies were identified by the CASAS National Consortium of field practitioners as being of high priority for the transition from school-to-work The assessment tests/items, individual and class profiles, work maturity checklists, workplace analysis and the curriculum materials including the additional materials validated by the teachers as appropriate for students who have learning disabilities - all match the specific competency statements in Appendix M.
Data Collection Procedures
The data collection forms completed by the sites for this project are in Appendix N. The forms include the following: Field-Test Timeline, Instructions for Completing the ECS Appraisal Program Answer Sheet, Sample ECS Appraisal Answer Sheet, Accommodations Form, Instructions for Completing the ECS Pre- and Post Tests, Sample CASAS General Purpose Answer Sheet (for pre- and post testing), and sample Parent Permission Form. The completion of these forms resulted in the information produced in the tables shown in Appendices C through F. These tables were shared among school districts and discussed at the Evaluation Workshop for the CASAS Summer Institute in June of 1997.
The individual and class profiles resulting from the assessment were the most important component for individual classroom improvement. The final data tables were the most valuable component of the project in order to improve the local transition program and to document needed curriculum and resources for the local board of education.
Evaluation of Project
The management plan for this project had process evaluation questions which addressed the completion of each activity. A self-evaluation form (Appendix 0) was used as a guide for training, technical assistance, and to help the sites implement ECS during all three years of the project. Focus Groups were used in year three to discuss how to use the data tables and other results of the project for program improvement. These focus groups completed the CASAS Projects Data Evaluation forms (Appendix 0) in preparation for discussion.
All three states (Arizona, California and Idaho) used the same basic definition of learning disabilities to identify the students for this project, namely "a disorder in one or more of the basic psychological processes involved in understanding or in using language, spoken or written, which may manifest itself in an imperfect ability to listen, think. speak, read, write, spell or do mathematical calculations." All three states had the majority of their students in resource support programs or special service classes. California had some of their students in an additional specially funded Transition Partnership Program (TPP). Local sites were able to see the more positive results from students who participated in focused transition programs. The additional benefits of a transition focus can be seen in the ECS Site Report for Victor Valley Union High School District in Appendix C.
The Workplace Analysis and Individual and Class Profiles were used to redirect class curriculum as well as assist students to build needed skills and recognize limitations. The assessment process through pre- and post-testing enabled students to track their progress and find long-term employment after high school.
Marketing and Dissemination
The final outcome for this project is an Employability Competency System (ECS) validated for secondary students who have specific learning disabilities. The project results were presented at numerous workshops and conferences the last three years. A listing of these presentations and trainings can be found in Appendix P.
All 1998-99 marketing materials from CASAS will reflect the results of this project. Presentations and workshops will continue to be a high priority, especially since there are new opportunities for training and development as a result of this project.
Persons who are interested in the results and further plans related to this project, may contact the Project Director, Patricia Rickard at (6 19) 292-2900, e-mail address: rickard@casas.org or the Project Manager, Virginia (Ginny) Posey, Ph.D. at (619) 292-2900, extension 363, e-mail address: gposey@casas.org or write to either person at CASAS, 8910 Clairemont Mesa Blvd., San Diego, CA 92123.
Persons interested in the products used in this project, such as the ECS assessment tool, Instructional Materials Guide, QuickSearch, Computerized Workplace Analysis, and training opportunities, can call CASAS Customer Service at (619) 292-2900, extension 310.
Please contact CASAS for copies of appendices.
Back to program evaluation technical assistance