Writing Assessment - Online Interactive Training System

In September of 2001, the North Carolina Department of Public Instruction (NCDPI) Testing Section, NCDPI English Language Arts, and North Carolina State University-Technical Outreach for Public Schools (NCSU-TOPS) staff met and began the process that would result in new writing assessments and scoring procedures for grades 4, 7, and 10. Writing committees for each grade level were established that included regular education teachers, ESL teachers, exceptional children teachers, curriculum supervisors, principals, and university faculty. The North Carolina Writing Assessment Scoring Model was developed by NCDPI Testing Section and NCSU-TOPS staff, refined by the writing assessment advisory committees, and approved by the State Board of Education on January 9, 2003.

The selection of a scoring procedure to be used in a writing assessment program must be influenced by the program's purpose and by knowledge about the information that the program is expected to provide. The State Board of Education, representing the interests of North Carolina students, educators, and other stakeholders, proposed to collect and provide information using the North Carolina Writing Assessment Scoring Model to achieve these ends.

The North Carolina Writing Assessment Scoring Model is comprised of a content component with a 1-4 score scale and a conventions component with a 0-2 score scale. To report a total writing scale score for each student, the score is computed by combining the content and conventions scores using the following model:

Total Writing Assessment Score = sum of the (content component scores from two independent readers multiplied by 2) + the sum of the (conventions component scores from two independent readers multiplied by 1).

The combined raw scale score may be a minimum of 4 and a maximum of 20 for a given student.

On October 2, 2003, the State Board of Education approved the Writing Assessment Achievement Levels for grades 4, 7, and 10. Those achievement levels are as follows and can be found in the State Board of Education Policy Manual listed as HSP-C-018:

The State Board also decided that the writing results for grades 4, 7, and 10 would not be part of the ABCs Accountability Program for the 2003-04 school year, although the results will be reported by school and be used as part of the Student Accountability Standards (SAS) policy for grades 5 & 8. The Board did say that results for grades 4 and 7 will be part of the performance composite scores for the ABCs starting with the 2004-05 school year.

The SBE Ad Hoc Writing Committee met on November 20, 2003 to begin discussions about when the results for grade 10 will become part of ABCs Accountability Program. The Ad Hoc Writing Committee will also discuss possible revisions to the SAS policy to consider including the grade 10 writing results. Discussion of the recommendations of the Ad Hoc Writing Committee occurred at the January 2004 State Board of Education meeting and has yet to be determined.

Students will receive the following information from operational statewide writing assessments: (a) point totals for content, (b) point totals for conventions, (c) total writing scores, and (d) Achievement Level rankings. A review procedure will be incorporated into the scoring process for those students whose Total Writing Score falls within one point of the cut line at Achievement Level III (Total Writing Score = 11). This procedure will preclude an LEA appeal mechanism, as conducted under the previous focused holistic scoring system.

The North Carolina Writing Assessment at Grades 4, 7, and 10 reports a separate score for content and for conventions (sentence formation, usage, and mechanics) as part of a students’ total writing score. Students using the dictation to scribe accommodation will not receive a conventions score as the use of this accommodation invalidates the conventions component of the writing assessment. This does not mean, however, that students using this accommodation cannot achieve a proficient score on this assessment.

The first objective of reader training will be to remove any biases that readers might hold concerning the related importance and appropriateness of certain features of written composition. Training materials will be designed to give clear definition to each feature that readers will evaluate and will reduce subjectivity to the lowest possible level. It will be mandatory that readers accept these definitions.

Before scoring begins, test booklets will be divided so that student and school identification information is separated from the response. Thus, factors that could potentially influence reader bias such as geographical location, ethnicity, and gender will be minimized. To ensure accuracy during the scoring process, readers will be required to understand the definitions of each feature and its application. For this reason, inter-rater reliability and validity will be monitored closely throughout the scoring process.

Strict security guidelines will be observed during the scoring process. Readers will sign a test security agreement, wear an identification badge at all times while in the scoring area, and leave all scoring materials in the scoring rooms. Security personnel will monitor compliance with all security guidelines.

The purpose of the Online Interactive Training System is to provide teachers, administrators, parents, and students the opportunity to practice applying the North Carolina Writing Assessment Scoring Model to student responses.

The process of training scorers involves a strict adherence to the guidelines set forth by the North Carolina Standard Course of Study and the Writing Advisory Committees. The training of scorers is monitored daily by the North Carolina Department of Education Testing Section Project Manger. The Online Interactive Training System is not design to prepare individuals for scoring the North Carolina Writing Assessment at Grades 4, 7, and 10. This program intends to assist stakeholders in the application of the North Carolina Writing Assessment at Grades 4, 7, and 10 Scoring Model.