Interactive Training System
Carolina Scoring Model
September of 2001, the North Carolina Department of Public Instruction
(NCDPI) Testing Section, NCDPI English Language Arts, and North
Carolina State University-Technical Outreach for Public Schools
(NCSU-TOPS) staff met and began the process that would result
in new writing assessments and scoring procedures for grades 4,
7, and 10. Writing committees for each grade level were established
that included regular education teachers, ESL teachers, exceptional
children teachers, curriculum supervisors, principals, and university
faculty. The North Carolina Writing Assessment Scoring Model was
developed by NCDPI Testing Section and NCSU-TOPS staff, refined
by the writing assessment advisory committees, and approved by
the State Board of Education on January 9, 2003.
selection of a scoring procedure to be used in a writing assessment
program must be influenced by the program's purpose and by knowledge
about the information that the program is expected to provide.
The State Board of Education, representing the interests of North
Carolina students, educators, and other stakeholders, proposed
to collect and provide information using the North Carolina Writing
Assessment Scoring Model to achieve these ends.
North Carolina Writing Assessment Scoring Model is comprised of
a content component with a 1-4 score scale and a conventions component
with a 0-2 score scale. To report a total writing scale score
for each student, the score is computed by combining the content
and conventions scores using the following model:
Writing Assessment Score = sum of the (content component scores
from two independent readers multiplied by 2) + the sum of the (conventions
component scores from two independent readers multiplied by 1).
combined raw scale score may be a minimum of 4 and a maximum of
20 for a given student.
October 2, 2003, the State Board of Education approved the Writing
Assessment Achievement Levels for grades 4, 7, and 10. Those achievement
levels are as follows and can be found in the State Board of Education
Policy Manual listed as HSP-C-018:
The State Board
also decided that the writing results for grades 4, 7, and 10 would
not be part of the ABCs Accountability Program for the 2003-04
school year, although the results will be reported by school and
be used as part of the Student Accountability Standards (SAS) policy
for grades 5 & 8. The Board did say that results
for grades 4 and 7 will be part of the performance composite scores
for the ABCs starting with the 2004-05 school year.
The SBE Ad Hoc Writing Committee met on November 20, 2003 to begin
discussions about when the results for grade 10 will become part of
ABCs Accountability Program. The Ad Hoc Writing Committee will also
discuss possible revisions to the SAS policy to consider including
the grade 10 writing results. Discussion of the recommendations of
the Ad Hoc Writing Committee occurred at the January 2004 State Board
of Education meeting and has yet to be determined.
Students will receive the following information from operational statewide
writing assessments: (a) point totals for content, (b) point totals
for conventions, (c) total writing scores, and (d) Achievement Level
rankings. A review procedure will be incorporated into the scoring
process for those students whose Total Writing Score falls within
one point of the cut line at Achievement Level III (Total Writing
Score = 11). This procedure will preclude an LEA appeal mechanism,
as conducted under the previous focused holistic scoring system.
The North Carolina Writing Assessment at Grades 4, 7, and 10 reports
a separate score for content and for conventions (sentence formation,
usage, and mechanics) as part of a students’ total writing
score. Students using the dictation to scribe accommodation will
not receive a conventions score as the use of this accommodation
invalidates the conventions component of the writing assessment.
This does not mean, however, that students using this accommodation
cannot achieve a proficient score on this assessment.
first objective of reader training will be to remove any biases
that readers might hold concerning the related importance and appropriateness
of certain features of written composition. Training materials will
be designed to give clear definition to each feature that readers
will evaluate and will reduce subjectivity to the lowest possible
level. It will be mandatory that readers accept these definitions.
Before scoring begins, test booklets will be divided so that student
and school identification information is separated from the response.
Thus, factors that could potentially influence reader bias such
as geographical location, ethnicity, and gender will be minimized.
To ensure accuracy during the scoring process, readers will be required
to understand the definitions of each feature and its application.
For this reason, inter-rater reliability and validity will be monitored
closely throughout the scoring process.
Strict security guidelines will be observed during the scoring process.
Readers will sign a test security agreement, wear an identification
badge at all times while in the scoring area, and leave all scoring
materials in the scoring rooms. Security personnel will monitor
compliance with all security guidelines.
of the Online Interactive Training System
purpose of the Online Interactive Training System is to provide
teachers, administrators, parents, and students the opportunity
to practice applying the North Carolina Writing Assessment Scoring
Model to student responses.
process of training scorers involves a strict adherence to the guidelines
set forth by the North Carolina Standard Course of Study and the
Writing Advisory Committees. The training of scorers is monitored
daily by the North Carolina Department of Education Testing Section
Project Manger. The Online Interactive Training System is not design
to prepare individuals for scoring the North Carolina Writing Assessment
at Grades 4, 7, and 10. This program intends to assist stakeholders
in the application of the North Carolina Writing Assessment at Grades
4, 7, and 10 Scoring Model.