Shermis and Burstein (2013) メモ

langstat @langstat

A simple definition of automated essay evaluation is "the process of evaluating and scoring written prose via computer programs"… #AEE

langstat @langstat

…(Shermis and Burstein, 2013). The evaluation label intentionally recognizes that the capabilities of the technology can go beyond… #AEE

langstat @langstat

…the task of scoring, or assigning a number to an essay. (p.1) #AEE

langstat @langstat

Longer essays tend to be assigned higher scores by human raters. However, Page found that the relationship between the number of words… #AEE

langstat @langstat

…used and the score assignment was not linear, but rather logarithmic. (pp.8-9) #AEE

langstat @langstat

At lower levels of language proficiency, then, the focus of assessment is generally on linguistic issues; that is, the degree to which… #AEE

langstat @langstat

…writes have control over basic vocabulary, syntax and paragraph structure. As writers gain more control over these skills, the focus… #AEE

langstat @langstat

…can shift to higher order concerns such as development, strength of argument, and precision in language use. Finally, at the higher… #AEE

langstat @langstat

…levels of proficiency, second language writers may still retain an "accent" in writing but otherwise do not need to be distinguished… #AEE

langstat @langstat

…from first language writers in terms of assessment. (p.38) #AEE

langstat @langstat

1. The rubric is appropriate 2. Raters have appropriate qualifications 3. Raters have appropriate training … #AEE

langstat @langstat

… 4. Scoring is conducted in a way as to ensure appropriate application of the rubric (p.158) #AAE

langstat @langstat

1. AES covers the construct appropriately 2. Automated scoring models are appropriately derived … #AEE

langstat @langstat

… 3. Scoring us conducted so as to ensure appropriate scores (p.166) #AEE

langstat @langstat

In theory, human scoring is fully capable of representing the entire construct of interest for the assessment. (p.173) #AEE

langstat @langstat

While this is theoretically possible, research suggests that it is difficult to achieve as there is a constant need to guard against… #AEE

langstat @langstat

…common problems in human scoring that include inattentiveness, halo effects, sequence effects, central tendency effects, lenience, … #AEE

langstat @langstat

…severity, bias, and other undesirable characteristics of human scoring (p.173) #AEE

langstat @langstat

the most notable distinction between human and AES scoring is that with human scoring we have greater confidence in the potential for… #AEE

langstat @langstat

…the construct to be well-represented but less confidence in the conscientiousness and consistency of scoring, while AES scoring may… #AEE

langstat @langstat

…entail some inadequacies in construct representation but provide highly conscientious measures and complete consistency in the use of… #AEE

langstat @langstat

…features to determine summary scores. (p.174) #AEE

langstat @langstat

The first difficulties is that, if AES is to function as a replacement for human assessment, then it is necessary for AES validation… #AEE

langstat @langstat

…to show that machine scores measure the same construct(s) as human rating. One problem with this requirement is that we do not… #AEE

langstat @langstat

…actually have a good understanding of what human raters do in their evaluation of student essays. (p.182) #AEE