RSS

“Writeplacer Plus in Place: An Exploratory Case Study” by Anne Herrington and Charles Moran (from Machine Scoring of Student Essays: Truth and Consequences, pages 114-127)

Rank: 2-3 (Herrington and Moran are suspicious enough to be opposed to the decision to implement Writeplacer, but they do see its upsides (cost/efficiency).

Argument: The researchers are convinced of Writeplacer’s cost-effectiveness as a placement exam scorer (or, at least, they see how it is a cheaper and more efficient method than hand-scoring), but they are troubled by the removal of human assessors for multiple reasons.

Assumptions: This is an interview-based case study of the implementation of Writeplacer Plus (as an entrance-level sorting device for newly-registered students) at Valley College). The researchers interviewed administrators, teachers, and students about the use of Writeplacer. Writeplacer, in this situation, was used as a one-time test (meant to act as a filter), and so, the administrators see no problem using it as a filter (since it’s not functioning within the classroom itself).

Points of Interest: 

  • The core (and fascinating) problem that the researchers isolate is that the students are writing not to humans, but to computers (a problem that extends beyond this study to the greater realm of AES) (114). They wonder how knowledge of this different audience could affect students’ rhetorical practice.
  • Administrators and faculty have distinctly opposed opinions about Writeplacer (the former think of Writeplacer as a great “filter” (124); the latter are suspicious and don’t want to be blindsided (because when they were hand-scoring exams, they could get an idea of what kind of writing they would have to work with). Students were more likely to shift between those positive and negative poles of opinion. Overall, they were shocked (specifically, some approved, some disapproved, but all wanted actual human teachers in their courses).
  • No matter what, everyone involved (essentially) was in favor of keeping AES out of the actual classroom (even though Writeplacer Plus was justified by administrators as a one-time “filter”).
 
Leave a comment

Posted by on November 14, 2013 in Uncategorized

 

Chaitanya Ramineni “Validating automated essay scoring for online writing placement”

Rating: 5; This is a quantitative study about AES as a way to assess placement tests, so this article feels like a more objective (or maybe just a less *explicitly* opinionated) piece of writing.

Argument: Based on a quantitative study with 879 participants under timed conditions, Ramineni found that students perform better with prompts that are tailored for specific universities, that the AES system, Criterion, provided for an assessment that better distinguished writing ability from general academic ability (measured by GPA), and testing conditions (proctored v. nonproctered conditions) had a statistically insignificant effect.

Assumptions:

  • AES is direct writing assessment
  • all digital platforms are the same; in other words, the Criterion platform mimics any other platform: Students can submit writing samples in a digital platform that reflects the contemporary communication environment (43)
  • AES is a learning support tool & that is always the case
  • scores generated from admissions test are not suitable for making placement decisions

Points of interest

  • She makes some important calls for further research based on what is missing in her study — issues with the sample
  • Her methods section is really thorough and a good example of method sections generally
 
Leave a comment

Posted by on November 13, 2013 in Uncategorized

 

Deane, Brent and Townsend

For this week, I read Paul Deane’s article “On the relation between between automated essay scoring and modern views of the writing construct” as well as Edward Brent and Martha Townsend’s chapter “Automated Essay Grading in the Sociology Classroom: Finding Common Ground.” On the whole, these two pieces leave me with a mixed view of AES. I think it can be useful, but within a narrow set of circumstances, and never as a means to completely replace a human reader. My analysis of each piece will treat the particular implications of each article.
Deane
Rank: 7
Arguments
Objections to AES are based on objections to the general construct of writing employed in standardized testing. Objections to AES, then, can be seen to speak less to any particular feature of the technology than to the construct of writing employed in the assessment of student writing.
Assumption
The writing construct of AES is a valid construct.

POI
This article raises interesting questions about the kind of writing construct we need to assume in order to oppose or defend AES.

Brent and Townsend
Rank: 5
Arguments
AES can be helpful, but only in limited situations and never as a replacement for a human reader.
Assumptions
AES challenges the value of writing as communication and contextual.
POI
Brent, a Sociology instructor, was able to use AES with some success in a large, 200 person introductory sociology class, where the focus is on key concepts and terms. But he explicitly mentions that he would not use AES in a more research-based class.

 
Leave a comment

Posted by on November 13, 2013 in Uncategorized

 

“The Meaning of Meaning: Is a Paragraph More Than an Equation?” by Ericsson

Ericsson is a critic of AES – I’d rate her a 1.

Argument: Machines cannot read and do not know the meaning of the text.  Any company claiming that their AES system can understand the meaning of a text is misleading people.  Rhetoricians, teachers, psychologists, and testing experts should be part of conversations about writing assessment.  Ericsson also criticizes AES using rhetorical theory.  She draws on Richards and Ogden’s The Meaning of Meaning to argue that words by themselves mean nothing, that all meaning is interpreted.  (Many of us read Richards and Ogden in Rhetorical Theory)  AES is built on the belief that meaning resides in the words, which the computer can quantify.  Students will not learn to write for real audiences or how to use language if they have to learn to write for a computer AES system.

Assumptions:

  • That AES is always bad in all pedagogical situations
  • That writing’s most important function is making meaning

Points of Interest:

  • Ericsson makes some compelling arguments.  However, couldn’t many of the same criticisms be leveled at the holistic scoring of standardized writing assessments?  I don’t think the essay high schoolers write on the state test or the essay on the SAT or the essay on the GRE are interested in meaning making.  Shouldn’t she be arguing for larger changes in how we teach and assess writing, which would include but not be limited to rejecting AES?
  • “A machine that equates meaning to a combination of word+word+word reduces the reader/word relationship to a one-dimensional ‘stimulus-response’ connection” (p. 32).
  • “In a sociocultural perspective, meaning cannot exist in isolation from the social and cultural milieu in which those meanings are made” (p. 36).
  • “Students who learn to write for these machines will see writing and composing as a process of getting the write words in the ‘bag of words’ without a concern for a human audience or any legitimate communicative purpose” (p.37).
 
Leave a comment

Posted by on November 12, 2013 in Uncategorized

 

William W. Zeigler Computerized Writing Assessmen: Community College Faculty Find Reasons to Say “Not Yet”

Rank: 4 – While they conclude that AES should not currently be used for making placement decisions at their community college, they were at least open to completing a trial run, and they also still use high SAT scores to exempt students from taking the essay placement exam (140).

Argument: “e-Write could not produce valid writing placements” (140), because:

  • 82% of students received a score of 5 or 6 on an 8 point scale (which made it difficult for faculty to discern any meaningful difference between scores other than the extremely high or extremely low)
  • AES did not do any better in correlating placement with student grades than faculty placement
  • and 25% of the student essays were unable to be scored by the machine, resulting in higher costs and longer delays in receiving score results, which defeated the purpose for them in considering the use of AES for their placement decisions

Assumptions: While faculty did not object to running the pilot study, the author freely admits that most English faculty would “agree with Joanne Drechsel’s (1999) objections to computerized evaluation of writing: it dehumanizes the writing situation, discounts the complexity of written communication, and tells student writers that their voice does not deserve a human audience” (139).

Points of Interest: While I agreed with the authors that the scale (and the distribution of student scores on that scale) was too narrow to be very useful, I did think that some of their findings regarding the breakdown of scores into various areas were more fruitful: a high score in the “content” area (which is not defined) had the highest correlation with student success, compared to the “conventions” area, where more of the students who scored lower on this category were actually more successful than students who scored higher.

 
Leave a comment

Posted by on November 12, 2013 in Uncategorized

 

Bill Condon: “Large Scale Assessment”

I’m not sure that I would necessarily place Condon on the scale because he argues that AES is actually a red herring in our discussion of writing assessment. He is against it, though, so I suppose I’d rate him as a 1 if I had to.

Argument:

Discussing AES is actually not useful for our discussion. Rather, we should be focusing on the invalid constructs of writing that current assessments purport to measure. In place of this invalid assessments, Condon suggests using assessments that are richer and provide fuller descriptions of writing (see the scale that was posted much earlier this semester on the blog).

Assumptions:

-writing assessments should draw from multiple pieces of student work
-writing assessments should provide as much information as possible about the student

Points of Interest:

-we already looked at and discussed Condon’s scale, but you can see the context he provides for it on p 104-105
-106-107 Condon states that we already have valid assessments, so why not use those? He seems to think that AES might possibly catch up with current, valid assessments.

 
Leave a comment

Posted by on November 12, 2013 in Uncategorized

 

Chris Anson: “Can’t Touch This: Reflections on the Servitude of Computers as Readers”

I suppose I would say that Anson falls as 5 on a scale of 1-10 because he presents a clear argument that computers cannot actually read and interpret texts in the same ways that humans can, but he tentatively asserts that e-raters could be used to provide some formative assessment in the classroom.

Argument: Anson argues that computers cannot assess student writing because they are unable to read for context and interpret writing in the same ways that humans can. He presents information about research that has been done using AI, and he provides multiple examples that serve to show that using and interpreting natural language is a skill that computers do not yet possess. He also gives an example of his writing a summary containing intentionally false information alongside the e-rater’s response and high score.

Assumptions:

-to assess writing, we need to interpret the content and understand the context
-“factual” information is important for certain types of writing
-“counting” could reveal useful pieces of information about student writing
-computers might one day have natural language abilities

Points of Interest:

-p 42 has a useful example of why natural language is difficult for AI to work with
-p 47 has a discussion of how multiple interpretations are part of the human reading process, which further complicates a machine’s ability to “read” text

 
Leave a comment

Posted by on November 12, 2013 in Uncategorized