Automated Essay Scoring Software Timeline

26 Sep

NOTE: Because of the size of the file, we’re linking to it here instead of embedding it! Enjoy!

Also: web-based full-screen version

Assessment Timeline Process Memo

Jason Custer and Kendall Parris

September 2013

For our timeline project, we wanted to discuss some of the unique features that belonged to certain machine scoring systems. These different features can, arguably, constitute a kind of technological identity that is unique to each system; thus, we spent a good while trying to figure out the best way to formulate a visual that would represent both a time spectrum and the technologies’ different “identities” simultaneously. After some brainstorming, and a brief recollection of a similar historical parody one of us saw on the internet some months before (you can see it here:, we decided that creating a spoof Facebook wall (including a series of posts and discussions) could provide us with the ability to visualize both the advent and individuality of each of the ten machine scoring technologies we chose to represent.

In an effort to create believable Facebook posts, we incorporated Facebook vernacular, emoticons, carefully-selected profile pictures, hashtags (because for some reason Facebook has integrated them…), et cetera into our project. We also tried to isolate the core concept or ability that individualized each technology so that we could use it in a post to define that particular system as a kind of Facebook “personality.” Sometimes we put the systems into conversations with each other in the comments box to help elucidate a particular system’s function. (and sometimes we did this just for fun). Also, instead of “likes.” we made it so that that section of the post relayed validity percentages.

Additionally, to represent the relationships between software (“better,” “worse,” “more/less effective,” etc.) we found traditional means of representation (proximity, size, color) lacking, since these measures do not immediately visually indicate the benefits of each piece of software. As a result of choosing the Facebook theme for our work, we were able to use a somewhat discrete method to signify “performance” by using Facebook’s built-in, already somewhat vague “like” system. By equating Likes to Performance, we were able to visually represent an otherwise abstract concept for which traditional visual arrangement seemed lacking. In this way, like the measures of performance and validity themselves, the numbers are presented and left for the reader to determine the value of beyond the simple numbers presented here.

Bibliography and Works Cited

Burstein, Jill, Claudia Leacock, and Richard Swartz. “Automated Evaluation of Essays and Short Answers .” Loughborough University (2001): 1-13.

Hearst, Mari A. “The Debate on Automated Essay Grading.” IEEE Intelligent Systems & Their Applications 15.5 (2000): 22.

Herrington, Anne, and Charles Moran. “What Happens when Machines Read our Students’ Writing?” College English 63.4 (2001): 480.

Jordan, Sally. “E-Assessment: Past, Present and Future  .” The Open University: Pedagogic Directions: 1-20.

Leacock, Claudia, and Martin Chodorow. “C-Rater: Automated Scoring of Short-Answer Questions.” Computers & the Humanities 37.4 (2003): 389-405.

Mitchell, Tom, et al. “Towards Robust Computerised Marking of Free-Text Responses.” Loughborough University (2002): 233-249.

Valenti, Salvatore, Francesca Neri, and Alessandro Cuchiarelli. “An Overview of Current Research on Automated Essay Grading.” Journal of Information Technology Education 2 (2003).

Werner, Gergory J. “A Complete Approach to Automated Essay Grading .” George Washington University: Department of Computer Science: 1-6.

Zhang, Mo. “Contrasting Automated and Human Scoring of Essays.” Educational Testing Service: R & D Connections 21 (2013).


Posted by on September 26, 2013 in Uncategorized


8 responses to “Automated Essay Scoring Software Timeline

  1. DB

    September 26, 2013 at 3:48 pm

    I imagine I’m not alone in thinking that the Facebook spoof is clever. What attitudes toward technology do you think you’re revealing in using that particular genre? I wonder if you could incorporate something about the advent of Facebook itself (around 2005): I wonder if it could count as a kind of machine-scoring technology. After all, it’s ranking our writing and comparing it to databases in order to classify us. Maybe I’m reaching.

  2. brucebowlesjr

    September 26, 2013 at 3:50 pm

    Jason and Kendall,

    I really dug the social media slant and the slang in this timeline. It had me cracking up at time internally (no lol as to not be creepy).

    Could you possibly project the future on this timeline? For example, where will automated essay scoring software be in 15-20 years?

    Some more history about each could have been helpful.

  3. jeskew2013

    September 26, 2013 at 3:54 pm

    Very clever presentation of the material. I especially liked presenting likes in terms of percentages. The one thing I noticed was that you focused a lot on specific testing softwares and machine-grading methods and left out some of the people involved.

  4. amypiotrowski

    September 26, 2013 at 3:54 pm

    First of all, hilarious! I like how you’re giving each program its own voice in which to explain what it does. Sounds like some of these automated scoring programs have taken different approaches to machine essay grading. It’s a little hard to determine how different some of these programs are. Are there things that automated scoring programs all try to do in their design or algorithms?

  5. andrewdavidburgess

    September 26, 2013 at 3:56 pm

    This is great, guys. Even while poking fun at social media, you manage to demonstrate the relationship between these various electronic scoring systems and to point out points of comparison and analysis between them.

    Question: Where is this going? Is there some evolution that you see in this timeline? Can you guess what the future of electronic scoring might look like?

    Suggestion: I’d like to see some naysayers inserted here to counterbalance the machines. Maybe some of the folks we’ve read this semester to challenge the claims of accuracy and reliability. Could be fun.

  6. sarahm1320

    September 26, 2013 at 4:01 pm

    Very fun idea – I especially liked how you put the different machines into dialogue with one another (maybe I’m reading into this too much, but it seemed like a gesture towards the economic competition that many of these scoring machines arose from). I would have liked to know more about why these machines were created – did their creators have a specific market in mind? It’s tough because you don’t want to add too much to keep the posts appropriate to the genre/style of the piece, but I would have liked to know more information about each of the systems, and if/how they’re currently being used.

  7. jacobwcraig

    September 26, 2013 at 4:38 pm

    Comment: I second the comment about your decisions about the presentation of the material.
    Question: Like Andrew, I wonder what kind of connective tissue you could develop in addition to the kind of dialogic relationships that you set up between the devices. I’m really interested in the idea that Sarah brings up.
    Suggestion: If you wanted to publish this (and I think that you could), I think that creating hyperlinks/profiles would be interesting. A basic html image map in Fireworks or through one of the online generators would get you the link-to and link-from capability.

  8. jeffnaftzinger

    October 9, 2013 at 6:55 pm

    Comment: I’m blown away by the creativity you guys exhibited in this “timeline.” Great job playing with requirements of the assignment.
    Question: What was the reason for making this more of a humorous timeline? To reach different audiences?
    Suggestion: I agree with Jacob about hyperlinking profiles would be interesting, that way you could give the viewers a little more background into the different assessment machines.


Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: