grading your student essay -- a computer

Now grading your student essay -- a computer

University of Missouri professor Ed Brent encourages his students to use the SAGrader to give them a better shot of earning a better grade.

RELATED
SAGrader software

National Science Foundation (NSF)

University of Missouri-Columbia

Higher Education


COLUMBIA, Missouri (AP) -- Student essays always seem to be riddled with the same sorts of flaws. So sociology professor Ed Brent decided to hand the work over to a computer.

Students in Brent's Introduction to Sociology course at the University of Missouri-Columbia now submit drafts through the SAGrader software he designed. It counts the number of points he wanted his students to include and analyzes how well concepts are explained.

And within seconds, students have a score.

It used to be the students who looked for shortcuts, shopping for papers online or pilfering parts of an assignment with a simple Google search. Now, teachers and professors are realizing that they, too, can tap technology for a facet of academia long reserved for a teacher alone with a red pen.

Software now scores everything from routine assignments in high school English classes to an essay on the GMAT, the standardized test for business school admission. (The essay section just added to the Scholastic Aptitude Test or SAT for the college-bound is graded by humans).

Though Brent and his two teaching assistants still handle final papers -- and grades -- students are encouraged to use SAGrader for a better shot at an "A."

"I don't think we want to replace humans," Brent said. "But we want to do the fun stuff, the challenging stuff. And the computer can do the tedious but necessary stuff."

Developed with National Science Foundation funding, SAGrader is so far used only in Brent's classroom. Like other essay-grading software, it analyzes sentences and paragraphs, looking for keywords as well as the relationship between terms.

Other programs compare a student's paper with a database of already-scored papers, seeking to assign it a score based on what other similar-quality assignments have received.

Educational Testing Service sells Criterion, which includes the "e-Rater" used to score GMAT essays. Vantage Learning has IntelliMetric, Maplesoft sells Maple T.A., and numerous other programs are used on a smaller scale.

Most companies are private and offer no sales figures, but educators say use of such technology is growing.

Consider the reach of e-Rater: 400,000 GMAT test-takers annually, a half-million U.S. K-12 students and 46 international schools and districts. ETS says an additional 2,000 teachers begin using its technology each month.

But it's tough to tout a product that tinkers with something many educators believe only a human can do.

"That's the biggest obstacle for this technology," said Frank Catalano, a senior vice president for Pearson Assessments and Testing, whose Intelligent Essay Assessor is used in middle schools and the military alike. "It's not its accuracy. It's not its suitability. It's the believability that it can do the things it already can do."

South Dakota is one of several states that has tested essay-grading software. Officials there decided against using it widely, saying feedback was negative.

Not all districts had the same experience. Watertown, South Dakota, students are among those who now have their writing-assessment tests scored by computer.

Lesli Hanson, an assistant superintendent in Watertown, said students like taking the test by computer and teachers are relieved to end an annual ritual that kept two dozen people holed up for three days to score 1,500 tests.

"It almost got to be torture," she said.

Some 80 percent of Indiana's 60,000 11th-graders have their English assessment scored by computer, and another 10,000 ninth-graders are taking part in a trial in which computers assess some routine written assignments.

Stan Jones, Indiana's commissioner of higher education, said the technology isn't as good as a teacher but cuts turnaround time, trims costs and allows overworked teachers to give written assignments without fearing the workload.

"This (allows) them to require more essays, more writing, and have it graded very painlessly," Jones said.

Software can also remove a degree of subjectivity.

"It's fairly consistent. Different teachers grade different papers differently." -- Keith Kelly, 21, of Cleveland, one of Brent's sociology students.

The software is not flawless, even its most ardent supporters admit.

When the University of California at Davis tried out such technology a couple years back, lecturer Andy Jones decided to try to trick e-Rater.

Prompted to write on workplace injuries, Jones instead input a letter of recommendation, substituting "risk of personal injury" for the student's name.

"My thinking was, 'This is ridiculous, I'm sure it will get a zero,"' he said.

He got a five out of six.

A second time around, Jones scattered "chimpanzee" throughout the essay, guessing unusual words would yield him a higher score.

He got a six.

In Brent's class, sophomore Brady Didion submitted drafts of his papers numerous times to ensure his final version included everything the computer wanted.

"What you're learning, really, is how to cheat the program," he said.

Work to automate analysis of the written word dates back to the 1950s, when such technology was used largely to adjust the grade level of textbooks, said Henry Lieberman, a research scientist at the Massachusetts Institute of Technology.

Before long, researchers aimed to use such applications to evaluate student writing.

SAGrader, like other programs, needs significant prep work by teachers. For each of the four papers Brent assigns during his semester-long course, he must essentially enter all the components he wants an assignment to include and take into account the hundreds of ways a student might say them.

Part of one assignment for Brent's class was for students to pick a crime and explain how it fit into sociologists' categories. Brent had to key in dozens of words in order to ensure all types of transgressions would be identified.

What a writer gets back is quite detailed.

A criminology paper resulted in a nuance evaluation offering feedback such as this: "This paper does not do a good job of relating white-collar crime to various concepts in labeling theory of deviance."

Brent -- who earned a postdoctoral degree in artificial intelligence and is also an adjunct professor in the computer science department -- said the software may have limitations, but allows teachers to do things they weren't able to do before.

Before Brent wrote SAGrader, a part of his broader data-analysis program Qualrus, he only gave students multiple-choice tests.

"Now we can focus more," he said. "Are they making a good argument? Do they seem to understand? Are they being creative?"


source: cnn.com