Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How Good Are Robo-Graders?

samzenpus posted more than 2 years ago | from the reading-robots dept.

Education 157

stoolpigeon writes "With a large study showing software grades essays as well as humans, but much faster, it might seem that soon humans will be completely out of the loop when it comes to evaluating standardized tests. But Les Perelman, a writing teacher at MIT, has shown the limits of algorithms used for grading with an essay that got a top score from an automated system but contained no relevant information and many inaccuracies. Mr. Perelman outlined his approach for the NY Times after he was given a month to analyze E-Rater, one of the software packages that grades essays."

Sorry! There are no comments related to the filter you selected.

More importantly (5, Insightful)

crazyjj (2598719) | more than 2 years ago | (#39770563)

How quickly will students learn to game the system to get perfect scores with perfect gibberish?

Re:More importantly (1)

LostCluster (625375) | more than 2 years ago | (#39770609)

How quickly will teachers become completely automated? That's a bit of a scary concept. You can't just have "teachers" who do nothing but press "Play" on a video machine.

Re:More importantly (2)

alen (225700) | more than 2 years ago | (#39770681)

yes you can

most of the skill of a good teacher is know child psychology and how to handle kids with different issues and different stages of development

memorizing a few facts is fairly easy

Re:More importantly (5, Insightful)

masternerdguy (2468142) | more than 2 years ago | (#39771047)

No. This education degree stuff is crap. A teacher should have at least a masters degree in the topic they intend to teach.

Re:More importantly (1)

Anonymous Coward | more than 2 years ago | (#39771165)

No. This education degree stuff is crap. A teacher should have at least a masters degree in the topic they intend to teach.

And they should beat it into the children!

Re:More importantly (0)

Anonymous Coward | more than 2 years ago | (#39771345)

No. This education degree stuff is crap. A teacher should have at least a masters degree in the topic they intend to teach.

And they should beat it into the children!

with various methods of abuse/attack!

Re:More importantly (4, Insightful)

anyGould (1295481) | more than 2 years ago | (#39771529)

No. This education degree stuff is crap. A teacher should have at least a masters degree in the topic they intend to teach.

Problem 1: Teachers don't get to choose what classes they get - I knew an English teacher who ended up teaching Intro Computing because.. they needed a computing teacher and he was available. Especially for newer teachers - you teach what they tell you to teach.

Problem 2: Are you intending to pay all those teachers in accordance with the extra 2+ years of education you're requiring?

Problem 3: At lower levels, you have A Teacher, not A Math Teacher and An English Teacher. Do you expect your kid's grade 1 teacher to hold multiple degrees? (And see problem #2, expanded to pay for a teacher holding half a dozen post-grad degrees so you feel comfortable letting them teach your kid ABCs.)

Re:More importantly (2)

sourcerror (1718066) | more than 2 years ago | (#39771571)

Even in Europe they only require this from highschool teachers.

Re:More importantly (1)

Anonymous Coward | more than 2 years ago | (#39772741)

Nobody wants to pay teachers enough to make it worth having a Master's degree. In many districts (not all) it's not really worth having a bachelor's degree.

Re:More importantly (1)

oh_my_080980980 (773867) | more than 2 years ago | (#39771065)

"memorizing a few facts is fairly easy"

If you think that's learning, you are sadly mistaken.

Re:More importantly (0)

Anonymous Coward | more than 2 years ago | (#39772733)

It isn't learning, but it *is* according to the public education system in the US.

Re:More importantly (3, Insightful)

TheRaven64 (641858) | more than 2 years ago | (#39770915)

The tube drivers in London were recently on strike over pay. Their salaries are around £40k (about $65K), but for a decade or so most of the train control has been completely automated: they're just there to press the emergency stop button if there is something wrong with the automated system (which a human will notice but another automated system won't and, for example, cut power to that segment of track). So, judging by the past, teachers that did nothing but press play on a video machine would be better paid than ones that actually taught...

Re:More importantly (1)

alltradeschools (2623811) | more than 2 years ago | (#39771593)

It will never happen, but yes it is a very scary concept. Another scary thought is that people are so dependent on software and computers they would not know what to do if for whatever reason there was an extended power outage or worse. Can you image if cashiers had to figure out your change without the register telling them how much money to give back to the customer (when they use cash that is).

Re:More importantly (1)

Half-pint HAL (718102) | more than 2 years ago | (#39771839)

Another scary thought is that people are so dependent on software and computers they would not know what to do if for whatever reason there was an extended power outage or worse.

...which was precisely the problem at Fukushima. The emergency cooling system was electric, and the tsunami knocked out the emergency backup generator that powered it. There was a manual override, but everyone had forgotten that they needed to go and crank a few handles because as far as they were concerned, the system was "automatic". Untold millions of pound/euros/dollars/yen worth of material damage, hundreds of people put in direct physical danger from radiation, extensive contamination of the environs. Because it's "automatic".

Re:More importantly (1)

D'Sphitz (699604) | more than 2 years ago | (#39771889)

Can you image if cashiers had to figure out your change without the register telling them how much money to give back to the customer

So it's hard for you to imagine someone using basic math? Did you know they do still teach math in school? Most people surely can handle basic subtraction by 3rd grade, including cashiers.

Re:More importantly (0)

Anonymous Coward | more than 2 years ago | (#39772249)

You've obviously never had to deal with the vast bulk of the American Public.

It's not that it isn't taught but that it isn't learned. Remembering it long enough to pass next week's test isn't learning. Far too many students view school as something to be suffered.

Re:More importantly (1)

jc42 (318812) | more than 2 years ago | (#39772517)

It's not that it isn't taught but that it isn't learned.

Indeed. I've recently seen people using calculators to add 10 to 5. I'd bet that all of them "passed" their school math classes.

I've also seen a number of cases of people presented with a grid of numbers, needing to know how many items were in the grid, and laboriously counting them one by one. They probably "passed" tests on multiplying, but have no concept of why they were taught multiplication or how it might actually be used. It's just a mysterious rite of passage that they're required to perform for no apparent reason.

But this isn't really anything very new. You can find complaints about such ignorance from before everyone had cheap, portable calculators or computers.

Re:More importantly (1)

NeverSuchBefore (2613927) | more than 2 years ago | (#39772765)

Far too many students view school as something to be suffered.

And with the way it's done now, why is that such a surprise? It actually emphasizes teaching to the test and rote memorization. If you make school boring and unbearable, very few people will find it anything but.

Re:More importantly (3, Informative)

s0nicfreak (615390) | more than 2 years ago | (#39772303)

Did you know that often schools only teach students what is required to pass the tests, and much of that is forgotten during school vacations, not to mention after several years of being out of school?

Just the day before yesterday I was behind someone in a checkout line that didn't have enough to pay their bill on their debit card. So the cashier and the lady were trying to work out how much would be remaining after the amount on the debit card was used. After several minutes of both of then failing to figure it out, and the customer just handing the cashier some money (though not enough to cover the whole bill) they called over a manager, who showed the cashier that if she charged the debit card first it would show her the remaining amount. So then they counted how much money the customer had handed the cashier... and both tried to work out how much more was needed. After a minute the manager figured out how to type the amount into the register and be told the remaining bill.

I'm not saying cashiers don't know basic math, but quite a few of them would not be able to do their job without a register or at least a calculator.

Re:More importantly (2)

EmperorOfCanada (1332175) | more than 2 years ago | (#39772487)

Nope. I would be willing to say every cashier that I have ever seen manually do math has failed. If I pull a stunt like handing them a 20 and then a dime for something that cost 19.01 they are often lost calculating the 1.09 change if they had entered 20 into the till. Another store's till broke and the cashier was nearly in tears trying to work out tax with a calculator, and this was a single item sort of store.She was taking say a 25 dollar purchase and applying 15% tax and coming up with a total purchase price of 8 dollars. Car salesman take advantage of this every day. They will sell you a car and tell you that it is one price and you are getting it at a certain interest rate and your monthly payments will be another price. But if you do the math it will usually turn out you are paying a grand or so more. They know that 99% of people can't work out loan payments.

I don't know how exactly the schools are failing but almost regardless of the level of grade school math education people are usually unable to apply math to real life. Tell them that half the population is below any average and they will tell you that you are below average. Show them that the fees in mutual funds work against the whole idea of compound interest and they stare at you like you are speaking Greek.

Re:More importantly (1)

Squeeonline (1323439) | more than 2 years ago | (#39772005)

How quickly will teachers become completely automated? That's a bit of a scary concept. You can't just have "teachers" who do nothing but press "Play" on a video machine.

That's what most of my lectures in college were like.

Re:More importantly (1)

NJRoadfan (1254248) | more than 2 years ago | (#39772549)

I had teachers in both middle school and high school who couldn't figure out how to press play on a VCR. This was 15 years ago.

Re:More importantly (1)

jellomizer (103300) | more than 2 years ago | (#39772595)

Actually if you can get a lot of that boring grading out of the way, you can free up teachers time to focus on the Human Element.
While some kids can probably learn better without teachers, others will need them to help guide their education, to spot when they have problems and not learning something to stop and help them get caught up.

Re:More importantly (1)

s0nicfreak (615390) | more than 2 years ago | (#39772693)

But you could have students that press play themselves while a parent guides them. And instead of DVDs, you could put the videos on the internet so that they are more easily accessible. We have the ability to spread knowledge around the world, yet we still hold on to the archaic idea that knowledge must be hoarded and given only to people that sit in certain classrooms...

Re:More importantly (3, Insightful)

sglewis100 (916818) | more than 2 years ago | (#39770615)

How quickly will students learn to game the system to get perfect scores with perfect gibberish?

Spammers with poor spelling and grammar figured out combinations of gibberish to get around Bayesian spam filtering, I can only imagine relatively smart students will figure out ways to beat the software in time. But hopefully, if people implement systems like this, there will be some checks and balances. Fear of receiving a '0' for a test coupled with having essays randomly graded (smaller numbers) and reviewed / skimmed quickly (larger numbers) ought to be a good start.

Re:More importantly (1)

anyGould (1295481) | more than 2 years ago | (#39771673)

Here's the fun part - it would entirely depend on how the teacher sells this at the beginning of the year. I could see an argument for unfairness if they're picking out kids for "manual grading" - especially when the difference in marks will be vast (the robo's "this meets all my criteria - A+" vs. the teachers "you spewed out random crap for 500 words - F"). How many teachers (and schools) are going to want to walk into that quagmire?

Putting on "angry parent hat", the argument would go roughly - Why does my kid get marked differently than the other kids in class? Oh, you suspected my kid is gaming the system - how many other kids have you checked? None - oh, so you don't have a problem letting your time-saver software hand out A's unchecked to the other kids in class?

(And to forestall the obvious - no, my kid would *not* be in the room for that - I firmly believe students should respect their teachers (at least publicly). And depending on the kid's age, I'm torn between "do it right!" and "good job working the system - now rewrite it properly for your mother and I". But honestly - if schools are teaching to the test, they can't complain when the kids do it better than expected.)

Re:More importantly (3, Insightful)

Half-pint HAL (718102) | more than 2 years ago | (#39771875)

I don't see why this would be different from current auditing practices. If an external examiner finds that your students have been incorrectly marked, it's either an automatic scaling of grades for everyone, or back to the red pen and regrade everything.

Re:More importantly (5, Funny)

Anonymous Coward | more than 2 years ago | (#39770633)

How quickly will students learn to game the system to get perfect scores with perfect gibberish?

Noooooooooo.

I had to deal with a Robo grader once during an exam. Time was up and I was still writing. Several large automatic weapons appeared and in a robotic voice it said, "Drop your pen!"

I did immediately and it said, "Thank you for your cooperation."

Or that might have been when I was taking an art class taught by Peter Weller [ew.com] .... I don't remember now.

Re:More importantly (3, Funny)

gnick (1211984) | more than 2 years ago | (#39770953)

Several large automatic weapons appeared and in a robotic voice it said, "Drop your pen!"

I did immediately and it said, "Thank you for your cooperation."

You were lucky. You should see what happened to the guy in this documentary [wikipedia.org] when the robo-grader didn't hear the pen hit the floor.

100% A+ Perfect Reply (4, Insightful)

MyLongNickName (822545) | more than 2 years ago | (#39770655)

After thorough consideration of this first post and its contents, I find this I must respond in the most considerate and throughtful way possible. This first post was clearly written before the second post and well in advance of this reply. Based on this, it is only logical to assume that this first post was written before any other posts. This leads me to think that crazyjj was quicker reflexes and reading skills than his compatriots.

My research has shown that people with quick reflexes make 80% more in real dollar terms than others[1] and are more likely to lead a longer life than their slower reading friends [2]. Clearly crazyjj is at an extreme advantage compared to the rest of slashdot.

Can America survive with this type of inequality? I think not. We must institue some type of equalizer. Perhaps crazyjj should be given a keyboard with several broken keys. Or perhaps we should simply bash his fingers a few times. In the words of Abraham Lincoln, "A man who types too fast can't be trusted."[3] Abraham Lincoln saw the danger that crazyjj represents and warned us. Will we listen?

Re:100% A+ Perfect Reply (0)

Anonymous Coward | more than 2 years ago | (#39770819)

I don't think you would get a very high grade with the obvious typos and incorrect wording in some of your sentences.

Re:More importantly (3, Insightful)

BravoZuluM (232200) | more than 2 years ago | (#39770731)

What does it mean to game the system? The game paper, while not pertaining to the subject, is a well written paper. It is not gibberish. It would take some talent to produce the gamed paper and probably more time. Given that, why wouldn't the student just write an on topic paper?

Given the bigger picture, writing is an art form. An essay is an art form. Even a human grading the paper might miss the nuances of what is being written. Who can truly say what the author has written is incorrect, when in writing, there is no incorrect or correct. There is just a continuum from bad to good writing.

Re:More importantly (0)

Anonymous Coward | more than 2 years ago | (#39770959)

Exactly. A computer cannot (yet) possibly parse, understand, and grade the essay. A human probably will not, but there is zero illusion to the student that someone is actually going to read their paper.

Re:More importantly (3, Insightful)

bhlowe (1803290) | more than 2 years ago | (#39772203)

A student can game the system by writing their paper, running it through one or more "grading" systems... and making changes until it comes out an "A". Obviously, you would want to do this in a way that it does this while retaining the content and expected "readability" desired.

The fact is most "jobs" that humans do will be able to be done by a robot or computer. I can easily envision a future where kids get the best personalized teaching experience from a computer "coach"... who can tailor each kid's lesson much more skillfully than the average teacher trying to teach to 120 kids of a multitude of abilities. Teacher will be left to enforce discipline, dry tears, lead group exercises (as determined by the computer) and smile and wave at the kids as they come and go.

Re:More importantly (5, Insightful)

NReitzel (77941) | more than 2 years ago | (#39770991)

Well, yes.

E-Rater (a product with which I have some familiarity) is specifically sold to improve form and grammer, and the product explicitly states that it does not grade content.

So, what you are saying is that the students will figure out how to write with excellent grammar and form, in order to get good grades.

Well, yeah.

That's the whole point. That, and the fact that you can have a student write a short essay in 30 minutes, and give them immediate feedback on what they have done wrong, as far as sentence form and grammar are concerned.

Generally, a student may know what they want to say, and have difficulty putting it into English prose in a way that might convince the reader that they have a clue about that of which they speak.

Don't think it matters? What kind of result do you think Mr. Churchill might have received if he had stated, "Them Nazis is bad, we gots to beat em."

Mr. Perelman spent a month of effort carefully crafting an essay that said nothing, eloquently. If our students can do that, more power to them.

Re:More importantly (0)

Anonymous Coward | more than 2 years ago | (#39771101)

What kind of result do you think Mr. Churchill might have received if he had stated, "Them Nazis is bad, we gots to beat em."

Hells yeah! Me and my cousin Larry, we's made this cannon that shoots flamin taters. My brother Larry and I, we's gonna go and blast dem Nazzys to kingdom come!

Re:More importantly (3, Insightful)

anyGould (1295481) | more than 2 years ago | (#39771775)

So, what you are saying is that the students will figure out how to write with excellent grammar and form, in order to get good grades.

I think that's naive. I think one kid will figure out how to get the computer to kick out excellent grammar and form (a lot easier when you don't actually care about the content), and in short order most of the smart/cunning kids will be using that (the cunning ones because it's a cheap A; the smart ones because they'll want to concentrate on subjects where knowledge matters, as opposed to something that can be outsourced to small shell scripts).

Re:More importantly (3, Funny)

jc42 (318812) | more than 2 years ago | (#39772647)

What kind of result do you think Mr. Churchill might have received if he had stated, "Them Nazis is bad, we gots to beat em."

Here in the US, we'd just elect him president.

Re:More importantly (0)

Anonymous Coward | more than 2 years ago | (#39771289)

One student will learn to game the system and sell the results to his classmates. Or, perhaps, the MIT grad students mentioned in the article will release their Android/iPhone app that generates 6-scoring papers and the students will forego the learning experience entirely.

Re:More importantly (1)

anyGould (1295481) | more than 2 years ago | (#39771459)

How quickly will students learn to game the system to get perfect scores with perfect gibberish?

Considering TFA already notes that people can (or have) designed Android apps that would automatically generate essays designed to pass the robo-maker?

If they're using this system at a school, I would be astounded if there *wasn't* an automated system (or twelve) already.

Re:More importantly (1)

Endo13 (1000782) | more than 2 years ago | (#39771503)

I think the better question is how quickly will someone learn to game the system, and come up with a program to generate unique "top quality" essays.

Sorry, human intervention required (4, Insightful)

LostCluster (625375) | more than 2 years ago | (#39770573)

I don't think auto-graders are a good idea. Where is the information exchange between student and teachers? Teachers need to read student essays not just to assign the grade, but to exchange knowledge with their students Opinions and comments should be two-sided exchanges, if students are writing things that aren't going to be read, how does that work?

Re:Sorry, human intervention required (4, Insightful)

Dyinobal (1427207) | more than 2 years ago | (#39770611)

Yep any essay should come back with feed back written on it in the margins/space between lines. Plus I doubt auto graders will mean anything except for kids learning to write a specific way that the auto grader is programmed to grade well.

Re:Sorry, human intervention required (2)

dkleinsc (563838) | more than 2 years ago | (#39770709)

At the same time, I've seen significant flaws in the grading practices of human graders. For instance, I distinctly remember the paper I got back in my college years that said something along the lines of "Really interesting, well written, and insightful. B-". I also remember some essays that were pure unadulterated nonsense that got very high grades (including a 4-week project that I started on during school the day it was due and received an A).

Re:Sorry, human intervention required (0)

oh_my_080980980 (773867) | more than 2 years ago | (#39771103)

Because you got a bad grade there's a flaw with human graders? Somebody wants cheese with that wine.

Re:Sorry, human intervention required (1)

Zordak (123132) | more than 2 years ago | (#39772453)

I think the flaw was completely positive feedback coupled with a poor grade. The poor grade should have been accompanied by some useful suggestions for improvement.

Re:Sorry, human intervention required (2)

dkleinsc (563838) | more than 2 years ago | (#39772645)

Zordak explained it: If my paper had sucked, I would have been fine with a bad grade and ideally some information on why it sucked so I could do better the next time. But instead what I got was "good work, I'm still giving you a bad grade for reasons I won't explain to you".

Re:Sorry, human intervention required (3, Funny)

Zordak (123132) | more than 2 years ago | (#39772333)

When I was in high school, we read A Portrait of the Artist as a Young Man. This is literally the worst alleged novel I have ever read. I actively despised it with my entire soul. So I skipped huge chunks of it wherever I figured I could get away with doing so and still pick up the threads of the mostly nonexistent plot.

When we (finally) finished the thing, we had to write a series of short essays responsive to several prompts. One of the prompts told us to describe the symbolism and significance of the "rose."

Having skipped huge portions of the book, I had never encountered this purported rose. And I certainly wasn't going to go back and pick through the dense, sophomoric prose to find it. Instead, I figured I could probably pick up some partial credit by saying some random insightful-sounding thing. So I started spewing what English teachers love. I used words like "juxtaposition" and "antithesis" and compared the rose to some other random symbolic object in the book. It was pure, unadulterated, Grade A, premium All-American BS.

I got an A on the paper. The teacher was particularly profuse in her praise of my short essay on the "rose," commented that I had captured the symbolism of the "rose" perfectly. I couldn't have agreed more.

Re:Sorry, human intervention required (1)

NJRoadfan (1254248) | more than 2 years ago | (#39772641)

Whats the old saying? "If you can't dazzle them with your brilliance, baffle them with your BS." Worked for me through high school and college on those silly assignments. Make some crazy theory up, make sure the paper meets the checklist of requirements for the writing assignment and you get a good grade... even if they can't figure out what the heck you were talking about.

Re:Sorry, human intervention required (0)

Anonymous Coward | more than 2 years ago | (#39770841)

Isn't this a good thing? Teaching kids how to write the Executive Summary, since no pointy-haired boss really reads the entire project proposal

Re:Sorry, human intervention required (1)

fermion (181285) | more than 2 years ago | (#39771469)

These are graders for standardized tests in the middle and high school grades. There is still plenty of time to interact with students. Automated graders, however, provide significant benefit. First the provide the first level of grading to insure that the students is doing what is necessary to pass the end of course or end of semester test. If a computer is going to grade the work, then the student needs to write for the computer. Likewise, the computer grading papers helps insure the teaching is teaching and students are learning the skills mandated by the state. This helps the student as the student is less likely to arrive at a test after being taught skills not tested by a creative teacher. It also provides feedback on progress

Presumably during the year the teacher is reading for content, and not just stucture. If a computer is grading the teacher can spend more time on content, thus helping the student get into the habit of critical thinking that will lead to more factual essays. But how much time needs to be spent of facts for an language essa? My impression is that the essay will either have a short prompt, where the student will state feelings or impressions or opinions of the prompt, or will read a story, and then pull quotes from that story to support and opinion on the story. Neither of these are particularly connected to reality.

Re:Sorry, human intervention required (1)

kaiser423 (828989) | more than 2 years ago | (#39771515)

This.

I have to write some manuals, etc sometimes that are very likely to never be read by humans, and just scanned by a computer for relevant info. It still has to be in human readable format in case the scan fails, or a human wants more info. But in general, I find it incredibly de-motivating to write good sentences or go back and polish my work when I know that it's likely to never be read.

And this is for my job where they pay me money. I can't imagine how de-motivating it would be to students.

This is my big problem with education currently. When I went through school, it was very, very easy to tell what was important to the adults (and as they shoved more portables and temp teachers into the school, it became very obvious that it wasn't education). As such, I paid less and less attention and cared less and less. More standardized testing was the same thing; if all that mattered to the admins was passing that silly test, then why care about anything but that? As a result, more goofing around in class. Going to where the teacher doesn't even freakin' look at your papers?!? My god, how ridiculously demotivating to students.

Re:Sorry, human intervention required (0)

Anonymous Coward | more than 2 years ago | (#39772015)

Opinions and comments should be two-sided exchanges, if students are writing things that aren't going to be read, how does that work?

I had an online programming course, a couple years ago. After writing several insane programs that worked like they were supposed to, but were coded to look like they shouldn't, I determined none of the code I wrote was actually being looked at. The teacher would just run the program, if it worked it passed with no further examination. I never once received feedback from the teacher, but it was honestly just a minor disappointment.

Worse yet, in an English Composition class, I was totally ignored in person. And that was much more discouraging, especially since it was the most in-depth class I've had to this day. That was the class where I was introduced to locial fallacies, and more.

Some teachers spend all their time dealing with the biggest problems, and miss the opportunity to teach those who aren't.

Anyhow, I think classes would benefit from using both robo and human graders together.

The benefit of the robo-grader is a lack of insight, and it's speed. The person that grades it can benefit from seeing what the robo-grader would give as an unbiased second opinion. The other beneficial thing about a robo-grader is it doesn't remember the previous paper it graded, so it doesn't compare the extravagant papers against plain ones. The paper that pads the ego of the teacher correctly, has an unfair advantage that the robo-grader pointedly ignores. The nonsense papers the robo-grader likes, a person can spot and quickly weed out.

Technology in general is the average of all tools.

but how well does it work in the real world (2)

LetterRip (30937) | more than 2 years ago | (#39770617)

While it is true that you can engineer essays to be 'bad' and still score 'good' - the question is - are there natural essays that score good but are actually bad; and good essays that score bad but are actually good.

Every analysis I've seen suggests that these algorithms do have problems with good essays that are highly creative. Essay graders also have difficulties with this kind of essay - giving drastically varied scores.

However there doesn't seem to be much evidence of other issues except when an extremely knowledgable issue deliberately trys to make the algorithm fail. Any student or other individual who can do this probably knows that material well enough to 'get an A' if they were to properly apply what they know so this seems like a non issue.

Re:but how well does it work in the real world (1)

Hentes (2461350) | more than 2 years ago | (#39770901)

What's the difference between "natural" and "engineered" essays? If you use algorithmic grading the students will find out, and then the engineered essays will appear.

While it's true that an average grad student grader can have problems with a creative essay, they can just forward those few cases to the teacher. This behavior, of course, could also be implemented in software, but currently it isn't.

Re:but how well does it work in the real world (1)

kjoyce (123612) | more than 2 years ago | (#39771253)

Did you read the essay he posted? It doesn't look like you need to understand the material at all to get the top mark.

I don't care; standardized tests are corrupt (1)

Anonymous Coward | more than 2 years ago | (#39770643)

Read "Making the grades" by Todd Farley. Robotic graders just make the tests even more farcical.

Re:I don't care; standardized tests are corrupt (0)

Anonymous Coward | more than 2 years ago | (#39770753)

The difference between teaching the test and not doing standardized testing is that now we teach the test, instead of nothing at all. If the students game the robo-grader, they've learned *something*. Standardized testing is a bad answer to a problem that's so bad that every other approach we've tried has failed. The real solution is to make parents care. However, punishment is highly unlikely to work, and we really, really shouldn't have the government trying any other approaches (propoganda is bad, government propaganda is worse).

Give me a better solution. I reject your "more money" approach; it's been demosntrated over the last 50 years to be a national scale disaster.

Re:I don't care; standardized tests are corrupt (2)

jimbolauski (882977) | more than 2 years ago | (#39771137)

The difference between teaching the test and not doing standardized testing is that now we teach the test, instead of nothing at all. If the students game the robo-grader, they've learned *something*. Standardized testing is a bad answer to a problem that's so bad that every other approach we've tried has failed. The real solution is to make parents care. However, punishment is highly unlikely to work, and we really, really shouldn't have the government trying any other approaches (propoganda is bad, government propaganda is worse).

Give me a better solution. I reject your "more money" approach; it's been demosntrated over the last 50 years to be a national scale disaster.

The solution is simple remove the kids that don't care, it seems harsh but they are the reason classrooms get stuck in a quagmire. Offer an education to everyone but do not force it on people that don't want it and will waste people's time that do want it. The the true secret of private schools is that everyone there has parents that value education and for the most part they do too. Once disruptive and unmotivated students are removed from the class the teachers can be held accountable for their classrooms, and are typically motivated because the students genuinely care.

Re:I don't care; standardized tests are corrupt (0)

Anonymous Coward | more than 2 years ago | (#39771297)

That's a great answer for high school, but how about throwing out my 2nd grader's partner. Ed is a nice kid, throws things in class, and spends half the day with special ed teachers. For Ed, it's great that he's being mainlined. For the kids who should be learning instead of dodging pencils, Ed's a menace. For the teacher, Ed is a half a day wasted.

It's very challenging to create a useable distinction between throw away people and only teaching the ones who want to learn. I suspect that if we removed the financial incentive to stay in high school (ADC, whatnot) that it would improve, but at what cost to society?

well we need more tech / vocational schools! (0)

Anonymous Coward | more than 2 years ago | (#39771667)

well we need more tech / vocational schools!

also more jobs that don't need BA for jobs that used to not need it.

Human vs. Software (4, Insightful)

Anti_Climax (447121) | more than 2 years ago | (#39770657)

But Les Perelman, a writing teacher at MIT, has shown the limits of algorithms used for grading with an essay that got a top score from an automated system but contained no relevant information and many inaccuracies.

Considering the fake generated paper [googleusercontent.com] that was published in a peer reviewed journal, I'd say that means the robo-graders are on par with human proof readers.

Re:Human vs. Software (1)

sribe (304414) | more than 2 years ago | (#39770817)

... I'd say that means the robo-graders are on par with human proof readers.

That's an effective humorous post you made, but in the story you referenced it appears the peer review was a lie, that no human read the submission prior to acceptance.

Re:Human vs. Software (1)

nedlohs (1335013) | more than 2 years ago | (#39770855)

There were no human proof readers and there was no peer review and there was no publication.

Did you not read the article you posted a link to? Or is the deception intentional?

Re:Human vs. Software (1)

gweihir (88907) | more than 2 years ago | (#39771111)

There are low-quality conferences that accept everything. The paper was published at such a conference.

Re:Human vs. Software (1)

FrootLoops (1817694) | more than 2 years ago | (#39771989)

No, the paper was not published and was not accepted at a conference. According to the article, the authors received word that the fake, computer generated paper had "been accepted for publication after peer-reviewing process in TOISCIJ [The Open Information Science Journal]".

They didn't take the hoax any further, though:

Davis said that he considered scraping together the $800 to see if Bentham would actually publish the fake paper, but considered that taking the hoax further would be "unethical."

"I think that the point has been made," he said. "And, I mean, it's $800, and I'm a graduate student."

The paper is clearly nonsense; here's a few lines from the beginning:

"Compact symmetries and compilers have garnered tremendous interest from both futurists and biologists in the last several years. The flaw of this type of solution, however, is that DHTs can be made empathic, large-scale, and extensible. Along these same lines, the drawback of this type of approach, however, is that active networks and SMPs can agree to fix this riddle."

Re:Human vs. Software (1)

Anti_Climax (447121) | more than 2 years ago | (#39771141)

You got me - I generated this post algorithmically... Guess I need to work on it.

Re:Human vs. Software (1)

FrootLoops (1817694) | more than 2 years ago | (#39772407)

The relevant journal is a non-traditional "open access journal" where articles are freely available (pseudo-random sample [benthamscience.com] ; others here [benthamscience.com] ), but article authors pay the publisher to publish. It's similar to self-publishing. I imagine TOISCIJ is not respected at all since in a brief search the only info I could find on it was related to the fake paper incident. While it is technically a "peer reviewed journal" (or at least it calls itself that, present evidence to the contrary), it's misleading not to immediately point out how it differs from most people's idea of traditional "peer reviewed journals".

Some scandals along the same lines:
  * The Bogdanov affair [wikipedia.org] , where two French twins, one a mathematician and the other a physicist, published apparent nonsense in respectable journals. Physicist John Baez (singer Joan's cousin, actually) called the papers "a mishmash of superficially plausible sentences containing the right buzzwords in approximately the right order. There is no logic or cohesion in what they write."
  * The Schön scandal [wikipedia.org] , where a German physicist claimed breakthroughs and published a number of papers. Journals withdrawing his papers include Science, Physical Review, Applied Physics Letters, Advanced Materials, and Nature.
  * The Sokal affair [wikipedia.org] , where a physicist published a rather hilarious paper [nyu.edu] in the journal Social Text. To be fair, that article was not peer-reviewed by a physicist.

Robo-graders? (5, Insightful)

Anonymous Coward | more than 2 years ago | (#39770673)

So you're telling me we've not only solved the natural language problem, we're also wasting it on grading essays?

We're not even close. Robo-grading essays is not only cheating, it's probably the worst disservice a school could do to its students. When you grade an essay you're looking at far more than technical accuracy (spelling, word count, formatting, valid citations). You're looking for meaning, articulation and interesting points of view. Robots can't teach critical analysis, can't offer helpful critiques of writing style, and certainly can't make judgement calls on how "good" an essay is.

Re:Robo-graders? (2)

masternerdguy (2468142) | more than 2 years ago | (#39770987)

The problem is human graders at the high school level only look at the things this program looks at. I've read and graded the kinds of 5 paragraph theme essays they are talking about and we don't look at content. It's sad you can replace SUBJECT in those essays with any noun (frogs, cars, china, hellcats) and the essay makes the same amount of sense.

Re:Robo-graders? (1)

i kan reed (749298) | more than 2 years ago | (#39771591)

Which should reveal that the problem is with the test, not with the mechanism of grading.

Re:Robo-graders? (1)

s0nicfreak (615390) | more than 2 years ago | (#39772537)

Unfortunately, with the teacher to student ratio most schools have today, there just isn't time to fix that problem...

Re:Robo-graders? (1)

Troyusrex (2446430) | more than 2 years ago | (#39771061)

I'd rather be graded by a robo grader than a real person. They are more consistent and aren't biased towards giving the students they like better grades. Yes they have their short comings but teachers do also.

Re:Robo-graders? (1)

s0nicfreak (615390) | more than 2 years ago | (#39772525)

Perhaps we could have such things submitted via computers, and the computer knows who submitted the paper and attaches the grade to that student, but the teacher does not know.

Grades without feedback (1)

Neil_Brown (1568845) | more than 2 years ago | (#39770755)

There may be situations in which simply getting a grade is of use, but, in most cases, I'd have thought that getting feedback was as important as getting the grade — knowing I have a good essay is one thing, but knowing where I went wrong, with guidance from someone skilled in the area, is the most important thing, since, otherwise, I have to guess as to where I need to improve.

Re:Grades without feedback (0)

Anonymous Coward | more than 2 years ago | (#39771019)

I see the application of this to be more for stuff like the GRE. Write two essays, get a numeric score 3 weeks later and that's the end of the feedback.

Its like any auto-text parsing - it gets it wrong. (2)

Chrisq (894406) | more than 2 years ago | (#39770761)

Our "corporate firewall" frequently gets things wrong. A site on "Sharp calculators" was classified as a weapons site, though I would imagine that stabbing anyone with one would be difficult. A "security software slap-down" was classed as "tasteless and violence", though no security software was injured. In short robo-graders are probably only any good for politicians, where the content doesn't matter as long as its delivered right.

Re:Its like any auto-text parsing - it gets it wro (0)

Anonymous Coward | more than 2 years ago | (#39771099)

And even if it isn't delivered right, you just shake the etch-a-sketch and start over.

Dr. vs. Mr. (0)

Anonymous Coward | more than 2 years ago | (#39770869)

According to the MIT website [mit.edu] , the "Mr. Perelman" the NYT article keeps mentioning is actually "Dr. Perelman". Does the NYT not believe in honorifics? Or do they just think that only MDs should be called "Dr."?

Then why should students bother? (1)

Picass0 (147474) | more than 2 years ago | (#39770895)

It a teacher is going to phone it in what does that tell the class? Why should a student even bother to write a paper? Maybe students should have auto-generation software.

For all of the things we screw up in the US one thing we've done (mostly) right is college education. People travel from all over the world to go to school in the US.

It's shit like this that will ruin it.

Re:Then why should students bother? (2)

DragonWriter (970822) | more than 2 years ago | (#39771277)

It a teacher is going to phone it in what does that tell the class?

Since when are teachers with any relationship to the students involved in standardized tests, except as proctors?

Re:Then why should students bother? (1)

Picass0 (147474) | more than 2 years ago | (#39772257)

The scope of software assisted grading goes beyond standardized testing. We've not talking about software looking at what ovals you filled in. It's using in grading essays, book reports, research papers, and I shudder to think perhaps even thesis papers.

This paper does raise an important point (0)

Anonymous Coward | more than 2 years ago | (#39770903)

When will something be done about the exorbitant pay that teachers' assistants receive?

free graders to jusdge content (3, Insightful)

peter303 (12292) | more than 2 years ago | (#39770977)

The rob grader can check spelling, grammar, structural style. The human grader can check for content accuracy and essay quality and creativity.

beast way to fool robot is to learn how to write (2)

peter303 (12292) | more than 2 years ago | (#39770999)

I dont worry too much about gaming the system. To "fool" the grader you'll have to learn spelling, grammar and structural style - exactly what the test-makers want.

Re:beast way to fool robot is to learn how to writ (0)

Anonymous Coward | more than 2 years ago | (#39771069)

I dont worry too much about gaming the system. To "fool" the grader you'll have to learn spelling, grammar and structural style - exactly what the test-makers want.

Did you actually read the paper that was used as an example? It is hilarious. And unfortunately, not that different from a lot of ramblings I've come across on the internet. If these eGraders are ONLY used to evaluate spelling and grammar and a human then evaluates the content to make sure it is not just random gibberish, then fine. But of course that is not how they will be used...

New York Times article snippet and more (2)

davidwr (791652) | more than 2 years ago | (#39771071)

News good. Paywall bad. A Google News search for the first couple of paragraphs should bring up either the NYT article or another copy of it.

Note that "em-dashes" have been changed to hyphens and "curly" apostrophes and quotation marks have been changed to "straight" versions marks to accommodate /. as viewed in my browser. Please avoid blocks of text that have -, ', or " when selecting text for search engines.

--cut here--
Testing Absurdities, Reading Worries and Robo-Grading
April 23, 2012, 8:19 a.m.
By Mary Ann Giordano

Week 2 of standardized testing begins in the New York City public schools - and so, it seems, does another week of testing wackiness.

The English Language Arts exam week ended on Friday with the decision by the state education commissioner, John B. King Jr., to scrap the answers to an absurd question - literally and otherwise - about a pineapple and a hare that had stymied eighth-grade test takers.

--cut here--

Further down we get to the relevant part:

--cut here--
Mr. Perelman tested the e-Rater and found that âoethe automated reader can be easily gamed, is vulnerable to test prep, sets a very limited and rigid standard for what good writing is, and will pressure teachers to dumb down writing instruction.â

You have to read the column to find out the many ways that the e-Rater misreads good writing. The examples are delicious - and pitiful. But to reveal one issue identified by Mr. Perelman:

  The e-Rater's biggest problem, he says, is that it can't identify truth. He tells students not to waste time worrying about whether their facts are accurate, since pretty much any fact will do as long as it is incorporated into a well-structured sentence. "E-Rater doesn't care if you say the War of 1812 started in 1945," he said.

Give E.T.S. credit for allowing Mr. Perelman to conduct his testing. Two other major testing services, Vantage Learning and Pearson - developer of the offending English Language Arts exam - said no.
--cut here--

The article linked in this /. article's summary refers to another article:

https://www.nytimes.com/2012/04/23/education/robo-readers-used-to-grade-test-essays.html [nytimes.com]

Here are some snippets from it, in case you need them for your search engine:

--cut here--
Facing a Robo-Grader? Just Keep Obfuscating Mellifluously
By MICHAEL WINERIP
Published: April 22, 2012

A recently released study has concluded that computers are capable of scoring essays on standardized tests as well as human beings do.

Mark Shermis, dean of the College of Education at the University of Akron, collected more than 16,000 middle school and high school test essays from six states that had been graded by humans. He then used automated systems developed by nine companies to score those essays.
--cut here--

This article in turn links to:
www.documentcloud.org/documents/346138-essay-awarded-a-top-grade-by-e-rater.html
which is also linked in this /. article's summary.

Robo Graders (1)

MrKaos (858439) | more than 2 years ago | (#39771109)

When I saw the title I thought it was referring to those robot graders that they use to level the road substrate whilst making roads (A new bridge is being built near my work) They are quite fascinating to watch work but I wouldn't want to get in the way of one of them.

It must be done (1)

MrKaos (858439) | more than 2 years ago | (#39771159)

In Soviet Russia:

Grades rob you!

We now return you to normal discussion.

Without strong AI, robo-graders are worthless (2)

gweihir (88907) | more than 2 years ago | (#39771201)

Unless what you teach the students is worthless as well. If it is just conformance to secondary things like spelling, basic grammar, sentence-length, superficial structure, etc. then robo-grading will do fine. Of course, none of the students being taught this way will learn to write anything of worth, ever. For that you need a competent and intelligent human being (or at least an equivalent intelligence) that understands what the student was trying to say and whether he/she succeeded or not, and why precisely. Grading involves as its most important component the feedback to the student, the actual grade is secondary and does not help the student improve his/her writing at all.

When my robot .... (1)

PPH (736903) | more than 2 years ago | (#39771219)

... knows its being graded, it just gets nervous and starts giving wrong answers.

The whole idea is ridiculous (1)

aaaaaaargh! (1150173) | more than 2 years ago | (#39771783)

As someone who is working in linguistics close to AI research I can attest that the whole idea of automated grading of essays is completely ridiculous and if it is indeed used as the post suggests will likely ruin generations of students. Apart from not working, it is also wrong in various other respects such as sending the wrong signals to young students, implicitly ridiculing the hard work that writing actually is, saving money in the wrong place, and so forth.

I mean, com'on ... all of the above is so obvious that it shouldn't even have to be mentioned. What kind of imbecile illiterate would allow grading of essays by a statistical text-mining program anyway?

Re:The whole idea is ridiculous (1)

misexistentialist (1537887) | more than 2 years ago | (#39772161)

Teachers don't really read 100 essays. This system is just more accurate and honest.

This isn't robo-grader specific... (3, Interesting)

digitalsolo (1175321) | more than 2 years ago | (#39771795)

This problem is not specific to robo-graders. I made a solid rule of finding topics that I found interesting -and- were highly unlikely to be areas of specialty for the teacher/professor/TA grading the paper. It took slightly more effort to find the "right" topics, but it more than paid off in the long run, since the likelihood of the average test grader spending days researching every 10+ page paper they are grading is pretty low.

Obviously as your volume of large papers and required topics narrows this becomes less effective, but it's quite a good system in high school through most of undergrad studies. I guess I assumed most people did this. FWIW, I did write pretty good papers, they weren't full of B.S. (well, just average volumes of B.S.) but by getting the topic as far "out" as possible, it helped minimize criticism outside of the basic structure, citation, etc.

Abused like Google (1)

sattu94 (1989362) | more than 2 years ago | (#39771975)

This program could be abused just like some websites manage to fool Google's PageRank algorithm.
People say E-Rater checks for proper grammar and spellings, the pages in Google's results also have proper grammar and all. The actual content is what is not wanted. If someone manages to write a completely unrelated essay.. but complete with proper spellings and grammar, he might be able to fool the software.

A better approach would be a software that would require you to input the essay topic for which it would then scourge the internet for related keywords and all. Something just like Siri does. AND THEN checks you essay for proper content + grammar + spellings. If any weird exceptions are encountered, they are flagged for manual checking.

This should give a full-proof E-Rater.

SAT essays never required factual information (0)

Anonymous Coward | more than 2 years ago | (#39772003)

I took the SAT a few years after they added the essay section. We were always told that it's important to back up any argument you make with facts, but the accuracy of these facts would not be checked. If you wanted to support your argument with events from a war but you weren't sure what year it was? Just guess. This is when the essays were scored by humans (maybe they still are?) according to a rubric.

Here's an interesting blog post [applerouth.com] on the subject.

Masking another problem? (1)

jandrese (485) | more than 2 years ago | (#39772137)

Are robo-graders as good as or better than human graders because the quality of the human graders is so low? When you have literally millions of SAT essays to grade, you can't afford to be choosy with your staff and as a result the quality of the work is depressed.

Proof that colleges don't care (1)

MobyDisk (75490) | more than 2 years ago | (#39772661)

The article reveals frightening things about how colleges are structured:

They talk about how accurate the robo-graders are:

Computer scoring produced “virtually identical levels of accuracy, with the software in some cases proving to be more reliable,” according to a University of Akron news release.

That's amazing! So let us see why they are so good:

Graders working as quickly as they can — the Pearson education company expects readers to spend no more than two to three minutes per essay— might be capable of scoring 30 writing samples in an hour.

Aha! So it isn't that the robo-graders are as good as human graders. The robo-graders are as accurate as a person who is not given enough time to read the actual essay. So if I create a robo-surgeon that is as good as a surgeon who has only 5 minutes to perform open-heart surgery, can I then say that my robo-surgeon is as good as a real surgeon? Of course not - they gamed the metrics to make the robo-graders look good. Is anyone else concerned that the dean of University of Akron only cares about how fast the tests are graded?

Later on in the article:

They [E.T.S] say Mr. Perelman is setting a false premise when he treats e-Rater as if it is supposed to substitute for human scorers.

So the robo-graders are not a substitute for human scorers. That isn't what the schools seem to think.

This is great: students will be graded by robots, so they will get degrees with no real writing skills. Then those students become the teachers, who cannot grade essays, compounding the problem with each generation. I fear this is how Idiocracy will come to pass: Everyone will be trained and educated in professional nonsense.

Important to understand scale (1)

scamper_22 (1073470) | more than 2 years ago | (#39772697)

I think computers have the ability to automate huge areas people think require 'judgment'. Will they be perfect or catch odd cases? Probably not. Yet, that must be weighed against the ability to provide the service on mass.

For example, radiographers are currently some of the highest paid medical professionals. Today, automated detection is already quite high in terms of accuracy (80%+). About the same as human radiographers. For example.
http://www.breastcancer.org/symptoms/testing/new_research/20081001b.jsp [breastcancer.org]

Is it possible a human radiographer could detect weird anomalies or something. Of course. But as a mass provided service, the computer would be way cheaper and provide affordable healthcare. Obviously before surgery, a human should probably double check :P

While I doubt the technology is there yet, I certainly don't think it impossible to have robo-grading for the evaluation of mass essays. Again, we have to compare it to the real world with people. Sure, a human grader going through every essay in detail might be better. But on average how thorough are graders? How thorough are patent examiners in examining patents on a mass scale?

Could we not imagine a system where the professor lists points they 'expect' to see in the essay. Somehow natural language processing can check for these points.
I could certainly imagine that working for essays you might write in high school for Shakespeare or an analysis of a book.
If I remember my high school, there was always a limited set of themes and points discussed.

Of course professors can always recheck for really creative work that the program mucks up.

But I think people overestimate the creativity of people in the school environment when applied to a large user set.

If you loo

the GRE does this (0)

Anonymous Coward | more than 2 years ago | (#39772717)

The GRE, which is like the SAT for grad students, uses robotic graders to identify essays, and as it turns out, the robot is looking for three things; grammar, essay, and the traditional "5-paragraph" format. It's not interested in your ability to compose thought, and it's completely inept at judging whether or not a given student's writing is on-par for the expectation of graduate technical writing.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?