×

Announcing: Slashdot Deals - Explore geek apps, games, gadgets and more. (what is this?)

Thank you!

We are sorry to see you leave - Beta is different and we value the time you took to try it out. Before you decide to go, please take a look at some value-adds for Beta and learn more about it. Thank you for reading Slashdot, and for making the site better!

Grading Software Fooled By Nonsense Essay Generator

samzenpus posted about 7 months ago | from the it-is-good-report dept.

Education 187

An anonymous reader writes "A former MIT instructor and students have come up with software that can write an entire essay in less than one second; just feed it up to three keywords.The essays, though grammatically correct and structurally sound, have no coherent meaning and have proved to be graded highly by automated essay-grading software. From The Chronicle of Higher Education article: 'Critics of automated essay scoring are a small but lively band, and Mr. Perelman is perhaps the most theatrical. He has claimed to be able to guess, from across a room, the scores awarded to SAT essays, judging solely on the basis of length. (It’s a skill he happily demonstrated to a New York Times reporter in 2005.) In presentations, he likes to show how the Gettysburg Address would have scored poorly on the SAT writing test. (That test is graded by human readers, but Mr. Perelman says the rubric is so rigid, and time so short, that they may as well be robots.).'"

Sorry! There are no comments related to the filter you selected.

most schools ignore sat essay (3, Insightful)

litehacksaur111 (2895607) | about 7 months ago | (#46885707)

I though most schools don't even care about the essay. Also the elite schools nowadays prefer the ACT and SAT II subject tests to demonstrate real knowledge. The SAT is really a dumb test, especially with all the coaching resources available now.

Re:most schools ignore sat essay (3, Funny)

Anonymous Coward | about 7 months ago | (#46885989)

Your post tells me that you didn't score all that well on the SAT. Bad grammar, incoherent thoughts.

Re:most schools ignore sat essay (0, Troll)

Anonymous Coward | about 7 months ago | (#46885999)

The SAT only tests for rote memorization, anyway. Even the essays don't require any real critical thinking. Rote memorization != intelligence. Someone with terrible grammar can be far more intelligent than some worthless rote memorization monkey.

Re:most schools ignore sat essay (1)

AK Marc (707885) | about 7 months ago | (#46886155)

vocabulary = rote memorization? basic math = rote memorization? And anything learned by rote memorization is bad.

Sounds like someone who thinks he's smart who got a low score...

Re:most schools ignore sat essay (2)

lgw (121541) | about 7 months ago | (#46886221)

basic math = rote memorization

Yup, it sure is, and sadly this is contentious. Basic numeracy is impossible without memorizing tables for addition, and multiplication. Seen a modern math textbook that shows what buttons to press on the calculator? Seen the recent "common core" controversy about quite crazy approaches to basic math that seem motivated by avoidance of memorization (it's the revenge of new math!). Sigh. But then, do they allow calculators on the SAT?

Re:most schools ignore sat essay (2, Insightful)

AK Marc (707885) | about 7 months ago | (#46886257)

I got an 800 on my math (old SAT, back in 1990), and I still count on my fingers. They allowed calculators (only a few approved ones), and, of course, I used it. The SAT math was about speed, efficiency, and answering the right question. Most people had a problem with the latter. I don't know all of my multiplication tables, it wasn't my thing, but seeing the question and figuring the best way to word it to find the answer was my thing. Didn't miss a question.

Re:most schools ignore sat essay (1)

lgw (121541) | about 7 months ago | (#46886835)

Wow, they sure didn't allow calculators when I took it, not very long before that. Didn't slow me down. Couldn't use a calculator in high school physics either.

I don't know all of my multiplication tables,

Well, at least innumeracy isn't new? Eesh. I mean, I can understand not memorizing log tables with the rise of calculators, but can you really not tell from a glance at the unit price what 6 of something cost?

Re:most schools ignore sat essay (1)

mwvdlee (775178) | about 7 months ago | (#46886887)

6 of something is pretty easy.
Back in grade school we had to learn multiplication tables up to 20.
17 of something is a lot harder.

Re:most schools ignore sat essay (0)

Anonymous Coward | about 7 months ago | (#46887301)

6 of something is pretty easy.
Back in grade school we had to learn multiplication tables up to 20.
17 of something is a lot harder.

Why?
I happen to remember 17x17 is 289, but 17x17 is just (17x10=170)+ (17x7=10x7+7x7=70+49=110+9=119)
If I didn't offhand and that multiply by ten wasn't a mental shortcut I'd do 17x20=17x2x10=340-(17x3=30+21=51)=289

Or 39x39=40x40 - 40 (to get 39*40) which is (39*39+39) so -39=
40x40 -40-39=1600-79=1500+21=1521
67*67=67*60+7*67
=60*60+7*67+7*67
=3600+420+49+420+49
=4020,4440+98=4540-2=4538
The last line actually took the longest.

I wish I could have been able to do substitution in diffyQs as fluidly but that was my limit, perhaps since I knew I'd never use them for anything.

Re:most schools ignore sat essay (1)

AK Marc (707885) | about 7 months ago | (#46887049)

1.37 * 6 is beyond easy mental math for me. 1*6=6, 33*3*2=200, so 1.37 =~8. That's as close as I'd ever come to calculating 6 of something without taking out a calculator. I can calculate the trajectory of a bullet (harder than most think it) but rote memorization is not how my brain is wired. I also don't know the letter after "j" in the alphabet, but have a vocabulary in the 99+ percentile. Not having something "easy" committed to memory is unrelated to being able to do much much harder things.

Re:most schools ignore sat essay (0)

Anonymous Coward | about 7 months ago | (#46886259)

vocabulary != rote memorization. I have always had poor memorization skills. However, My vocabulary has always been tested to be far above my level. Granted if you were to just present me the words with semi-random definitions I would probably be screwed. But if you see how the word is used in a sentence like most test present it to you, it is fairly easy to determine the meaning of a word from just it's grammar, context and usage. I realized this early on and was able to exploit it all through school. The thing that is rote memorization is spelling, as the English language is not phonetic despite being taught that way. And grammar has a lot of unpredictable quarks as well.

Re:most schools ignore sat essay (1)

Anonymous Coward | about 7 months ago | (#46886329)

vocabulary = rote memorization?

To a large degree, what they test for is rote memorization, yes. Vocabulary is necessary (though a large vocabulary doesn't mean someone is intelligent, and it isn't impressive), but the drill-and-kill bullshit has to stop.

basic math = rote memorization?

The way we're teaching it? Yep. Our tests certainly don't reveal whether or not someone has any true understanding of the material.

The other guy (lgw) who said that basic math is all about memorization and memorizing tables is wrong. There are low-level processes to understand. Mindless memorization and repetition should not be encouraged.

And anything learned by rote memorization is bad.

The content itself is not bad, no. But rote memorization is often a terrible way to go about it, as it almost always means that someone isn't getting an actual understanding of the material.

No one on this planet would say that no one should retain any information; then you couldn't do a damn thing. The problem is, we rely far, far, far too much on memorization, and in the end, nothing is memorized and nothing is understood. Memorization should almost never be forced, and almost never be tested for. The better teachers at the better universities have realized this.

Sounds like someone who thinks he's smart who got a low score...

Sounds like someone who is desperately trying to defend poorly-designed tests so he can feel superior to others. Can't shatter that false sense of superiority, can we? That's about as valid as what you said, and just as likely to be false.

Attacks like those don't do a damn thing. I fail to see why people are so obsessed with making random guesses about others, as if it's relevant.

Re:most schools ignore sat essay (1)

AK Marc (707885) | about 7 months ago | (#46887107)

Attacks like those don't do a damn thing. I fail to see why people are so obsessed with making random guesses about others, as if it's relevant.

So the views and opinions of the "other side" are irrelevant? That's a little heartless and understanding someone's viewpoint helps understand their views. That you don't understand this indicates you need more rote memorization to make you smarter.

Re:most schools ignore sat essay (0)

Anonymous Coward | about 7 months ago | (#46887219)

So the views and opinions of the "other side" are irrelevant?

No, but attacks like those aren't even relevant to the discussion. Unless you think that randomly guessing that another person did poorly on the SAT was where the discussion should head? Because I don't.

To me, it's no different than randomly spewing forth a bunch of things you think are facts. Even if they are actually true, it's irrelevant. I don't need to understand what you think my SAT score was to understand your views about rote memorization.

That you don't understand this indicates you need more rote memorization to make you smarter.

Good idea.

Re:most schools ignore sat essay (1)

AK Marc (707885) | about 7 months ago | (#46887339)

To me, it's no different than randomly spewing forth a bunch of things you think are facts. Even if they are actually true, it's irrelevant. I don't need to understand what you think my SAT score was to understand your views about rote memorization.

What are my views on rote memorization? Do you know, or were you too busy attacking me to even bother to stop and understand.

Re:most schools ignore sat essay (-1, Flamebait)

Anonymous Coward | about 7 months ago | (#46886455)

The SAT only tests for rote memorization, anyway. Even the essays don't require any real critical thinking. Rote memorization != intelligence. Someone with terrible grammar can be far more intelligent than some worthless rote memorization monkey.

Sounds too much like butt-hurt over a low score.

1. The SAT "only tests for rote memorization"? Cite please.

2. "Rote memorization != intelligence"? Again, cite please.

3. "Someone with terrible grammar can be far more intelligent than some worthless rote memorization monkey." And a random schmuck off the street can be a better shot with a basketball than an NBA player, or drive a golf ball farther and more accurately than a PGA tour pro golfer. "Can be"? Sure. But that ain't the way to bet.

Whether you like it or not, if you get a good score on the SAT, you're smart.

If you get a bad score, well, you might be smart. But the odds are you ain't.

Sorry. The real world ain't Lake Woebegone. About half really are below average in any trait - some significantly so.

Re:most schools ignore sat essay (1)

Anonymous Coward | about 7 months ago | (#46886727)

Sounds too much like butt-hurt over a low score.

Sounds like an ad hominem. Randomly attacking other people by spewing forth irrelevant guesses isn't going to make your actual arguments more powerful, you know.

1. The SAT "only tests for rote memorization"? Cite please.

If you need a citation to show that you only need rote memorization to pass the SAT, then you don't understand intelligence or education. The math problems consist of solving arbitrary problems that are essentially spelled out for you, or solving idiotic word problems which are, again, basically spelled out for you. "Oh, I have to use this formula to solve this problem!" Never, at any point, do these problems test for actual understanding (such as a deep, intuitive understanding of why it all works) of the material. If you disagree with this, I suspect you yourself do not know what it means to understand mathematics, because you'd know there is more to it than just solving repetitive problems on a poorly-designed test.

2. "Rote memorization != intelligence"? Again, cite please.

You think that rote memorization is intelligence? If all you are is a rote memorization monkey, you'd have no way to take information and actually innovate in such a way that our understanding of the universe around us increases. Einstein was intelligent; he was an innovator. Rote memorization monkeys are not intelligent, though someone with a good memory may be intelligent, but it's by far no guarantee.

But that ain't the way to bet.

There's actually no reason to think either way, as the ability to use language 'correctly' is such a low bar for beings that evolved to use language that it isn't a real sign of intelligence when compared to the intelligence of people who actually innovate and increase our understanding of the universe.

Whether you like it or not, if you get a good score on the SAT, you're smart.

Whether you like it or not, your statement is false. You have no idea what it means to be intelligent.

If you get a bad score, well, you might be smart. But the odds are you ain't.

It's also funny that you tried to make me play the citation game, yet you put forth many baseless statements like this.

The chance that a specific individual is intelligent is very low, not just if you do poorly on the SAT.

About half really are below average in any trait - some significantly so.

About half? And who cares about averages? I care about someone's overall intelligence, not how they compare to others. A grand majority of people simply aren't intelligent.

Also, I'm not going to play the pointless 'Cite sources that reach conclusions you agree with, and others will try to counter you by doing the same!' game; it's useless bullshit. All the conclusions I reached can be verified by actually looking at the SAT. If you disagree with my conclusions, chances are you don't understand what a real education is, what intelligence is (i.e. have a general grasp of what an intelligent person might be able to do), or both.

Re:most schools ignore sat essay (1)

InsultsByThePound (3603437) | about 7 months ago | (#46886117)

Yanno, I don't think perfect grammar or lack of in an informal setting is a test of intelligence. Just be good enough to get the idea across.

I never sat in an English class and thought my teacher was an amazing wizard of thought through their impeccable language skills. Mostly it was "What a dull bitch." I have the same thoughts of spelling/grammar nazis online who try to use it as counterargument or to elevate themselves over the other person.

My math teachers otoh... those were wizards. Their grammar was sometimes horrible and I forgave their faults at every turn.

Re:most schools ignore sat essay (2, Insightful)

ceoyoyo (59147) | about 7 months ago | (#46886219)

Odd you choose math as an example, a subject where your grammar must be perfect or what you've written is nonsense.

Re:most schools ignore sat essay (1)

InsultsByThePound (3603437) | about 7 months ago | (#46887279)

What's funny is colloquial language is the exact opposite and yet some people treat any aberration as if it would cause a million parsing errors.

If yo udon't like don't use (-1)

Anonymous Coward | about 7 months ago | (#46885731)

Is my motto. Leave. Go away.

Architecture School! (0)

Anonymous Coward | about 7 months ago | (#46885761)

While they'll eventually be able to calibrate the computers to recognize the three keyword gibberish essay, it sounds like a perfect tool for writing architectural essays and presenting projects in school. Use the following three keywords - Perhaps; Palimpsest; Paradigm - and you'll be ready to design a building!

Re:Architecture School! (2)

Opportunist (166417) | about 7 months ago | (#46885829)

It sounds like the software would be perfect for writing audit reports. You hand in a phone book sized report, but all they ever read is the management summary.

But DARE to hand over just the relevant pages that you know will get read. Did you work at all, if THAT is your whole report?

Re:Architecture School! (2)

ewibble (1655195) | about 7 months ago | (#46885941)

Your right you are encouraged to write long documents, but it should really be the opposite, writing is about communicating, if your document is so long that people don't bother reading it, the document has failed in its main purpose.

This standard should be applied to legal documents, such as License agreements, Insurance agreement, What your ELA is more than 100 words long, you don't expect anyone to read this do you? Agreement Invalid. If you need longer it should ensure that people understand what they are agreeing to, maybe run a 1 year course of something.

100 words yes!

Re:Architecture School! (0)

Anonymous Coward | about 7 months ago | (#46886053)

Isn't our tax code already written this way?

Re:Architecture School! (1)

n6kuy (172098) | about 7 months ago | (#46886385)

Yes, but they run it through spinbot.com before it becomes official regullations.

Irrelevant (2, Insightful)

Ol Olsoc (1175323) | about 7 months ago | (#46885771)

As long as Precious gets an "A', Helicopter Daddy, and Blackhawk Mommy won't try to have the school president fired for ruining Precious's permanent record.

Re:Irrelevant (3, Funny)

smitty_one_each (243267) | about 7 months ago | (#46885807)

Hey, Helicopter Daddy and Blackhawk Mommy dropped good boodle for that 'A', mister!
You can just stand down from all that meritocratic whinging right now, mister.

Re:Irrelevant (1)

Opportunist (166417) | about 7 months ago | (#46885833)

That's when I wish I had a SAM for such occasions...

Re:Irrelevant (3, Interesting)

TheMeuge (645043) | about 7 months ago | (#46885863)

At least helicopter daddy and blackhawk mommy give a shit about the Precious. Or do you prefer the absent daddy and welfare mommy? People DO go overboard... but I feel like the pendulum is starting to swing entirely too far the other way.

Re:Irrelevant (1)

AK Marc (707885) | about 7 months ago | (#46886165)

It's swung so far the wrong way that in many places, aggressive parents are required to get the minimum education proscribed by law. If you don't have active parents, the school actively punishes the children. The thought is that they are already failures because of their parents, so better to get them used to failure and hope they drop out, so as not to harm the schools statistics.

Can you blame them? (0)

Anonymous Coward | about 7 months ago | (#46886107)

If your kid doesn't get the grades, he's working at Walmart or serving coffee. Who cares if he truly learned anything. He's gotta get in the door. And starting out a great GPA is the best way to go - I don't care what the self proclaimed hiring managers say here on Slashdot. Anyway, if I looked at their hiring practices, I'd find that they are fooling themselves.

Re:Can you blame them? (2)

AK Marc (707885) | about 7 months ago | (#46886179)

What are you talking about? I've gotten lots of good paying jobs, and nobody once looked at my grades. Except when applying for further education, and even then, they aren't important if you test well. Where have you seen where a transcript is required for a job application? Never, that's why so many CEOs get caught lying on resumes (until they post to LinkedIn and someone recognizes them and knows they didn't get what they claim and turn them in). Even the $10,000,000 a year jobs don't look at actual grades. But no, some AC claims that grades matter. So they must, even if they don't.

Re:Can you blame them? (1)

lgw (121541) | about 7 months ago | (#46886237)

Hiring by big companies for internships and recruiting for new college hires are usually filtered by GPA before any engineer or manager sees the stack or resumes. I never had a company care about my grades, but then I was never actively recruited at college age, and so my first dev job was remarkably exploitive, not one of the good ones.

Beyond one's first full-time job, I can't imagine grades ever coming up.

Re:Can you blame them? (1)

Jane Q. Public (1010737) | about 7 months ago | (#46886895)

Hiring by big companies for internships and recruiting for new college hires are usually filtered by GPA before any engineer or manager sees the stack or resumes.

That's a really dumb thing to do, since in most fields the majority of the best performers were not the ones with the highest grades in school, and vice versa.

Statistically you're usually better off taking the B or A- student than the one who got a 4.0.

To generate the keywords takes knowledge (1)

LetterRip (30937) | about 7 months ago | (#46885791)

Since the essays are grading subject knowledge, and it takes subject knowledge to provide the keywords, it is fairly irrelevant if the essay happens to be structured in a manner that is nonsensical.

Demonstration of deeper understanding, if it needs to be tested, can be achieved via other types of questions.

Re:To generate the keywords takes knowledge (1)

Opportunist (166417) | about 7 months ago | (#46885805)

The next generation of the software will have a keyword database attached for every subject possible to ensure that every student takes different keywords (chosen randomly from the stock).

Then your grades are pretty much dependent on whether the random number generator chooses keywords that the grading software likes.

I fail to see the difference to now, to be honest, it's just way less work on the student's side.

Re:To generate the keywords takes knowledge (2)

EmagGeek (574360) | about 7 months ago | (#46885823)

Did you happen to read TFA? In the TFA, it is said that the College Board does not take points off for factual errors. In fact, it says that it cares not for factual errors, because errors in fact seldom subtract from the quality of the essay being graded.

WTF, right?

Re:To generate the keywords takes knowledge (5, Insightful)

MrBigInThePants (624986) | about 7 months ago | (#46885851)

Not being from the USA, every article I ever read about your education system just leaves me scratching my head.

How on earth did you guys let it get so ridiculous??

Re:To generate the keywords takes knowledge (0, Flamebait)

AK Marc (707885) | about 7 months ago | (#46886193)

Teachers are in unions, so the Republicans want to close all public schools. When 1/2 of your government wants to harm the children, and the other 1/2 is worried about other things more, then you get a government that makes sure that every year is the worst year for education in the US. So this year is the worst year in education in the US. Eventually, it will fail, and we'll be left with government-funded for-profit schools pushing the Coke or Baptist agenda, and no minimum education for anyone.

Re:To generate the keywords takes knowledge (1)

lgw (121541) | about 7 months ago | (#46886261)

so the Republicans want to close all public schools

This right here is why the USA is fucked. People just can't speak coherently about politics any more. (And, of course, it's the teacher's unions who are forcing high-performing charter schools to close, but that's hardly part of the Dem platform, just a consequence of seeking the public sector employee vote).

Do you actually think either party has a goal other than the best schools? The disagreement is over how to achieve that, and I've never heard anyone arguing for no public schools, and more than for no private schools (but "no religious schools"? That one I hear occasionally.)

Re:To generate the keywords takes knowledge (1)

MrBigInThePants (624986) | about 7 months ago | (#46886961)

The problem is that corruption, cronyism and a shite environment leads to corrupt and shite people all over in my experience.

Corruption is rot that spreads throughout - it does not care to limit itself to one species of host.

I always find it amusing when people try to blame only one source for this sort of thing.

Re:To generate the keywords takes knowledge (1)

MrBigInThePants (624986) | about 7 months ago | (#46886977)

PS: And I doubt they want to CLOSE the schools. Just make sure that they are vast slums churning out ignorant and easily led livestock to be harvested for their min-wage labour....and most likely their organs....

Re:To generate the keywords takes knowledge (2)

AK Marc (707885) | about 7 months ago | (#46887015)

Do you actually think either party has a goal other than the best schools?

Yes. I honestly believe that the Republicans want to disband public education and have a merit-based entry to private schools (parent's merit, not children's), paid for with taxpayer dollars. It's "revenge" for having forced them to educate the poor for so many years.

I've never heard anyone arguing for no public schools,

I have. Charter-schools and for-profit private schools only, and they would be banned by law from having unions and could reject children from admission for arbitrary reasons (including race and religion).

I've gone to plenty of party meetings for Libertarians and Republicans, and I've seen what some people have advocated.

Re:To generate the keywords takes knowledge (0)

Anonymous Coward | about 7 months ago | (#46887141)

Well, then you're crazy. They want to be able to choose good schools instead of the crap that passes for public ones. I vote more money to the schools all the time but it does little good, as you can see, as it's not a money problem, it's a matter of screwed up incentives all around. And it's funny you say that, as the Republicans want to let people choose schools based on how good the school is, rather than being locked into the public schools which, again, utterly suck for the most part.

Re:To generate the keywords takes knowledge (1)

AK Marc (707885) | about 7 months ago | (#46887329)

Money isn't the issue because the Republicans keep passing unfunded mandates that cost the schools money, but don't help education. There is no amount of money that the Republican's can't find a way to waste. The "cost" of public school education (in-class spending) is *below* private school cost. But we keep hearing about how public schools are so much more expensive. Why?

Re:To generate the keywords takes knowledge (1)

Kaenneth (82978) | about 7 months ago | (#46887019)

Both parties fear educated voters.

Re:To generate the keywords takes knowledge (2)

MrBigInThePants (624986) | about 7 months ago | (#46886957)

Teachers are in strong unions also here in NZ. (despite anti union legislation decimating them in the last decade)

The right wingers here (and their ex-currency trader, cheesy smiled leader) have been trying desperately to beat on them but NZ has one of the best bang for buck education systems in the world. (i.e. Our teachers are not paid that high but the performance indicators are in the top grouping.)

Just wanted to mention that for the inevitable people who will read your comments and think "unions baaaad" like some ideological zombie.

Re:To generate the keywords takes knowledge (1)

AK Marc (707885) | about 7 months ago | (#46887033)

What's funny about unions in NZ is they aren't all that strong because a non-union worker has more protections in NZ than most union workers in the US. My wife is in PSA, and they do nothing, as far as she can tell. Joining the union allows one to vote on the collective agreement, but non union members ended up with the same contract. Slightly different termination rules, but not very different.

In the US, "unions are bad" has been given as a reason to close all schools because it's better for the children to be uneducated than indoctrinated by union workers. It's insane, I was making fun of it, not supporting it.

Re:To generate the keywords takes knowledge (1)

TubeSteak (669689) | about 7 months ago | (#46886617)

How on earth did you guys let it get so ridiculous??

Never underestimate the power of Intelligent Design.

Re:To generate the keywords takes knowledge (0)

Anonymous Coward | about 7 months ago | (#46886939)

Too much time watching USian TV telling us how awesome we are, and that we don't need to be smart we just need to b cool. This became a self fullfuling prophecy when all the chicks didn't want to date smart guys just the cool ones. So now we have an entire contry filled with idiots who thing they are really cool and better than all those other backwards counties like Malaysia, where they don't even have roads and shit.

That and the Jews

You don't need software (4, Insightful)

Anonymous Coward | about 7 months ago | (#46885793)

... because Slashdot shows that humans already make evaluations about articles without reading them.

Quid pro quo (3, Insightful)

Opportunist (166417) | about 7 months ago | (#46885797)

When you're too lazy to read my essay to grade me and let software do it, I don't really see no moral problem with doing the same to write the essay.

Re: Quid pro quo (1)

Mr. Sketch (111112) | about 7 months ago | (#46885917)

> I don't really see no moral problem

I guess someone should have graded your essays a little more closely instead of relying on a robot.

Re:Quid pro quo (4, Insightful)

Anubis IV (1279820) | about 7 months ago | (#46886087)

As someone who graded hundreds of essays while serving as a teaching assistant for a senior-level engineering ethics course, I have to say that I find your lack of integrity rather appalling. Your moral obligation to write the essay yourself is independent of the method they use for grading it. Just because someone else is doing a lousy job does not mean that you suddenly have a license to short-change them for what you're obligated to do.

I would guess that I graded around 300-400 essays during the three semesters I served as a TA, and that I probably averaged around 20 minutes per essay, since I was a strong believer in providing useful feedback over things the students could improve, even if they weren't necessarily incorrect. That said, other TAs spent as little as a minute or two per essay, and barely provided any feedback at all. Regardless of how much time the TAs did or didn't spend on the essays, however, the students had the same obligations, and rightfully so.

Let me put it in Engineering terms... (2, Insightful)

Anonymous Coward | about 7 months ago | (#46886139)

If I've been hired to build a Potemkin village, then it would be unethical of me to spend time constructing interiors for the buildings.

The English department has some nice courses on compositional writing where I can get real feedback on my progress on those skills. As far as the machine-graded essays for any other Department -- either I understood the topic before writing the essay or I didn't and if I didn't then a no-feedback essay isn't going to fix the problem.

Re:Let me put it in Engineering terms... (1)

KingOfBLASH (620432) | about 7 months ago | (#46886811)

If I've been hired to build a Potemkin village, then it would be unethical of me to spend time constructing interiors for the buildings.

Not unethical, idiotic.

Re:Quid pro quo (3, Insightful)

number17 (952777) | about 7 months ago | (#46886201)

Your moral obligation to write the essay yourself is independent of the method they use for grading it.

Students pay big bucks and expect to have experts in the field teach them and grade their work. It sounds like these schools are off-shoring their marking so that they can do other work (ie Research). If the school was upfront, before paying tuition, that they were going to just send your essay to Bangladesh for marking then I would be ok with having a moral obligation to write the essay myself.

Re:Quid pro quo (0)

Anonymous Coward | about 7 months ago | (#46886307)

No, it isn't. Good try though.

It also means that teachers are not reviewing the CONTENT of this work, and in so doing the schools have broken their end of the social contract implicit in school work. Expecting the students to uphold their end knowing the school isn't going to uphold its end is ridiculous.

Re:Quid pro quo (1)

DoofusOfDeath (636671) | about 7 months ago | (#46886317)

Your moral obligation to write the essay yourself is independent of the method they use for grading it.

That's an interesting claim. I'd be curious to hear you make an argument to support it.

Re:Quid pro quo (0)

Anonymous Coward | about 7 months ago | (#46886415)

Incorrect. In all aspects of life we are pushed to doing things in a manner that gives us the feedback that we desire. If the feedback you desire is a good grade and that grade is obtained by putting the right words in the right places regardless of their meaning then that is what people will do and if the robot gives them an 'A' for that nonsense than so be it.

The problem here is just like bad management at pretty much every company ever. The goal is not to 'do it right' or 'do a good job' as so many new workers idealistically assume. The goal is simply to make yourself look good, and/or make others look bad such that you get a bigger piece of the pie. If a company is not recognizing and rewarding people that do good work than people will do whatever the company IS rewarding.

If you have ever worked for a company that evaluates people on metrics you know that you only get what you measure.

Re:Quid pro quo (1)

ub3r n3u7r4l1st (1388939) | about 7 months ago | (#46886469)

"I have to say that I find your lack of integrity rather appalling."

Unfortunately, engineering ethics is something that is normally taught in the undergrad level. With the onslaught of international graduate students and H1-B workers, engineering ethics become a "luggage" for competitiveness om the domestic student or workers to compete with these people who treat plagiarism as honorable activity. Any person with integrity will lose out to those who has no bottom line to achieve the goals.

Re:Quid pro quo (1)

ruhri (1480067) | about 7 months ago | (#46886677)

As someone who graded hundreds of essays while serving as a teaching assistant for a senior-level engineering ethics course, I have to say that I find your lack of integrity rather appalling.

As someone who served on the IEEE ethics committee I find your appeal to argumentum ab auctoritate [wikipedia.org] rather appalling. You should know the distinction between ethics and morals. One could make the Utilitarianist [wikipedia.org] case, in which (arguably) the behavior cited is morally OK. One could also make the Kantian [wikipedia.org] argument that (arguably) comes closer to what you were condoning.

Regardless of how much time the TAs did or didn't spend on the essays, however, the students had the same obligations, and rightfully so.

As an assignment for your ethics class: please elaborate, under which ethical systems, the above statement holds true or not, and why.

Re:Quid pro quo (2)

KingOfBLASH (620432) | about 7 months ago | (#46886805)

The problem is that technology allows universities to take short cuts in education, and not in the students advantage. Add to that some of the current goings on in the university system, and the future of the education system is a little worrisome (then again the future has always been worrisome and somehow we've muddled through).

But, while before you might have a few bad apples not providing sufficient feedback to students (or not doing it in a useful way) you have, as matters of policy, short cuts.

Why pay any attention at all to your students work when:

a) You can outsource checking for plagiarism to Turnitin or another similar service
b) A computer can do grading for you automatically. Never mind that it can't tell the difference between a right answer written a different way than in the answer key and a wrong answer.
c) An adjunct professor paid less than minimum wage can handle the actual teaching duties so the university can keep more of the students tuition.

Re:Quid pro quo ...Your moral obligation... (0)

Anonymous Coward | about 7 months ago | (#46886889)

The "rules" are the rules, be they right or be they wrong.

The objection you raise seems formally to be against the poster's ETHICS (... recognized rules of conduct...) rather than
"morality" (...according to an individuals' ideals and principles. ) http://www.diffen.com/difference/Ethics_vs_Morals [diffen.com]

You have no particular charter to decide what someone else's 'moral obligations' might be. I find this apparent presumption ...rather appalling.

(This is intended to be NOT sarcastic:) The above aside, as a please accept sincere appreciation for your effort to provide useful feedback to students.

Re: Quid pro quo (0)

Anonymous Coward | about 7 months ago | (#46886527)

Laziness is almost never the issue. Most instructors want to do excellent teaching. University administrators, on the other hand, want to reduce costs and increase revenue for each course, which means increasing class sizes. When hundreds or even thousands of students are registered in a course taught by just one salaried instructor, automated grading becomes nearly inevitable. Of course, the one salaried instructor has now been replaced in most cases by an overworked contract instructor making less than a basic living wage.

Re:Quid pro quo (3, Interesting)

clifyt (11768) | about 7 months ago | (#46886645)

As someone that wrote software like this -- and disagreed with the subject of the story a decade ago when he tried to get us with both the Gettysburg Address as well as Kennedy's inaugural address (both of which are GREAT speeches with historical value, but shitty college entrance exams) -- you are looking at this entirely wrong.

I can give you background of how these things are generally graded. 3 people get an essay, look at it for 30 to 45 seconds, throw a score and it and if they are all within a margin of error, they move on. If not, a senior rater comes in and and they can replace one other person and it is now within margin of error, they move on as well. If not, it is workshopped for 5 minutes.

In 99% of the cases, you have less than 2 minutes of viewing on your essay between 3 people.

Enter the computer...the raters are told they are going to be rated themselves. We can throw a lot more prerated essays that had been normed by a large group of raters, and train the rater. They know they are being measured and the average rater spends two or more minutes reading through these. You actually have MORE time with eyes on your essay with a computer rater involved than you do without. Having a computer rater doesn't remove humans -- it adds a safe guard. It means one person spends more time and is verified with something that is unbiased (within reason...actually was able to figure out subtle racism and otherwise that wouldn't have been detected with purely human raters...'black' or 'hispanic' names and scores go down...'asian' names and the scores go up...give the same essay with the names switched and the humans change ratings...the computer was actually more objective).

I haven't been involved with this sort of thing in a decade, and I can only assume it is much better than when I left my project...but lazy isn't the right word. Underpaid and overworked? Yeah...but not lazy.

Anonymous exams (2)

emilv (847905) | about 7 months ago | (#46887323)

Racism, sexism and other discrimination is quite effectively countered with anonymous grading. My university gave you a unique number before each exam and you put only that number on the sheets. Only afterwards did the administrators (not anyone involved in the course) look up and file the exam under your name. I found this helpful as a TA too because we really wanted to be fair both in grades and comments.

You can still be biased by the handwriting but we tried to counter that ourselves. If someone in my TA group recognized the handwriting of someone they knew we made sure to let someone else in the group grade that exam.

Babel? (0)

Anonymous Coward | about 7 months ago | (#46885811)

Anyone else notice the article used Babel instead of babble?

Re:Babel? (1, Informative)

geminidomino (614729) | about 7 months ago | (#46886073)

Reference to (Babel, Tower Of).

The story is a biblical "explanation" of why humanity, despite ostensibly originating as a single tribe, uses multiple languages.

Re:Babel? (2)

DoofusOfDeath (636671) | about 7 months ago | (#46886331)

Reference to (Babel, Tower Of).

The story is a biblical "explanation" of why humanity, despite ostensibly originating as a single tribe, uses multiple languages.

I could be wrong, but I think it's understood primarily as an allegory regarding man sinning(?) by aspiring to accomplish what only God can.

Pffft (0)

Anonymous Coward | about 7 months ago | (#46885837)

"News" sites have been doing this for ages. (Including \.)

student athlete need some like this with 60 hours (2)

Joe_Dragon (2206452) | about 7 months ago | (#46885871)

student athlete need some like this with 60 hours a week playing football they don't have time for class.

Can probably "write better" too (1)

penguinoid (724646) | about 7 months ago | (#46885895)

If they're using some stupid automated grader, odds are a computer-generated essay could consistently grade higher than any humans (because it can focus on scoring without worrying about content).

The answer: essay grader graders (2)

TsuruchiBrian (2731979) | about 7 months ago | (#46885955)

I don't see a problem with automated essay graders in principle. It's just that the current essay graders are no good. Once we are able to make computer software that can actually understand essays as well as a human it will be should be perfectly competent to grade an essay.

I certainly see the motivation to have a computer grade essays. Who wants to read multitudes of mediocre essays. I might rather be put in solitary confinement. I am all for the automated essay graders, but only after they can be proven to be as competent as a human.

I have no idea how to make a such a competent essay grader, but I do know how to grade an essay grader. You have a bunch of computer graders and human graders grading the same essays. If the computer graders show a more consistent performance than the humans (i.e. are the outlier less frequently), then the computer grader is better.

If a paper is scored by 4 human judges and a computer, and the humans score the paper 1, 2, 3, 4, and the computer scores the paper as a 9, then it means that according to most of the human graders, the computer was way off. Essays are inherently subjective. Are the humans right or is the computer right? Who cares it doesn't matter.

If a paper is scored by 4 human judges and a computer, and the humans score the paper 4, 5, 7, 9, and the computer scores the paper as a 6, then it means that according to every human grader, the computer did better than half the humans.

If a computer can do better than the humans even by human standards, then I think it's fair to say that a computer is good enough.

Re:The answer: essay grader graders (2)

clifyt (11768) | about 7 months ago | (#46886587)

I helped design one of these essay graders a decade+ ago with Dr. Ellis 'Bo' Page (Duke and MIT).

Even then, we were as good as humans in solely grammar and mechanics and all that sorta stuff. We were rating on a 6 point scale and something like 70% of the scores were a perfect match, and 85% were within 1 point.

Given that we were using professional human raters that were trained on weekly basis and had round tables to go over controversial papers, and these were considered some of the best in the US at their job...and that if you had 3 people rate the essay, take the mean score and ask the single human to rate it...they were at around 60% a perfect match.

Again, this was not for content...most college entrance exams are looking for your writing style and nothing else. If you can write well (and my writing on this site is not representative of my professional writing), you can research your material when you aren't writing content off the cuff and actually do well.

Re:The answer: essay grader graders (2)

TsuruchiBrian (2731979) | about 7 months ago | (#46887119)

If it becomes the case that writing style is able to be analyzed and produced by a computer algorithm, it seems to me that having a good writing style will become like having good arithmetic skills (i.e. less importance is placed on these skills as they become trivial for machines to replicate), and ironically this ability to automatically test and reproduce skills drives those very skills into obscurity.

It seems like the skills that computers can't do yet are the only ones that it is worthwhile for humans to do.

Re:The answer: essay grader graders (1)

Anonymous Coward | about 7 months ago | (#46886653)

As someone who does research in this area, I wanted to follow-up on your comments with a few observations, and to clear up some misconceptions.

1. There is actually a fair amount of empirical evidence at this point that these scoring systems do score as well as human graders, in the sense that the grades the assign tend to be as highly correlated with consensus grades by humans, as individual human grades are (if not more highly correlated). It's not even really a scientific question anymore as far as I'm concerned.

2. Having said that, the reason why they work as well as they do is because (as you point out) the human graders are incredibly rigid, and have very strict criteria to enforce fairness across different graders. There's many studies and investigations showing that the human graders often feel like they are giving poor grades because of the superficial criteria they have to assign (because if they don't, they will actually lower their levels of agreement and therefore increase variability due to grader). In this sense, if the human grading is superficial, why not have an algorithm or model do the grading? Similarly, if you want to complain, complain about the human grading, not the essay scoring program.

3. It's naive to think that you can't game the system regardless of scoring system. That is, regardless of whether you're using a human, or a algorithm, or a parrot or dolphin, or multiple-choice, or whatever, there will always be ways of superficially inflating your score. These people just seem to have identified the crack in the system; it will get patched, and then someone else will find a new one. That's just the nature of cheating on tests, or alternatively, of increasing your test-taking ability.

Re:The answer: essay grader graders (0)

Anonymous Coward | about 7 months ago | (#46886927)

Sure, that's all fine and good for the average essay. The problem is that once the human graders go away the random essay generator can be designed to specifically exploit the computer grader's evaluation criteria while completely ignoring human readability.

One human and a computer (1)

emilv (847905) | about 7 months ago | (#46887347)

The solution might be to have a human sanity filter checking semantics and throwing out gibberish, and a computer grader doing the fair grading.

Re:The answer: essay grader graders (0)

Anonymous Coward | about 7 months ago | (#46887261)

Once we are able to make computer software that can actually understand essays

John Searle would like to have a word with you: http://en.wikipedia.org/wiki/Chinese_room

If this argument is correct, it would mean that computers can't "actually understand" anything, essays included.

(Not that I agree with him - personally, I think the problem with his argument is he never gets around to defining what "understanding" Chinese means beyond the ability to produce the appropriate symbols in response to a query presented in Chinese. If you were an English speaker, how exactly would you learn to speak and understand Chinese, except by forming a mapping (or translation) between English phases and Chinese phrases?)

Works on Slashdot posts, too! (5, Insightful)

sootman (158191) | about 7 months ago | (#46885997)

Artificial intelligence, while seemingly tasty on the surface, tends to be underwhelmed by insufficient fish, with regard to warrantless searches.

Re:Works on Slashdot posts, too! (0)

Anonymous Coward | about 7 months ago | (#46886033)

Shit, you're right. Not enough fish has been the problem with Beta all along. With this, all our problems are solved! Thanks, sootman.

- the real honest timothy, scout's honor.

Re:Works on Slashdot posts, too! (0)

Anonymous Coward | about 7 months ago | (#46886041)

Bonus point: guess the three keywords used to seed the above post.

(My guess is: "AI", "fish sticks", "NSA")

Re:Works on Slashdot posts, too! (2)

mpe (36238) | about 7 months ago | (#46887145)

Artificial intelligence, while seemingly tasty on the surface, tends to be underwhelmed by insufficient fish, with regard to warrantless searches.

AI is HARD. Plenty of tasks which people can do easily are difficult to get machines to do, even throwing lots of processing resources at the problem.
Natural Language Processing is one of these difficult problems. With "grading essays" also being nowhere near beginner level NLP.
Quite possibly actual NLP experts would not attempt to write such software, because they understand exactly how difficult a task it is. (Similar issues apply to "Internet filtering software".)

Oh, those things. (1)

Anonymous Coward | about 7 months ago | (#46886045)

Whenever I had to do one of these I just copy and pasted the question or prompt a couple dozen times or until the length was good, never failed to work.

Essay grading:Dubious in the best of circumstances (0)

Anonymous Coward | about 7 months ago | (#46886057)

Unlike hard science or math problems, where the answer is either right or it isn't, essay grading is always subjective. And let's face it, the liberal arts majors who go on to become English teachers/professors aren't the brightest bulbs. They often fail to see the brilliance in an essay written by a student who's smarter then they are. If it doesn't appeal to their limited aesthetic, or fulfill all the checkboxes in some list in the Teacher's Guide, it gets a poor grade.

Re:Essay grading:Dubious in the best of circumstan (0)

Anonymous Coward | about 7 months ago | (#46886489)

Unlike hard science or math problems, where the answer is either right or it isn't, essay grading is always subjective. And let's face it, the liberal arts majors who go on to become English teachers/professors aren't the brightest bulbs. They often fail to see the brilliance in an essay written by a student who's smarter then they are. If it doesn't appeal to their limited aesthetic, or fulfill all the checkboxes in some list in the Teacher's Guide, it gets a poor grade.

Still bitter about your poor grades in English, eh?

Obligatory Offal (4, Interesting)

TheRealHocusLocus (2319802) | about 7 months ago | (#46886085)

Auto Comment (0)

vortex2.71 (802986) | about 7 months ago | (#46886233)

This and all of my slashdot comments were generated by an automated commenter and I always get modded up!

Re:Auto Comment (1)

chrism238 (657741) | about 7 months ago | (#46887153)

Well, not this time buddy!

Re:Auto Comment (0)

Anonymous Coward | about 7 months ago | (#46887297)

You are missing one or two commas. Or is this just to trick me into believing you are a real person!?

Re:Auto Comment (0)

Anonymous Coward | about 7 months ago | (#46887307)

I woulda probably gone with

"This, and all of my Slashdot comments, were generated by an automated commenter. And I always get modded up!"

so two commas, a period, and a capital S. You almost fooled me, realistic computer text

I have heard the machines singing... (1)

Nova Express (100383) | about 7 months ago | (#46886383)

Each to each.

I do not think that they will sing to me....

Is the essay generation software available? (1)

mark-t (151149) | about 7 months ago | (#46886501)

[nt]

Re:Is the essay generation software available? (2)

bobjr94 (1120555) | about 7 months ago | (#46886767)

I have checked a bunch of websites and some searching and found no link to this babel generator or even a small excerpt from the submitted paper. I would have expected at least one if not both to be easily found.

This would explain... (0)

Anonymous Coward | about 7 months ago | (#46886683)

Bennett Haselton!

"The essays, though grammatically correct and structurally sound, have no coherent meaning"

Looks like many comments on political forums (1)

Animats (122034) | about 7 months ago | (#46886861)

Example from article: "Privateness has not been and undoubtedly never will be lauded, precarious, and decent.". There are too many comments on news sites which read like that.

Re:Looks like many comments on political forums (0)

Anonymous Coward | about 7 months ago | (#46887289)

That is largely from character flaws and drug problems. If your speech can't be parsed, it makes poor propaganda. (you could argue that it is simply bypassing filters/mods while drowning out true discussion, but "people be crazy" seems like the more obvious explanation)

BS is the key to a BA or BS (1)

Anonymous Coward | about 7 months ago | (#46887275)

But, shockingly, intelligent students can just write any old random bullshit and get an equally good or better score. (the "bullshit" part might prove the article's point, except of course human-written bullshit works on people and auto-grading software)

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?