×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Automated System Developed To Grade Student Essays

samzenpus posted 1 year,18 days | from the machine-learning dept.

Education 253

RougeFemme points out this story at the Times about software that can be used to grade student essays and offer almost instant feedback. "Imagine taking a college exam, and, instead of handing in a blue book and getting a grade from a professor a few weeks later, clicking the 'send' button when you are done and receiving a grade back instantly, your essay scored by a software program. And then, instead of being done with that exam, imagine that the system would immediately let you rewrite the test to try to improve your grade. EdX, the nonprofit enterprise founded by Harvard and the Massachusetts Institute of Technology to offer courses on the Internet, has just introduced such a system and will make its automated software available free on the Web to any institution that wants to use it. The software uses artificial intelligence to grade student essays and short written answers, freeing professors for other tasks."

cancel ×
This is a preview of your comment

No Comment Title Entered

Anonymous Coward 1 minute ago

No Comment Entered

253 comments

This is horrid (5, Insightful)

swm (171547) | 1 year,18 days | (#43364089)

One of my kids had something like this: not for English, but for physics.
The teacher couldn't be bothered to assign and grade proper homework.
Instead, he fobbed the kids off onto a web app.
- go to the site
- get a problem
- solve the problem
- type in the numerical answer
- right answer? go on to the next problem
- wrong answer? try again
The web app allowed maybe 0.5% margin for rounding error, and you got 5 tries before it failed you on that problem.

It sounds reasonable in the abstract, but in practice it was utterly wretched.
All learning is, at some level, an interaction--a conversation--between student and teacher.
Even if it is nothing more than a red check mark or a red X on a homework paper,
you have communicated some thing to some person and gotten some response.
You don't realize how important this is until it is gone.

With nothing but a machine to talk to, it stops being about learning.
It is just about satisfying the machine by whatever means necessary.
In his rage and frustration my son told me that the easiest way to solve the problems was to copy and paste the problem text in to google.
This would reliably return the general formula for solving that problem;
plugging in the numbers that the web app had generated for your instance of the problem would then yield the correct answer.
By the end of the school year, I was telling him that if he didn't want to deal with the web app, he should use google to get his grade,
and if he wanted to learn physics, I would teach it to him.

Automated essay grading is going to be even worse.
There is no point writing prose unless a human is going to read it.
When I want to talk to machines, I write code.

Writing songs, that voices never shared...
-- Paul Simon

Our Machine Overlords (0)

dragon-file (2241656) | 1 year,18 days | (#43364341)

Before computers, teachers actually had to do things by hand. But computers made things easy so we gave every teacher a computer.

Even that was to hard and so teachers turned to AI to grade their students papers.

Soon going into work was to hard (especially with hangovers from all the free time spent partying) and so they made robots to give lectures in their place.

Then the students couldn't be bothered to come in so they just started recording the lectures and making them available online.

Then learning things was to labor intensive and for that matter so was work so we let robots and computers do everything.

The year is now 2635 and we spend our days watching Jersey Shore season 628. Obesity is the new norm and from birth every waking moment is 'free time' but life is better. Right?

Re:This is horrid (5, Interesting)

Anonymous Coward | 1 year,18 days | (#43364537)

I went through the same system and it taught me all sorts of useful things unrelated to my actual physics curriculum, like
1/2 != 2/4
0.5 != 1/2
x != x+1-1
x^2 != x*x

Re:This is horrid (4, Interesting)

reve_etrange (2377702) | 1 year,18 days | (#43364557)

I had the same experience in university calculus and physics. Even for problems with one right answer, there are typically many (even infinite) ways of expressing that answer. Even something as advanced as Mathematica or Maple can be fooled, and the websites in question are no Mathematica.

Re:This is horrid (0)

Anonymous Coward | 1 year,18 days | (#43364759)

Even something as advanced as Mathematica or Maple can be fooled, and the websites in question are no Mathematica.

Which is really just pure laziness on the part of the designers of the system. They could have used Maxima which is probably about as good (or at least, not a lot worse than) Mathematica or Maple for simplifying expressions and is open source.

Re:This is horrid (0)

Anonymous Coward | 1 year,18 days | (#43364819)

These have gotten better over time - WebAssign can now handle symbolic answers and manipulate numerical answers into canonical forms for comparison. Depending on the level, students are often expected to express answers in simplest form, so the grandparent's 2/4 as 1/2 and x+1-1 as x would not be acceptable. Similarly, if asked for a polynomial, x^2 and x*x really would be different things. There are great horror stories for math/science implementations, but the worst of them were from early versions from a decade ago. Similarly, this solution may not be great initially, but probably comparable to AP rubric grading where you check for whether certain items are included, rather than a holistic quality. Give it another decade and this could be a great system.

Re:This is horrid (1)

TapeCutter (624760) | 1 year,18 days | (#43364809)

Maths is all about simplification. Examples 1, 3 & 4 are poor answers because they haven't been fully "simplyfied" (which is more than just a character count), example 2 doesn't have enough context but it could be "correct" in the context of a lesson about decimal points. OTOH I doubt that was the logic behind the software's answer.

Re:This is horrid (3, Insightful)

TsuruchiBrian (2731979) | 1 year,18 days | (#43364925)

Math is about simplification, but simplicity is subjective.

1/2 is simpler than 2/4, but not if you have something like this: 2/4 * a + 3/4 * b + 1/4 * c

maybe all this would be simpler as: (2a + 3b + c)/ 4 or maybe not it depends on the application...

Re:This is horrid (0)

Anonymous Coward | 1 year,18 days | (#43364653)

yeah, been there with my HS kid at university courses. Profs outsource the homework to... the publisher of the BOOK! (It's another story about specialized university editions that kill the used textbook market) Grad student TA's don't even have to do anything anymore. I remember one particular instance of her typing in an answer of 123,456 it was the correct solution... but the computer saw it as two numerical entries eg 123 and 456 after a half hour I told her to fork on it.... cmon, even I could parse that much 20 years ago in high school. At the end of the course there was a note from the TA that said "were told not to worry about our TA assignments and focus on our own studies"

This model is horrid, a completely wrong answer gives the same feed back as one that is rounded improperly.

I remember another instance when trying to get the teachers up to speed with online portion of the new math curriculum at the high shcool (new text books get adopted every so many years)... I read all the manuals and things were still not working properly, so I actually called... Support agreed quickly that there was a problem... the first thing they asked for was the usernames and PASSWORDS for the teacher accounts involved. Refused any help when I didn't provide account passwords.

Follow the dollars in the end, universities are big businesses as well. One has to wonder what their purpose will be in a short while if online education (or more likely qualification) really takes hold.

Re:This is horrid (2)

sm284614 (946088) | 1 year,18 days | (#43364687)

There's nothing wrong with using automated marking where it's actually saving time, as long as the results are used to further learning. The simple idea (employed by the Khan academy and others) is to get students/pupils to practise at home, a system tracks what they can do and what they struggle with, and the teacher-pupil/student-professor contact time is used to address any issues. Grading simple, repetitive tasks is a waste of time if it can be automated; it's the feedback that matters. I assume that this paper marking thing is able to generate reasons for the grades it gives, and some of that is probably also a little useful.

Re:This is horrid (3, Informative)

RougeFemme (2871421) | 1 year,18 days | (#43364691)

I'm currently tutoring my daughter in statistics for the same reason. She's in college and while she's flipping through her homework appliication and her e-textbook, I'm flipping through my old statistics books, plus a couple of study guides I picked up. Also, sometimes the homework application is simply wrong. (Doesn't every tool/program have at least one bug?) My sister, a teacher, uses one - mandated by the community college where she teaches. Occasionally, she has to override the application so that she can mark correct problems that the application marked wrong. The students alert her, she checks and then overrides when the application is clearly wrong.

Re:This is horrid (0)

Anonymous Coward | 1 year,18 days | (#43364839)

Is this really unique to technology though? Even if you have a 99.9% accuracy rate in grading, you are likely to miss mark one question on one student's paper per class each week or so.

Re:This is horrid (0)

Anonymous Coward | 1 year,18 days | (#43364693)

I agree on the essay grading; I haven't seen anything suggesting that real NLP is applied to actually ensure the essay even makes sense, which is a rather low bar for grading an essay. On the other hand, as you say, it seems like computer grading should work for physics.

As a student, the system you describe sounds terrible. As a researcher vaguely involved in those technologies (on the problem generation side, not the UI side, although they certainly interact), I am wondering how those problems can be solved. Obviously, the system you describe didn't teach much (I guess your son had to know enough to figure out how to plug in the numbers), so it failed at its intended purpose.

You suggest the problem is the lack of communication. I assume there is no mechanism for partial credit or otherwise identifying where the student went wrong due to the student only being given a chance to enter their final solution. Along with being able to grade answers where a student shows their work, there is work on modeling incorrect variants of the correct procedure in order to be able to figure out which incorrect procedure the student followed (which in physics/chemistry would probably often including rounding to different amounts in different places, which may or may not be considered incorrect), which then can be used to give proper feedback to the student either in the form of an explanation of why their answer is different from the correct answer or in the form of a simpler problem for them to solve that isolates the error.

Of course, giving useful feedback is quite different from grading understanding. The 5 retries system you mention sounds like it was designed to ensure the student would try 3 or 4 times at most before cheating like you describe, but zero retries seems overly strict for grading (and infinite retries makes no sense for grading). The real solution is to not grade homeworks on correctness because that's silly; give the students tests in class to find out if they actually understand the material: homework is for learning the material.

Better than you think (3, Informative)

Roger W Moore (538166) | 1 year,18 days | (#43364739)

It sounds reasonable in the abstract, but in practice it was utterly wretched.

No, the abstract does not sound reasonable: as with most things online you can always find bad ways to do it. I'm a physics prof working as part of a team to develop an open source, algebra capable question and content system. However even the current capabilities of something like Moodle [moodle.org] (which is Open Source) is far in excess of what you describe. You can type in multiple "answers" to a problem and have the student get feedback and a partial grade if they get the problem wrong in a way that you managed to guess. Obviously if they find a new way to get it wrong then they will not get feedback though.

Commercial systems go even further with the student having the option to click on a help button which can break the question into steps for the student to complete in rder to guide them through to the right answer. This can be configured to give a grade penalty at the choice of the instructor - this is one of the features we want to add to an Open Source solution.

However even with current Moodle capabilities you can build a system that, I would argue, is better pedagogically for many physics problems (those with numerical or symbolic responses) than paper-graded assignments because, with an online system with some feedback and multiple attempts the student is encouraged to keep trying until they figure out how to get it right. This encourages them to think out the solution themselves whereas with a paper assignment they get one try and are then given the answer. To make this work though you need some means for students to come and talk to you and/or TAs to provide some help towards getting the right method. So you still need the student-teacher interaction but computers can provide a first line of contact and so let a teacher help more students.

That being said I find it exceedingly unlikely that this EdX system can work for written responses beyond checking that their english is good. For physics how can it possibly know that the statement "the Higgs boson has a mass of 140 GeV/c2" is wrong and "Dark Matter does not interact with photons" is correct? To be able to grade it will have to know a huge amount of information about a massive range of topics - and looking this stuff up on Google is not an option given all the crazy people and their wacky physics theories which they stick on a web page.

Re:This is horrid (0)

icebike (68054) | 1 year,18 days | (#43364757)

One of my kids had something like this: not for English, but for physics.
The teacher couldn't be bothered to assign and grade proper homework.
Instead, he fobbed the kids off onto a web app.

Something like what you described is indeed no more than filling out yet another web page, hoping to win a prize.

That doesn't mean that grading compositions would be the same, since by their very nature they have to accept human language in and assign a grade to it by some hocus pocus. The game becomes figuring out the limits and parameters of said hocus pocus. Allowing them to revise their paper is a way of training them on the dimensions of said hocus pocus, and probably not all that useful to either the student or the system.

I suspect that after a couple hundred thousand compositions are graded, everything looks like plagiarism. And I suspect that is part of the process here.

Re:This is horrid (1)

sgt scrub (869860) | 1 year,18 days | (#43364785)

That sounds like he has a pretty good teacher. If it were me, I'd be fine with the link Google produced. :P

Re:This is horrid (1)

TsuruchiBrian (2731979) | 1 year,18 days | (#43364899)

In his rage and frustration my son told me that the easiest way to solve the problems was to copy and paste the problem text in to google. This would reliably return the general formula for solving that problem

Isn't that what physics is? Applying the right formula? I don't think much is gained by having students memorize formulas anyway. In real life you just look up general formulas on google/wikipedia/wolfram and apply them to your specific problem. I suppose it's useful to be able to derive certain formulas as a method to gain greater insight into patterns and ways of thinking, but I don't think this depth is commonly required in high school.

If it were my kid, I'd rather he knew how to google formulas and apply them to specific problems than have a bunch of formulas memorized. People naturally memorize things if they use them enough. Rather than having kids memorize specific formulas, I think it's more useful that they memorize the best way to find specific formulas (i.e. google, wolfram, etc). That's much more efficient.

I have a bunch of physics formulas memorized because I used them a lot. The ones I don't use a lot I just look up when they are needed. If I forget one, it's not a big deal. If I forget how to look up formulas (not sure how that would happen) I would be screwed.

Re:This is horrid (1)

Shavano (2541114) | 1 year,18 days | (#43365005)

Isn't that what physics is? Applying the right formula? I don't think much is gained by having students memorize formulas anyway. In real life you just look up general formulas on google/wikipedia/wolfram and apply them to your specific problem. I suppose it's useful to be able to derive certain formulas as a method to gain greater insight into patterns and ways of thinking, but I don't think this depth is commonly required in high school.

If it were my kid, I'd rather he knew how to google formulas and apply them to specific problems than have a bunch of formulas memorized. People naturally memorize things if they use them enough. Rather than having kids memorize specific formulas, I think it's more useful that they memorize the best way to find specific formulas (i.e. google, wolfram, etc). That's much more efficient.

I have a bunch of physics formulas memorized because I used them a lot. The ones I don't use a lot I just look up when they are needed. If I forget one, it's not a big deal. If I forget how to look up formulas (not sure how that would happen) I would be screwed.

The purpose of EDUCATION is to teach the student how to understand and think about the material.

They'll work the bugs out (3)

rsilvergun (571051) | 1 year,18 days | (#43364937)

Face it, we're all going to get replaced by Expert Systems. They talked about this in the 80s and you didn't believe. 95% of us follow pretty simple patterns. There's damn little that most of us can do that a machine can't. Sure, there are exceptions. But most of us don't qualify, we just think [xkcd.com] we do.

You're being replaced. The real question is how are you going to deal with it? What do we do when 95% of us are completely unnecessary?

Re:This is horrid (1)

EvanED (569694) | 1 year,18 days | (#43364973)

When I want to talk to machines, I write code.

I've thought a bit about how I would like to grade a typical upper-division project-based CS course, like your typical OS class, where there are assignments of the form "okay, now go program such-and-such". (As opposed to theory courses, or more open-ended project classes where the students can pick what they want to do or different people do different projects.)

What I'd kind of like to do is have a submission server which would do three things that each contribute to the final grade. First: before the submission deadline, people could submit at any point. It would build the project and run a set of tests which was released to the students, reporting full results and that portion of the grade. (Students could submit multiple times without penalty, and it would use either the final or highest submission. Not sure which.) Second: before the submission deadline, it would do the same thing with a set of hidden tests, except not report information about what the tests were, just giving a number or letter grade. If students can't figure out what they're doing wrong, they could come talk to me. Third: after the submission deadline closes, it would run another set of secret tests to get the third component of the grade, and report it to the students at that time. In addition, there would be a fourth portion of the grade which was not automated. I'm not totally sure what the mix would be between them.

(I've also thought about making students submit enough test cases to get good coverage on either their or my solution or both, which would be a fifth category.)

Any thoughts?

Re:This is horrid (0)

Anonymous Coward | 1 year,18 days | (#43364979)

Who even grades homework?
All my teachers ever did was check if you'd actually done it, if that.
Maybe they'd have you write one of the problems on the board.
Homework is there to help you.

Re:This is horrid (2)

Scarred Intellect (1648867) | 1 year,18 days | (#43365067)

I've got that for Chemistry AND Physics. It's horrible. I've entirely missed problems because I couldn't figure out how it wanted the answer represented.

I mirror your sentiment exactly:

The teacher couldn't be bothered to assign and grade proper homework.

The kicker is that teachers who DO assign real homework have TA's or graders to grade the homework, all they actually grade are our tests.

Oh yea, and I have to pay extra for this online bullshit. It's required in a more in-depth way than textbooks (I literally can't pass the course without it) but isn't paid by the school, it's paid by me alongside my already-ridiculously overpriced textbook.

Yes, I have to pay for the teacher to be lazy.

My TA had that 35 years ago (5, Funny)

gewalker (57809) | 1 year,18 days | (#43364107)

Take one lab report for Fluid Mechanics, measure the thickness with a micrometer -- look up the grade on the curve.

Re:My TA had that 35 years ago (1)

mrchaotica (681592) | 1 year,18 days | (#43364531)

Did he at least flip through to make sure there weren't a bunch of blank pages in the middle?

Re:My TA had that 35 years ago (0)

Anonymous Coward | 1 year,18 days | (#43364637)

This page seditiously left blank.

Re:My TA had that 35 years ago (0)

Anonymous Coward | 1 year,18 days | (#43364879)

Use thicker paper?

Re:My TA had that 35 years ago (1)

camperdave (969942) | 1 year,18 days | (#43364897)

Take one lab report for Fluid Mechanics, measure the thickness with a micrometer -- look up the grade on the curve.

I guess that beats throwing them down a flight of ten stairs twice.

Re:My TA had that 35 years ago (0)

Anonymous Coward | 1 year,18 days | (#43365013)

Hell, when I was a TA I knew of an even simpler method employed by some of my colleagues: give everyone a 100. It's fair and takes less time than measuring with a micrometer.

Alternatively, as actually happened in one case that I found out about after it was far too late to do anything (and, ironically, in an engineering ethics class), one of my fellow TAs didn't bother grading the final essays, instead just averaging each student's previous essay scores, ranking those averages, and then distributing scores between 90 and 100 along a bell curve on which those rankings sat. I was rather pissed when I found out about that one much later, though the students were glad from what he had said, since all of them received grades that were higher than what they had been averaging.

freeing professors for other tasks (-1)

Anonymous Coward | 1 year,18 days | (#43364115)

Knowing a good number of male professors, this would be mean more time to seduce coeds!

AI has not come far enough for this (0)

Anonymous Coward | 1 year,18 days | (#43364145)

To actually grade an essay well you need to analyze its logic as well as its sentence structures. Computers cannot do this yet! Human writing assumes all sorts of "common" knowledge in order to convey a point concisely, in addition to omitting a completely detailed logical argument to leave out obvious steps (often steps which are related to the aforementioned common knowledge). It could be done one day, but that day is not today.

In the meantime, expect these systems to be gamed pretty hard.

Re:AI has not come far enough for this (5, Interesting)

dgatwood (11270) | 1 year,18 days | (#43364381)

Computers suck at even the most basic grammar checking. I once decided to try a bunch of online grammar checkers to see if they would be useful at providing a sanity check for my novels. I concluded that they report so many bogus mistakes that it simply wasn't practical to use their output at all. To test them, I fed them a block of content, some with intentional errors that the grammar checker should have caught, others with deliberately (or accidentally) tricky bits that should not have produced any errors.

  • Upon seeing that, Joseph resolved to stop. Several grammar checkers thought "seeing that" was used idiomatically, and suggested replacing it with because. Upon because, Joseph resolved to stop. Yes. Much better.... Oh, and some others suggested that "Upon" is archaic.
  • “Time to impact: seventy-six hours, fifteen minutes, twelve seconds,” the computer intoned. Oddly, several checkers suggested that "twelve seconds" was a fraction and should be hyphenated. Ugh.
  • It's simple, really. There must be some mistake. Several spell checkers suggested "their". Others said that "must be" is passive voice. Uh, no, not every use of "to be" is passive construction.
  • This isn’t your class anymore. Some checkers reported an agreement problem with "class". Huh?
  • The room was dark, its plant-covered landscape shimmering green in the light of their headlamps. At least one checker suggested replacing "in the light of" with "considering". Eek!
  • Joseph climbed up first. Several spell checkers suggested that "climbed up" is redundant. Apparently, their editors have never climbed down something.
  • One checker even called "chided" archaic, but did not comment on the highly offensive swear word that I placed elsewhere in the sentence.

And so on. Heck, my phone doesn't even know the difference between "its" and "it's" and tries to auto-correct me into looking like I failed first grade English. And these folks expect me to believe that computers can feasibly help students learn to write better papers? Give me a break. Maybe in thirty to fifty years (*) we'll get there, but....

* Which many grammar checkers would probably suggest changing to "thirty-two fifty".

Re:AI has not come far enough for this (0)

camperdave (969942) | 1 year,18 days | (#43364961)

Did any pick up that the comma between "dark" and "its" in the fifth example should really be a period?

also short answers?? How does it know the answer? (1)

Joe_Dragon (2206452) | 1 year,18 days | (#43364783)

also short answers?? How does it know the answer? and what if there is more then 1 answer / a wide range of part answers.

Also what about questions on the level of how do you do X in X os and you fail as the way you put down to get to control panel is not the way the gradeing software is looking for.

Have a computer write your submission too (5, Interesting)

hawguy (1600213) | 1 year,18 days | (#43364161)

Seems like it's a small step from this to having computer algorithms that automatically write your paper for you too - then you can let it go through thousands of submit-edit-submit cycles until the scoring computer gives you a perfect score.

Kind of like the guys that came up with software to generate nonsense scientific papers and actually had a few accepted [wikipedia.org] at conferences and journals.

Re:Have a computer write your submission too (0)

Anonymous Coward | 1 year,18 days | (#43364747)

Or at the very least, you could crank out a bad paper really quickly, and then let the machine evolve it for a while until it gets a high score on the marking algorithms.

Re:Have a computer write your submission too (2)

lgw (121541) | 1 year,18 days | (#43364963)

Much of modern finance actually works this way. Press releases about mergers and such are written by computer to be read by computer and fed into algorithmic trading. The relationship to nonsense is left as an exercise for the student's computer.

Sample Admittance Essay (4, Funny)

milbournosphere (1273186) | 1 year,18 days | (#43364179)

Why I want to goto Harvard By P Q Student Up up down down left right left right B A

Congratulations, you have been admitted... (3, Funny)

Anonymous Coward | 1 year,18 days | (#43364265)

And you have been awarded 2500 extra HP.

feedback... (5, Insightful)

retchdog (1319261) | 1 year,18 days | (#43364191)

``Your grade is C. To improve your grade in the future, you need to do the following:

use 25-30 words per sentence; include more words from the wordnet entry for the topic of your essay; avoid simplistic or run-on sentences as measured by number of noun and verb phrases detected by our proprietary NLP tokenizer.

As a helpful reminder, our preparatory guides are available as a subscription service and include 100 practice submissions per week; only $29.95 per month."

Re:feedback... (1)

JMZero (449047) | 1 year,18 days | (#43364833)

If their use was kept secret, systems like this would likely perform well most of the time; the correlations these systems are based on are probably pretty steady.

Once students get any information about the system, however, it's doomed - and in any case it's unlikely the system will give real, useful assistance in improving skills beyond what you'd get from a grammar checker.

Grading is about feedback (5, Insightful)

FailedTheTuringTest (937776) | 1 year,18 days | (#43364193)

Grading is not, or should not be, about the grade, it should be about the feedback that the lecturer gives to the student. Even if the computer can grade an essay well (which I remain to be convinced of, although I am sure I will soon have the chance to test it for myself), there is no claim made about the computer giving useful advice to the student. Can a computer explain how to refine a research question or structure an argument? Sadly, many lecturers don't in fact give good feedback, but we should be looking for ways to enable lecturers to give better feedback, not accepting poor feedback as the norm.

"ways to enable lecturers to give feedback" (0)

Anonymous Coward | 1 year,18 days | (#43364319)

You're absolutely correct, this should be the goal.

Re:Grading is about feedback (2)

DragonWriter (970822) | 1 year,18 days | (#43364323)

Even if the computer can grade an essay well (which I remain to be convinced of, although I am sure I will soon have the chance to test it for myself), there is no claim made about the computer giving useful advice to the student.

Since being able to grade well requires the ability to make the exact same distinctions required to identify the feedback that would need to be given, I would be very surprised if software that could do one could not also do the other.

I'd also be surprised if current software was able to do either.

Re:Grading is about feedback (2)

RougeFemme (2871421) | 1 year,18 days | (#43364711)

Grading is about the grade. Learning is about the feedback. Unfortunately, more and more, the educational experience is about the grade or standardized test score rather than learning. . .and learning to love learning. . .and learning how to learn. Kids don't have to show how their work in math anymore; all the teachers care about is the answer. We shouldn't be surprised at this latest development - well, not too terribly surprised.

Re:Grading is about feedback (2)

Roger W Moore (538166) | 1 year,18 days | (#43364811)

Grading is not, or should not be, about the grade, it should be about the feedback that the lecturer gives to the student.

Not always: there are two types of assessment [learnalberta.ca]. Formative assessment where the aim is to let the student know what they understand and what they need to work on. This is what you describe. There is also Summative Assessment where the aim to to assess what the student actually knows. Usually I try to get some of both - for example although a midterm is mainly aimed and finding out how much the students have learnt I'll also spend a lecture to go through the exam to give detailed solutions and feedback to students so they know what they did not understand and can learn from the experience. However since final exams occur after the end of the course these are pretty much entirely assessing a student's knowledge and have little educational value (although I post usually solutions online and will explain things to students if they come and ask).

New market for SEO companies (1)

Narrowband (2602733) | 1 year,18 days | (#43364221)

Watch as we move from "search engine optimization" to "grading engine optimization," as students look for AI solutions to write their papers, freeing them for other tasks.

The death of teaching. (0)

Anonymous Coward | 1 year,18 days | (#43364227)

I've before argued that most instructors today could be replaced with a video for lecture, a FAQ database for questions and a guy with a stopwatch for tests. I used to say proctor, but that implies the individual with the stopwatch is educated and I don't believe this is necessary anymore than a McDonald's fry jockey.

Anyway, this technology just makes it more true. God help you if you're just a little different from the majority somehow.

*sigh*

Not just ivory, but pointless too (0)

Anonymous Coward | 1 year,18 days | (#43364237)

And just what, pray tell, are the goddamn professors supposed to do at university? Collect honorariums and edit their book?

Re:Not just ivory, but pointless too (2)

DragonWriter (970822) | 1 year,18 days | (#43364337)

And just what, pray tell, are the goddamn professors supposed to do at university?

The same as what they get hired and paid for now, research.

You notice the common statement is "publish or perish" not "educate or expire", right?

Re:Not just ivory, but pointless too (1)

mark-t (151149) | 1 year,18 days | (#43364923)

I thought professors were hired to... you know... teach stuff.

If a professor isn't actually giving feedback to the students about their performance, then a person might as well just try to learn everything from books and not bother with classes at all.

Re:Not just ivory, but pointless too (1)

MightyYar (622222) | 1 year,18 days | (#43364789)

Teach more students? Current university education certainly does not scale very well.

They had these back in 1991 too (3, Interesting)

GoodNewsJimDotCom (2244874) | 1 year,18 days | (#43364239)

My friend wrote a story about his cat that was grammatically correct,and used big words, but made little to no sense. The auto-grader program told him he was approaching PHD level English. So he took his paper into school and showed it to the English teachers who reviled at it. He was like,"Show's what you know, the computer told me I'm university level."

Re:They had these back in 1991 too (0)

Anonymous Coward | 1 year,18 days | (#43364333)

Ah, that must have used the Flesch-Kincaid tests. I had a professor in college who used to required that you post the results of those at the end of your paper and would base a good chunk of your grade on them. What a waste of time that class was as more and more people discovered the literal magic formula for a good grade.

Re:They had these back in 1991 too (2)

DragonWriter (970822) | 1 year,18 days | (#43364343)

While I am rather skeptical about the quality of AI essay grading, I'd be very surprised if software in this area hasn't advanced since 1991. I mean, software in most other domains certainly has.

Re:They had these back in 1991 too (1)

GoodNewsJimDotCom (2244874) | 1 year,18 days | (#43364461)

One thing is for certain, it won't be able to understand context of the article. For AI to understand what you're talking about, it basically needs to be able to imagine it. For a computer to imagine stuff, it'd need a 3d model of the situation and a huge library of nouns, verbs, adjectives, adverbs. The day we have true AI, I might think we'll have the capabilities of natural language interpretation. Until then, the best it can do is weigh sentence structure "quality".

Some Things Never Change (3, Insightful)

skywire (469351) | 1 year,18 days | (#43364243)

Every era has its snake-oil salesmen and their marks. Sadly, in this case it will not be the customers who suffer, but their hapless students.

What, what? (1)

Anonymous Coward | 1 year,18 days | (#43364261)

Professors need more time for other things now? I'm sick of hearing this "for other things" crap we keep pushing. The reality is we're trying to push them out of their jobs via automation. The "other things" you're referring to include "standing in line for your welfare check" and "filling out job applications".

Part of the "joy" of grading is that you get feedback on how effective your teaching approach is for a given class. If you take the grading process away from the professor then how the heck are they supposed to evaluate their own performance? Not to mention the fact that the system can be gamed. Now we also get multiple tries at a test as well? The educational establishment really has gone all the way to fuck and buggery. The only good thing I can see in this is that it makes the grading more consistent. That's good...because people aren't different, no no no. No, all people are the same. You fill their heads with the same words and you'll get the same exam solution. Probably even the same handwriting. We should just make all of human existence a multiple choice question - the answers can be "Pay tax" and "Go to jail".

Computers have solved zero problems here, and they've created far more. This software will be written, casually proved to be barely adequate, and then shipped off to India for maintenance. We are now taught and graded by an underpaid maintenance programmer in India. Congratulations.

*facepalm*

Re:What, what? (1)

DragonWriter (970822) | 1 year,18 days | (#43364383)

Professors need more time for other things now?

Well, yeah, research, which is what they mostly get paid for. (Though, really, at many universities, the people who this is freeing up to do more of the primary focus—again, research—is likely to be more often graduate students working as graders/TAs, not professors.)

Grades grammar not content. A.I. not ready yet. (5, Informative)

doug141 (863552) | 1 year,18 days | (#43364275)

"A director of writing at MIT Les Perelman says that because these robo-graders work according to an algorithm, it is not hard to find out what it values and thus beat the system. He found that if you write long essays with big words, even if they are nonsensical, you will score high. The algorithm does not like short sentences or paragraphs or sentences that begin with ‘and’ or ‘or’ nor is it enamored of sentence fragments. In other words, all the little rules that good writers will break to create a particular effect will cause your essay to be marked down.

Perelman gives an example of how you can get a high score. The most interesting feature of the algorithm is that it doesn’t care about substance or even truth. It will ignore such trivialities as saying that the war of 1812 began in 1945, provided you say it grammatically. The substance of an argument doesn’t matter, he said, as long as it looks to the computer as if it’s nicely argued.

For a question asking students to discuss why college costs are so high, Mr. Perelman wrote that the No. 1 reason is excessive pay for greedy teaching assistants. “The average teaching assistant makes six times as much money as college presidents,” he wrote. “In addition, they often receive a plethora of extra benefits such as private jets, vacations in the south seas, starring roles in motion pictures.”

E-Rater gave him a [top score of] 6. He tossed in a line from Allen Ginsberg’s “Howl,” just to see if he could get away with it. He could."

http://freethoughtblogs.com/singham/2012/05/03/how-to-fool-a-computer-grader/ [freethoughtblogs.com]

Re:Grades grammar not content. A.I. not ready yet. (1)

dantotheman (2887483) | 1 year,18 days | (#43364345)

Where the hell are teaching assistants receiving...

extra benefits such as private jets, vacations in the south seas, starring roles in motion pictures

???

Re:Grades grammar not content. A.I. not ready yet. (0)

Anonymous Coward | 1 year,18 days | (#43364447)

I'm pretty sure that was sarcasm and was meant to show how the algorithms don't actually do any factchecking :)

Re:Grades grammar not content. A.I. not ready yet. (1)

RobKow (1787) | 1 year,18 days | (#43364453)

The point of that sentence is that it is obviously counterfactual, yet E-Rater gave him a top score for the insightful essay in which it was contained.

Re:Grades grammar not content. A.I. not ready yet. (0)

Anonymous Coward | 1 year,18 days | (#43364527)

"The most interesting feature of the algorithm is that it doesn’t care about substance or even truth."...

Well there you go - most university faculty do not care about substance or truth either, so in this respect I suppose the algorithm is pretty accurate.

Re:Grades grammar not content. A.I. not ready yet. (1)

dwhitaker (1500855) | 1 year,18 days | (#43364655)

But this is precisely why it makes this system useful AND should please teachers. If the system works well for grammar and ( hopefully programmable) essay structure, then the teachers can focus on the content, style, and finer points of writing. A computer can correct to/too/two, and if it frees up more time for the expert that is being paid to grade in depth, this is a good thing.

Re:Grades grammar not content. A.I. not ready yet. (1)

RougeFemme (2871421) | 1 year,18 days | (#43364723)

Reportedly, long essays with big words also help your SAT writing score. And in my state - which shall remain nameless - it definitely seems to help on the standardized writing tests.

Re:Grades grammar not content. A.I. not ready yet. (1)

Lemmeoutada Collecti (588075) | 1 year,18 days | (#43364745)

So all you really need is x (where x matches the length of the assignment) occurrences of the word buffalo, with the first one capitalized and the last followed by a period. A grammatically perfect sentence, with the maximum possible length results.

Any computer can be beat (1)

alen (225700) | 1 year,18 days | (#43364281)

Any algorithm can be beat
The all you have to do is fill your paper full of the right keywords

Re:Any computer can be beat (0)

Anonymous Coward | 1 year,18 days | (#43364857)

College is exactly like this. That's all that matters in every single interaction you have. Use the right keywords, you pass. These days people have no use... Oh what's the right word again, yeah, "time" for questions or reactions. Soon enough, civilizations will so thoroughly poison everyone with useless terminologies that we'll be programmed to behave at the drop of a hat. We can look forward to the glorious leaders taking control of our daily lives. So much for the low hanging fruit. Turns out, that fruit is what the devil wanted us to eat in the first place.

This won't last (1)

Xcott Craver (615642) | 1 year,18 days | (#43364285)

No matter how sophisticated the algorithm, the set of strings that get graded an A is bound to contain some weird and illegible elements. They probably won't be too hard to find by inspection of the algorithm and its training data. It will only take a few widely publicized examples of meaningless essays with a high auto-grade to cast doubts on this method of grading, no matter how effective it is in the common case.

Artificial Intelligence Software (0)

Anonymous Coward | 1 year,18 days | (#43364307)

// My version of this software
MIN_GRADE_FLOOR = 50;
STUDENT_FEEL_GOOD_PERCENT = 0.03;
score = randbetween(MIN_GRADE_FLOOR , 100);
score = score * (1 + STUDENT_FEEL_GOOD_PERCENT);

You ain't seen nothing yet (1)

David Govett (2825317) | 1 year,18 days | (#43364325)

With strong AI software, people could be tested continually, in real time. When the software detects a deficit, it could supply the relevant information for memorization. Ultimately, of course, the strong AI will become our brain, at least the rational part.

submissions (0)

Anonymous Coward | 1 year,18 days | (#43364359)

Perhaps /. should send all story postings to this and give editors (and contributors) a grade based on the results...

On the other hand, as this is not likely to change any behaviour, perhaps that says something about the value of grammar on the internet...

Anything profs can use, students can use (1)

Anonymous Coward | 1 year,18 days | (#43364395)

Imagine this post is an essay, and Slashdot has a little widget showing a letter grade on the right side. It starts out showing an 'F' and as I type more and more, and as my karma whores and whores, it moves up the scale until I have an 'A'. Then I hit send.

If this technology ends up in colleges, the whole essay writing process becomes moot... or does it? I suppose it's conceivable that at this point the essay writing is just a game. The exercise is tantamount to the professor saying, "get to level 6 on Super Mario College Essay" and if you do that you get an A.

I guess it boils down to a question of whether or not there's really any human skill in writing the essay, and whether or not that skill should matter. If algorithms can do it in school, then algorithms can do it for the media. There's no point learning journalism. There's a point in learning computer science to write the essay-writing algorithm... but only until the algorithm is good enough to take the place of humans. Then there's no point in that either.

I guess I could have gotten the same grade on this essay if I had simply cut to the case and said... "I for one welcome our robotic essay writing overlords".

free time? (1)

v1 (525388) | 1 year,18 days | (#43364417)

The software uses artificial intelligence to grade student essays and short written answers, freeing professors for other tasks."

There are probably at least several good jabs to make at this so I'll try to just address the best one.

First thing that comes to my mind is 'free them up to do what?" Education is adding more and more distance between the student and the teacher without throwing this into the mix. In a perfect world, teachers would be there to teach and that's it. But it's become more a problem of time management, lecture halls full of hundreds of students per teacher, and throw in the odd paper publish and grand write here and there. I think what we need to be doing is not looking for ways to further skew the student-to-teacher ratio, but to dial it back down a bit instead. Get me back to the good 'ol days where your prof knew your name.

Well, maybe two. "Artificial Intelligence beats Real Stupidity."

Free professors for other activities (1)

the eric conspiracy (20178) | 1 year,18 days | (#43364429)

Would you like fries with that?

Seriously though I don't think writing to what an algorithm wants is a bad thing if the algorithm wants the right stuff.

It's not as if students don't write to the algorithm the professor uses even now. The only difference is where the algorithm is stored and how flexible it is.

freeing professors for other tasks (-1)

home-electro.com (1284676) | 1 year,18 days | (#43364469)

What other tasks? The only task of the professor is to teach students. Anyway, this is probably at least third submission about automated grading systems over last 6 months, and it really is getting tired.

To any sane person it is obvious that the idea is retarded, so why do we keep discussing it?

Re: freeing professors for other tasks (1)

Kwyj1b0 (2757125) | 1 year,18 days | (#43364781)

What other tasks? The only task of the professor is to teach students. Anyway, this is probably at least third submission about automated grading systems over last 6 months, and it really is getting tired.

To any sane person it is obvious that the idea is retarded, so why do we keep discussing it?

Really? The last time I checked, no one got denied tenure for poor teaching. The faculty are there to bring money to the university - if they manage to teach well, that is an added bonus. However, they could be great teachers (I've known a few) who get fired because they didn't bring in hundred-thousand/million dollar grants every year of their assistant professorship.

While the idea is retarded, you seem to be missing the danger. When the university/faculty see the benefits of doing this, they will do it in a heartbeat. It frees up the teachers to spend more time on research and grant applications. It frees up teaching assistants (who are often hired as TAs to save grant money - they still have to do research). Everyone except the students (who have already paid for the class) benefit.

Creationism vs. Evolution seems retarded too - but because some people saw some idealogical/political benefit to pushing it through into the classroom, it did.

Re: freeing professors for other tasks (1)

Anonymous Coward | 1 year,18 days | (#43364793)

The main task of a professor is to do research, not teach. Sometimes, that is actual education research, research on how to teach.

That is Nothing.. (1)

wisnoskij (1206448) | 1 year,18 days | (#43364571)

At my university we have AI that continually checks our essays as we write them. It even points out the specific mistakes, gives suggestions to fix them, and allows us to rewrite that section of the essay.

We call this advanced AI MS Word.

No such thing as 'AI' (0)

Anonymous Coward | 1 year,18 days | (#43364573)

AI literally means "semantics automatically arising from syntax", and is thus a clear nonsense. It is the brain-dead rubbish believed by idiots that a convoluted and complex system will magically gain meta functionality.

So what does 'AI' really mean? It means computer systems that draw from databases that store some form of extracted Human knowledge. Usually the best so-called AI systems are using statistics in some clever and non-obvious way. Take this story...

I assume it actually draws from an April 1st joke, but let's assume it is real for the moment. Take a large collection of student essays, each graded by a first class teacher. Now, run a stats engine on the PATTERN (not meaning) of the syntactical form of each grade-group essay. Look for things like sentence length, paragraph length, number of sub-clauses, and occurrence of different reading-level-group words. Do a crude analysis on grammar and punctuation.

Now see if there is a strong correlation between a grade an essay earns, and its statistical form. If such a correlation exists, propose the absolutely dreadful idea of reversing the process- ie., use the same stats methods on ungraded essays, and apply a grade according to which grade-group the stats best correlate to.

Sad to say, betas will think this 'method' makes sense, just as betas tend to think 'pyramid' schemes are worth joining. To make it easy for you dumb-dumbs to understand why such a system is a terrible idea, here's the problem. Once such a system is applied to grading, two things happen. Firstly, essays that refuse to follow the dumb rules the AI system has extracted from the original essay pool will be mis-graded. Secondly, the dumb AI rules can be discovered, allowing 'essays' to be created automatically that will be nonsense to Human readers, but will score the highest grade from the 'AI' grading engine.

To make it clearer to you dumb-dumbs- it is impossible to create an essay auto-grading computer program that has any merit whatsoever. Sadly, dumb-dumbs will carry on falling for impossible compression schemes, fool-proof strategies for winning at roulette, and 'Human thought process like' AI computer programs.

PS Google creates its so-called 'intelligent' algorithms by employing vast numbers of Humans to enter new rules into their computer systems daily. Patterns of search requests are inspected by Humans, and enhanced by Humans identifying what the users were really after. AI, in the true meaning of the word, does not and cannot exist (on a Turing complete computer- and no, there is no magic other form of 'better' computer, or magic other form of maths that cannot run on a Turing complete computer).

Automated Grading AKA Spellcheck??? (1)

wisnoskij (1206448) | 1 year,18 days | (#43364603)

So basically they feed the essay into Google spellchecker and count the number of underlined words?

Freeing for? (-1)

Anonymous Coward | 1 year,18 days | (#43364727)

"freeing professors for other tasks." Tasks like what? Banging interns? Smoking more pot? Jesus, just do your whole job.

Rewrite the test? (1)

PPH (736903) | 1 year,18 days | (#43364737)

How many times?

I can envision an essay writing 'bot making multiple submissions with some AI to optimize subsequent submissions based on the .

I know it sounds like an awful lot of effort to build, but its time better spent than writing a two page essay on Dickens, IMO.

That BS again (1)

gweihir (88907) | 1 year,18 days | (#43364743)

This is now the n-th time something like this has been on /.

The answer is still the same: This is a very bad idea. Students will learn how to game the software instead of how to write well. No software can grade whether the reasoning is sound, the images vivid, the prose well readable. But what students will learn is that writing essays is not important, after all it is not even worth bothering a human being to read and grade.

"Freeing professors for other tasks"? (2)

msobkow (48369) | 1 year,18 days | (#43364765)

Any professors I've ever known or been taught by had their grad students doing the grading, anyhow.

Besides, what exactly are the professors being "freed up" for? Isn't their JOB to TEACH?

Oh, yeah, I'm thinking old school. Nowadays a professor's job is to find corporate grants...

Re:"Freeing professors for other tasks"? (0)

Anonymous Coward | 1 year,18 days | (#43364905)

Nowadays a professor's job is to find corporate grants...

That is totally not true! My adviser, for example, gets government grants!

Good for the filler / big lecture classes t online (1)

Joe_Dragon (2206452) | 1 year,18 days | (#43364815)

Good for the filler / big lecture classes to move to a full online / test only setting and maybe they can pass on the lower costs to the students.

It may just lead to people to gameing the system but what does that give them? More time to work on there core classes?? I think that it is better to say do the min to pass art history so I have the time to work on the classes I want to work on.

Where can I find more information on this? (2)

RobertinXinyang (1001181) | 1 year,18 days | (#43364835)

I read the article and went to edX. At edX I signed up; but, I can not find out about this system. Quite Frankly, I am a teacher and I need my students to be writing more; however, I do not have the time to grade all of their papers so I have been assigning more objective homework that I would like.

A system like this may work as a first pass filter to do the bulk of the grading, allowing me to focus on identifying common problems and developing lessons based on common errors rather than tying myself down with a huge stack of papers. This would also benefit the students by providing them with more consistent grading and feedback.

This may not be what I go with; but, I would like to have a look at it. That takes me back to my question, can anyone point me to somewhere that I can get more information on this?

SCIgen v2.0? (0)

Anonymous Coward | 1 year,18 days | (#43364849)

Come on, this is slashdot and noone has pointed http://pdos.csail.mit.edu/scigen/ yet?

All you need is to feed it with a new richer syntax tree generator, for getting an undisputed and bold 10!

Grade this!

side effect of big classes. also Multiple choice (1)

Joe_Dragon (2206452) | 1 year,18 days | (#43364921)

side effect of big classes. also can drop tests down to a Multiple hidden choice test. It's like a Multiple choice test without knowing what the choices are.

smaller classes and team based work maybe even apprenticeships / more of a tech / trades schools level testing where it's more about real skills and not about test cramming taken to the next level where you now just need to know the buzz words.

You do this at Berkeley (0)

Anonymous Coward | 1 year,18 days | (#43364995)

In the engineering courses, some teachers will assign you work to submit online and you have to type it in exactly as it should be written by the application's standards. The problem of course is that the teacher doesn't want to bother with false positives so they give you 3 chances to correct your work. If you don't get it right then you get nothing for that problem. What was strange is that there were often bonus questions as a sort of backup which they wouldn't count for your grade other than improve upon it if you missed other questions. Say they assign you 5 questions, you may only need to answer 2 correct to get 100%, the rest doesn't improve your grade. It's a stupid system and it certainly takes away from the 1 on 1 education that students need. The US has a strange concept of having a lot of students per class, I thought it was ridiculous and they still have the nerve to charge an arm and a leg for it.

Hard to believe (0)

Anonymous Coward | 1 year,18 days | (#43365017)

You are familiar with the email "Help desk" at web sites. That's where our email questions are read by software scanners that look for key words and automate a response unrelated to what we wrote about. Well guess who thinks that's a great paradigm for grading exams? Harvard and MIT, that's who.

On college exams, instead of handing in a blue book and getting a grade from a TA a few weeks later, click on a 'send' button when you're done and receive a grade instantly, your essays scored by software.

Then, instead of being done with the exam, imagine that the system would immediately let you rewrite the test to try and find more of the keywords it's programmed to reward.

EdX, an education disgrace founded by Harvard and the Massachusetts Institute of Technology to offer EasyA courses on the Internet, has just introduced this system and will make its automated software available free on the Web to any misguided institution that wants to use it. (Prediction at UVa: Dragas Yes!, Sullivan No!)

Like help desk software, this uses artificial intelligence to grade essays and written answers by keyword searching, preparing future faculty for writing unfathomable distorted prose making simple ideas incomprehensible.

Among the many losers in this rotten plan: the TAs who now read and grade undergraduate exams can be let go.

US Leads in Race to Bottom (1)

EmagGeek (574360) | 1 year,18 days | (#43365043)

This has got to be the dumbest fucking idea I have ever seen - and it's completely unsurprising where it is coming from.

Load More Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...