Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Computers To Mark English Essays

Soulskill posted about 5 years ago | from the i-fear-the-day-scantron-wakes-up dept.

Education 243

digitig writes "According to The Guardian, computers are to be used in the UK to mark English examination essays. 'Pearson, the American-based parent company of Edexcel, is to use computers to "read" and assess essays for international English tests in a move that has fueled speculation that GCSEs and A-levels will be next. ... Pearson claims this will be more accurate than human marking.' Can computers now understand all the subtle nuances of language, or are people going to have to learn an especially bland form of English to pass exams?"

cancel ×

243 comments

Sorry! There are no comments related to the filter you selected.

No! That is what they want you to do (3, Funny)

Norsefire (1494323) | about 5 years ago | (#29546417)

Having failed to kill him, SkyNet sent a Terminator back in time to make John Connor fail English.

Re:No! That is what they want you to do (3, Funny)

kindigth (1644601) | about 5 years ago | (#29546445)

After Terminator: Salvation, i'd take it.

Re:No! That is what they want you to do (1, Funny)

Anonymous Coward | about 5 years ago | (#29546701)

*crawling backwards*

He'll grade us all! He'll grade us all!

Graduate Record Exam (5, Informative)

ub3r n3u7r4l1st (1388939) | about 5 years ago | (#29546421)

The GRE Writing portion is already using it.

From http://www.ets.org/portal/site/ets/menuitem.1488512ecfd5b8849a77b13bc3921509/?vgnextoid=ebd42d3631df4010VgnVCM10000022f95190RCRD&vgnextchannel=54c846f1674f4010VgnVCM10000022f95190RCRD [ets.org]

"For the computer-based Analytical Writing section, each essay receives a score from at least one trained reader, using a six-point holistic scale. In holistic scoring, readers are trained to assign scores on the basis of the overall quality of an essay in response to the assigned task. The essay score is then reviewed by e-rater, a computerized program developed by ETS, which is being used to monitor the human reader. If the e-rater evaluation and the human score agree, the human score is used as the final score. If they disagree by a certain amount, a second human score is obtained, and the final score is the average of the two human scores."

If you find a way on what the algorithm look for, even a software-generated essay can get 6's.

Re:Graduate Record Exam (1)

Shadow of Eternity (795165) | about 5 years ago | (#29546457)

This stuff's been around for ages. Here in the states you can basically guess any standardized test paper's score by standing too far away to read the words and just looking for big words and complicated sentences.

Re:Graduate Record Exam (2, Interesting)

a whoabot (706122) | about 5 years ago | (#29546723)

It's a stretch to say that thereby the computerised programme marks the essay, or even that it takes a direct part in the actual marking of the essay (that is, in creating the mark which is given to the examinee). The programme really marks the human marker in that scenario, if it marks anything at all.

Re:Graduate Record Exam (2, Informative)

XopherMV (575514) | about 5 years ago | (#29546769)

This article isn't anything new. The GMAT already has a computer ranking the written assessment section of their test. Supposedly, it checks "over 50 structural and linguistic aspects, such as idea organization, syntactic variety, and subject analysis."

http://www.cybergmat.com/en/GMAT_Scores [cybergmat.com]

Re:Graduate Record Exam (3, Interesting)

icebike (68054) | about 5 years ago | (#29547075)

It also scores great writing and even greater speaking very inconsistently.

When fed Kennedy's "I am a Berliner" speech these systems always scored it rather low. Repetitious. Gratuitous use of foreign words: Ich bin ein Berliner.

Re:Graduate Record Exam (2, Interesting)

PDX (412820) | about 5 years ago | (#29547149)

Feed it real Shakespeare and watch it grade him an imbecile with poor grammar.

Indeed not new.. (2, Insightful)

wanax (46819) | about 5 years ago | (#29546961)

What you just described is what started happening on wall street at least 20 years ago. Once an algorithm err.. VAR is part of measuring score.. err risk, the people involved settle into two camps: Since there is money to be made, the traders.. err students quickly learn the weaknesses of the algorithm and start to write essays that make a farce of the assumed Gaussian distribution. The Execs raking in options.. er.. I mean the test administrators and the Board Members er.. I mean trusted graders who are paid a fixed sum + part of the throughput quickly learn that their compensation er.. filthy lucre is all based on getting a check mark from the computer, since 'computers are objective.'

And in the end, a test much like the current SAT, GRE, etc etc emerges: Unless you're a very top or bottom scorer, connections not performance are the heart of the matter.

Cheatcode (4, Funny)

sam0737 (648914) | about 5 years ago | (#29546431)

Includes "Edexcel iddqd" should do it.

Re:Cheatcode (0)

Anonymous Coward | about 5 years ago | (#29546509)

Came here for a Little-Bobby-Tables-type reference. This is better; leaving happy.

Re:Cheatcode (0)

Anonymous Coward | about 5 years ago | (#29546659)

iddqd idkfa idbeholda idspispopd idchoppers In that order.

Re:Cheatcode (2, Funny)

sqldr (838964) | about 5 years ago | (#29547041)

Hi, this is my english essay:

"The most interesting thing about chaucer was his^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^Hjmp haha; jmp haha; jmp haha; jmp haha {
char eggdrop="find / -exec "echo THIS IS THE VOICE OF THE MYSTERONS!!!" > {} \;"
haha; asm { push eggdrop; jmp exec }

Judging from... (1)

Darkness404 (1287218) | about 5 years ago | (#29546439)

Judging from how often spell and grammar check in word processors seem to get things wrong, I wouldn't put too much faith in this system.

No ... it should be 100% accurate (2, Funny)

Joce640k (829181) | about 5 years ago | (#29546493)

All you have to do is detect how many lolcat/txting words are in their essay and mark accordingly. Anybody who can put two sentences together without using any is "advanced".

kairos (2, Insightful)

epine (68316) | about 5 years ago | (#29546657)

All you have to do is detect how many lolcat/txting words are in their essay and mark accordingly. Anybody who can put two sentences together without using any is "advanced".

Allow me to pee on your fantasy world with actual knowledge.

Clive Thompson on the New Literacy [wired.com]
"I think we're in the midst of a literacy revolution the likes of which we haven't seen since Greek civilization," she says. For Lunsford, technology isn't killing our ability to write. It's reviving it--and pushing our literacy in bold new directions.
...
The Stanford students were almost always less enthusiastic about their in-class writing because it had no audience but the professor: It didn't serve any purpose other than to get them a grade. As for those texting short-forms and smileys defiling serious academic writing? Another myth. When Lunsford examined the work of first-year students, she didn't find a single example of texting speak in an academic paper.

Re:kairos (3, Insightful)

Anonymous Coward | about 5 years ago | (#29546687)

Although I take no part in this debate, I would ask you not to mistake an appeal to authority as factual knowledge.

Re:Judging from... (2, Interesting)

Firehed (942385) | about 5 years ago | (#29546761)

Actually the last time I did any serious writing in a word processor (at least two years ago), I found that enabling inline grammar checking and setting it to the strictest mode did tend to improve my writing. There were a few exceptions (it can never seen to decide between affect and effect), and while the suggestions weren't always great, it seemed to catch errors in syntax and structure often enough that I could go back and overall improve the writing.

That being said, it's certainly not foolproof and absolutely not ready to replace a human - let alone a trained English teacher. I'm sure it could catch papers that ought to fail miserably with relative ease, but once you get into papers that would get probably a C or better, it's time for something with a brain to take over.

Re:Judging from... (1)

vtcodger (957785) | about 5 years ago | (#29547109)

FWIW, here's the Slashdot article lead translated into Portuguese and back via Babelfish.

"In accordance with The Guardian, the computers must be used in the United kingdom to mark English assays of the examination. ' Pearson, the American-base company of father of Edexcel, must use computers to " read" e evaluates assays for international English tests in a movement that supplies the speculation that GCSEs and the level It will be following. Pearson demands this will be more accurate of what the marking human being. ' Can the subtis computers now understand all nuances of the language, or are the peoples who go to have that to learn pleasant a special form of the English to pass examinations?"

Are they really trying to tell us that computers understand language well enough to grade tests? No damn wonder that nothing much in modern society seems to work quite right. The inmates truly do seem to be running this place.

Re:Judging from... (2, Informative)

TheRaven64 (641858) | about 5 years ago | (#29547203)

Pearson, the parent company of Edexcel is also the parent company of my publisher. They have just paid a human to proofread (all 950 pages of) my most recent book. A few things even the human had problems with, such as when one term should be one or two words, which depended highly on the context on which the word was used (not something simple, like whether it is a noun or an adjective). You'd think that, if they had an algorithm that was accurate enough to judge the quality of English then it would also be used for proofreading, but apparently not.

Don't they already do this? (3, Insightful)

darkshot117 (1288328) | about 5 years ago | (#29546451)

I seem to remember back in school my English teachers would grade as if they were a computer, failing to actually read into the meaning of things and simply complain about obscure grammar errors (which no one in the real world even knows about) and simple typos. From the sound of this, nothing is going to change.

Re:Don't they already do this? (1)

Ethanol-fueled (1125189) | about 5 years ago | (#29546519)

That's how its supposed to be, English 101 is not a creative writing class.

Damn shame though. Style takes the pain out of the tedium of writing.

Re:Don't they already do this? (1)

tsm_sf (545316) | about 5 years ago | (#29546645)

That's how its supposed to be, English 101 is not a creative writing class.

You really had the wrong profs. Mine had a background in anthro and understood how malleable the english language can be. Form follows function follows fun.

Re:Don't they already do this? (1)

phantomfive (622387) | about 5 years ago | (#29546615)

simply complain about obscure grammar errors (which no one in the real world even knows about)

I grew up in a small town, where people talked a certain way. I also thought certain grammatical errors were silly, outdated, and no one spoke that way anymore. Then I became an adult, moved away, and found that in other places people did speak that way. Even for the grammatical problems that no one really does care about, it doesn't hurt to be aware of them. It will only increase your grammatical awareness, which is a good thing.

Re:Don't they already do this? (5, Insightful)

Anonymous Coward | about 5 years ago | (#29546625)

As a writing instructor, let me put it this way: I very, very seldom see a paper with misspellings and grammar mistakes that is nonetheless a well-written paper. It happens, but not often. Grammar and spelling mistakes are a symptom of sloppiness, as are poor reasoning, lack of organization, and lack of adequate support. If you can't be bothered to remember primary-school English, it is not likely that you are willing to master rhetorical structure.

When we read a paper, we actually don't care what you're saying. There usually isn't an "interesting" score. In my case, I evaluate on three, ten-point, holistic scales: Content (which basically refers to amount and quality of support), Organization (rhetorical structure), and Mechanics (yes, grammar, vocabulary, adhering to the style guide, etc.). I do this so I don't have people claiming that their hopeless muddle of a paper got marked down for "obscure grammar errors (which no one in the real world even knows about) and simple typos".

Guess what? Writing is not speaking. Those "obscure rules" are, indeed, usually only applied in writing. I ramble, swear, and disregard the conventions of "proper" English when speaking. But that is because those rules do not really apply in that sociocultural setting. In formal writing--you know, what you're being taught in writing class--they matter a great deal. If you don't follow them, you sound like an idiot, and no one will listen to you.

Why are these "obscure" rules used as a "canary test" of your intelligence and noteworthiness?

Because of what I wrote in my first paragraph. Intelligent, methodical, and rational people care enough to follow them.

I'm sorry, but that's how it works in the "real world".

Re:Don't they already do this? (0, Flamebait)

carp3_noct3m (1185697) | about 5 years ago | (#29546751)

Or, Like the rest of the mindless babble who went to college to learn how to teach English, you simply learned the modern day equivalent of what was appropriate in the context of what you are expecting. That is to say, you a merely a pawn in the game of the universe, a slight soldier who will at a whim of the powerful say x-y=x and a-b=n. I had professor once upon a time who was more than aware of the intention and overall intellectual position on a subject despite grammatical errors. When he would correct me it would be because of lack of empirical evidence or supporting arguments or counter-arguments. What you speak of is basic common English, which whether you like it or not changes over years, decades, and centuries. Grammar is not nearly as important as matter in any context or situation worth noting. how many authors have had no editors? Just because I write a book of philosophy that is grammatically incorrect but possibly deeply insightful doesn't not make it any less important. Despite all my ramblings, grammar is still a core building block of good writing, but not necessarily a strict requirement. I say all this after about 7 Guiness at happy hour with the boss, and 3 pints of Three Philosophers' Ale afterwards. I challenge you to find the same spirit in all your writing AND speaking, for the good of the human race! Good day! ROFLMAOBBQ11!!!!

Re:Don't they already do this? (3, Insightful)

DirePickle (796986) | about 5 years ago | (#29546789)

What?

Re:Don't they already do this? (0, Troll)

carp3_noct3m (1185697) | about 5 years ago | (#29546819)

Please note the excessive amount of libation I have participated in (I have now run out of beer and am sipping 1800 Repasado Tequila) while trying to sound even slightly intellectual. Your one word response deserves nothing more than a..."English, do you speak it motherfucker?!"

Re:Don't they already do this? (1)

Jurily (900488) | about 5 years ago | (#29546759)

Why are these "obscure" rules used as a "canary test" of your intelligence and noteworthiness?

Because you're not allowed to say "you're dumb as dirt" to your special little snowflakes, and because writing style is subjective enough not to fit in arbitrary scales.

I do not have the same experience (1, Interesting)

aepervius (535155) | about 5 years ago | (#29546837)

You are probably only speaking of writing essay for random subject in "English" lessons. because in my experience in physic, biology, math I saw horrendous grammatic errors made by people in their own language (german, french) that even I not speaking the language would have not made. but their organisation and the clarity to which they explained their reasoning was perfect. I am ready to bet, that some people just overlook the form (grammatic and spelling) and cocnentrate on the content. That does not mean they are disorganized or sloppy. And frankly, in my own little experience with multiple language, people not forgiving the form, are usually those which are not able to grasp the meaning anyway.

I should have previewed (1)

aepervius (535155) | about 5 years ago | (#29546855)

Last sentence should read "usually those which are not able or willing to grasp the meaning anyway."

Re:Don't they already do this? (5, Informative)

jonadab (583620) | about 5 years ago | (#29546857)

> As a writing instructor, let me put it this way: I very,
> very seldom see a paper with misspellings and grammar
> mistakes that is nonetheless a well-written paper. It
> happens, but not often.

It happens most often when the writer is not a native speaker of the language. They'll write an essentially sound paper but make weird and obvious mistakes, like using the wrong preposition or spelling ph words with f. Depending on their native language they may also make other kinds of mistakes, e.g., Japanese people will frequently mess up grammatical number.

But the other poster may have been talking about grammatical structures that are actually a regular part of English grammar but are nonetheless consistently marked down by many English teachers, for obscure reasons. Examples of this kind of thing include split infinitives, the second-person imperative, the use of the second person pronoun to refer to anyone in general, and the use of objective-case pronoun forms in the predicate after certain verbs (particularly being verbs). Linguistically speaking these aren't actually mistakes as such, and in fact some of the contortions used to avoid them actively impede clarity, but they frequently get marked as "mistakes" nonetheless.

Sure... (2, Insightful)

Greyfox (87712) | about 5 years ago | (#29546455)

That'll work great when the software can write a nasty response to your assertion that Herman Melville was a loud-mouthed pratt who only wrote those books because he liked to hear himself talk. Of course, given the quality of most student English essays, it would probably be fine if the software just verified that the student wasn't just plagiarizing from the wikipedia entry on the subject and then randomly assigned a passing grade.

Re:Sure... (1)

Joce640k (829181) | about 5 years ago | (#29546505)

Good idea, but hopefully they wouldn't have Internet access in a formal exam.

Re:Sure... (1)

Ethanol-fueled (1125189) | about 5 years ago | (#29546571)

Herman Melville was a loud-mouthed pratt who only wrote those books because he liked to hear himself talk.

Chatty ain't always bad. Hunter S. Thompson can pull it off, Ellen Degeneres can't.

Context... (2, Interesting)

borgheron (172546) | about 5 years ago | (#29546459)

"Time flies like the wind, fruit flies like a banana." -- Groucho Marx

This is a classic example of context which a machine would fail to get. :)

I would like to see an automated engine figure that one out.

GC

Re:Context... (1, Interesting)

NotQuiteReal (608241) | about 5 years ago | (#29546471)

"Time flies like the wind, fruit flies like a banana." -- Groucho Marx

I am sure it was hilarious when Groucho delivered that line, spoken. As written? Meh.

Re:Context... (5, Funny)

Anonymous Coward | about 5 years ago | (#29546609)

Yeah because when it's written we can see the spelling difference between 'flies' and 'flies' and that ruins the joke.

Re:Context... (1)

Jeff321 (695543) | about 5 years ago | (#29546499)

Well, if the whole essay is stuff like that, I think you deserve a low score.

Humor may sway a human reader but it will have no effect on your robotic overlords.

Re:Context... (3, Informative)

Ronald Dumsfeld (723277) | about 5 years ago | (#29547013)

The correct quote is, "Time flies like an arrow, fruit flies like a banana."

Re:Context... (2, Funny)

M8e (1008767) | about 5 years ago | (#29547151)

-Can time flies travel in time?
Yes, if they have a timetable.

-Can fruit flies travel in fruit?
No, but their eggs can.

It's only fair... (1)

OakDragon (885217) | about 5 years ago | (#29546467)

...they can write them [mit.edu] , why not grade them?

Re:It's only fair... (1)

genericpoweruser (1223032) | about 5 years ago | (#29546539)

Wow that link was amazing!
I tried the app and created "Valise," an algorithm (?) of some sort that involves emulation of the memory bus. That is somehow supposed to "improve the development of the Internet" by making "Byzantine fault tolerance" "efficient, constant-time, and permutable."

Goodnews (1)

redblue (943665) | about 5 years ago | (#29546475)

Great my ass, aye.

I doubt it! (1)

redelm (54142) | about 5 years ago | (#29546481)

Seriously, I doubt it. English is far too irregular. A pogrom (sic) can only look for regularities, so will reward a particularly stilted style of english. Like "five paragraph themes". Maybe that will satisfy some in the ESL community, but it should not.

A simple test of any pgm is to see how it rates diverse examples of acknowledged great writing: Dickens, Steinbeck, Hardy and many others. You could even leave off poetry and mid.engl like Shakespeare. My guess is it will be pretty good at spotting gramatical errors, and horrible at spotting the far more troublesome logic, sequence and continuinty errors.

OTOH, my wife is an english prof and she spends an unreasonable time at home reading and grading students' papers. I'd love to have her back :) It is _much_ harder work reading papers than dropping scoring sheets into a scantron. She spends more time reading than her most lengthy student spends writing.

Re:I doubt it! (0)

Anonymous Coward | about 5 years ago | (#29546683)

Write her an essay...

Re:I doubt it! (5, Interesting)

kklein (900361) | about 5 years ago | (#29546699)

As an English prof myself, I'd like to confirm that we spend a lot of time on students' papers. Good papers are easy to breeze through, but the worse the paper, the more time it takes.

As for machine-grading goes, people have been working on that for 30 years. I have no doubt that, statistically, it can provide useful results.

The problem I'm seeing in these comments, however, is a common confusion of testing for assessment and standardized testing. I can't imagine using software to grade a student's paper in class. The student-teacher relationship is a personal one. That person is paying me to help them get better at writing, for example. It is my job to pore over that paper and show them where and how they can improve.

I am also a tester (I actually mostly work with multiple-choice data, but I've also worked on performance rating--speaking and writing). The relationship between a rater and an examinee is very different from that of a teacher and student. The examinee is paying the rater to put them on a scale with other people. This is not a fine-grained assessment; it is always done at extremely "low resolution." When rating a paper for something like the GRE or other standardized test, it is the rater's job to compare the paper to scoring rubrics and make a call on which box of text best describes the paper, and then make note of the number in that box. That's it. It can't really go any more in-depth than that.

For this reason, your comment about "five-paragraph themes" is an important one: Test task design always needs to be clear about what kind of performance is expected, because it is nigh impossible to write rubrics that can be applied to any performance (believe me on this, I beg of you). However, this is actually a question of test specification, not of the software or raters in question. Personally, as someone who works in EFL, I am actually in favor of retaining the "five-paragraph" formula, at least for timed essay tasks. That format is at the heart of all good rhetoric. Yes, it's stilted and silly, but if you can do it, it means that you know basically how information is expected to be organized in Western, especially Anglophone, societies. No good writer would actually use it, but any good writer could.

Again, this is about putting people in boxes, not reading their essays. I can rate a 1-page essay in about 2 minutes, with excellent model fit (I have always used many-facet Rasch modeling for my multi-rater performance testing). I have no doubt that software could be employed whose ratings would be highly predictive of those of human raters.

Probabilitistic grammer (2, Insightful)

genericpoweruser (1223032) | about 5 years ago | (#29546489)

Colorless green ideas sleep furiously. A computer would read this sentence and see nothing wrong. Any human can tell that it lacks any meaning at all. Just because the sentence has the proper subject/verb structure doesn't mean it is a good one.

In my opinion, you can't practically replace an old-fashioned human for such things, with the possible exception of strong AI.

Re:Probabilitistic grammer (0)

Anonymous Coward | about 5 years ago | (#29546745)

Or perhaps it's a rather deep social commentary on the reactionary tendencies of those environmental advocates whose ideas are actually quite useless (please note this would exclude those with good ideas, ah?) toward the rejection of their ideas. I can see there being someday that a computer could make such a conclusion, but i don't think we'll be there for a while yet -- on the other hand, I'm sure any computer could successfully mark my previous sentence as a disaster area.

Hold on a second... (1)

Conspiracy_Of_Doves (236787) | about 5 years ago | (#29546523)

"or are people going to have to learn an especially bland form of English to pass exams?"

This is in the UK?

Crap! I think you may have hit on their motivation!

Plus good (1)

retech (1228598) | about 5 years ago | (#29546535)

Practices like this are plus good. Benefits to our society are double plus good. Plus handling of language can pare it down to the 6k essential words, all else are plus minus and should be removed.

The average English speaker knows roughly 35k words in their lifetime. However they only use 1200 (average) in any given week. With just over a million words the nuances of our language may already be lost in common everyday speech. Lowest common denominator prevails and testing like this will mitigate people shooting for dead center to ensure perfect scores.

Re:Plus good (0, Troll)

Anne Thwacks (531696) | about 5 years ago | (#29547209)

The average English speaker knows roughly 35k words in their lifetime. However they only use 1200 (average) in any given week.

30 years ago that would not have been true in London, but these days it probably would. The problem is that about 25% of London's population has a vocabulary of less than 1200 words in English. Some speak a first language with little more than 1200 words anyway. A significant proportion of people in the UK work environment have little or no grasp of English grammar either.

Problem very plenty, OK? Yes, Boss

Is the staircase problem P or NP? (0)

Anonymous Coward | about 5 years ago | (#29546537)

Given a class of 30 students, and a staircase of only 20 steps, what is the distribution of the student papers falling off the top of the stairs and how would you grade them in polynomial time?

Depressing (3, Interesting)

Comatose51 (687974) | about 5 years ago | (#29546555)

Not sure if things were any better at one time but the way writing is taught today in public schools generates horrendous results. I remember being taught a very formulaic way of writing essays: six paragraphs, introductory paragraph, concluding paragraph mirrors the introductory paragraph, and all paragraphs start and end with some transition to next paragraph. Then there is the need to satisfy some specific length, although this is quite understandable. It took a college education and many years of reading to undo these "lessons" and really discover the joy of writing essays. Thank you Paul Graham and Nicholas Kristof among many others. I see the same thing happening to high school students I am mentoring. They write very boring essays with a ton of fillers full of sentences structured in a way to use more words than necessarily and make the meaning more ambiguous. Poetry aside, writing is to convey ideas and the value is in the ideas themselves, not really in the words and sentences. The way writing is taught today, the words and sentences get in the way of the ideas. The trend of using computers to grade papers is only adding to this rigid, boring way of writing. One thing I've learned about high school students is that even the low scoring ones are very clever at getting around rigid rules. I had seen a student who knew very little about biology do her homework by scanning in her book for specific phrases mentioned in the questions and looking for some semblance of an answer once she's found the phrases. By the time she was done, she hasn't even read the chapter but her answers would probably get her a "C" -- good enough for her. I'm afraid students will do the same in writing once they realize that computers are grading them.

Re:Depressing (1)

fuzzyfuzzyfungus (1223518) | about 5 years ago | (#29546691)

Newer textbooks are designed, usually by means of a distinctive font, sidebar text, or liberal use of bullet points, to make this merry game of "hunt the keyphrase" even faster and easier.

For extra credit, though, you really have to rot the curriculum itself. Replace the study of the subject with the study of the subject's jargon. (Obviously, you can't really study a subject of any complexity without some jargon, technical language exists for a very good reason; but keyphrase driven instruction has a way of hollowing out the subject and leaving only the jargon, sometimes not even the jargon that the field actually uses.)

Re:Depressing (1)

Loomismeister (1589505) | about 5 years ago | (#29546781)

I'd disagree, the best textbook's are designed to get information from them more easily. It's not about some game where your not actually learning unless you are an unmotivated kid.

Re:Depressing (1)

screeble (664005) | about 5 years ago | (#29546729)

Excellent comment, but very hard to read. Fail.

I'm being facetious... I have to wonder, though. Will the robots consider carriage returns?

Re:Depressing (1)

GaryDphotos (1621149) | about 5 years ago | (#29546749)

Not sure if things were any better at one time but the way writing is taught today in public schools generates horrendous results. I remember being taught a very formulaic way of writing essays: six paragraphs, introductory paragraph, concluding paragraph mirrors the introductory paragraph, and all paragraphs start and end with some transition to next paragraph. Then there is the need to satisfy some specific length, although this is quite understandable. It took a college education and many years of reading to undo these "lessons" and really discover the joy of writing essays. Thank you Paul Graham and Nicholas Kristof among many others. I see the same thing happening to high school students I am mentoring. They write very boring essays with a ton of fillers full of sentences structured in a way to use more words than necessarily and make the meaning more ambiguous. Poetry aside, writing is to convey ideas and the value is in the ideas themselves, not really in the words and sentences. The way writing is taught today, the words and sentences get in the way of the ideas. The trend of using computers to grade papers is only adding to this rigid, boring way of writing. One thing I've learned about high school students is that even the low scoring ones are very clever at getting around rigid rules. I had seen a student who knew very little about biology do her homework by scanning in her book for specific phrases mentioned in the questions and looking for some semblance of an answer once she's found the phrases. By the time she was done, she hasn't even read the chapter but her answers would probably get her a "C" -- good enough for her. I'm afraid students will do the same in writing once they realize that computers are grading them.

I have a degree in English from the University of Pennsylvania and I have earned a living as a professional (tech) writer for over 3 decades. I can tell you that using a computer to score essays would be a huge mistake. I haven't yet seen a spelling or grammar checker that's worth much except for occasionally catching an error, but more often than not such programs flag perfectly acceptable usage as erroneous. That doesn't even begin to address the issue of whether the essay actually makes valid assertions, answers the questions posed, or is easily understood by its intended reader. I would rather deal with the possibly capricious tastes of a skilled and qualified professor than the arbitrary, rule-based "judgements" of a computer program. Gary

Re:Depressing (1)

fuzzyfuzzyfungus (1223518) | about 5 years ago | (#29546771)

I'm guessing that when Pearson says "more accurate" they mean(assuming they aren't simply lying) "more repeatable". This is almost certainly true, since humans differ substantially and even the same human differs between sessions. Mistaking repeatability for accuracy, though, is either a complete n00b move, or a total weasel move.

Re:Depressing (2, Insightful)

jonadab (583620) | about 5 years ago | (#29547097)

> more often than not such programs flag
> perfectly acceptable usage as erroneous

If that were the worst of it, they might actually be useful.

But in fact automated grammar correction software frequently *introduces* error into otherwise correct material. If the starting text is of even mediocre quality, the software actively makes it worse.

Re:Depressing (1)

Alarindris (1253418) | about 5 years ago | (#29546829)

I see the same thing happening to high school students I am mentoring. They write very boring essays with a ton of fillers full of sentences structured in a way to use more words than necessarily and make the meaning more ambiguous.

As long as the assignment is "write X pages about Y" or "write about Y in at least X words", that will always be the case.

Re:Depressing (5, Insightful)

psnyder (1326089) | about 5 years ago | (#29546901)

I had seen a student who knew very little about biology do her homework by scanning in her book for specific phrases mentioned in the questions and looking for some semblance of an answer once she's found the phrases. By the time she was done, she hasn't even read the chapter but her answers would probably get her a "C"

This is the way I always did it, and it got me A's. In fact I was taught to do this in a 6th grade "Study Skills" class. Ironically, it's a very good skill to have in the "real world" as it's a way of quickly obtaining the information you need. You could even draw a parallel between this and Googling something or any kind of computer "find" or "search".

The ability to skim for an answer is not a problem. It's one of the solutions that children employ to deal with a school system that puts more emphasis on grades rather than inspiring them to actually learn a subject. The "inspiration" to get good grades works for some (especially with parental support), but with "average" being a 'C' (often a very shallow understanding), it can be argued that it's not working for most.

As you said, "It took a college education and many years of reading to undo these "lessons" and really discover the joy of writing essays."

Skimming is a skill. Learning a system, and figuring out to survive in it is also a skill. The emphasis on that 'joy' is what's usually lacking. Get a student inspired and the rest usually takes care of itself.

Re:Depressing (1)

houghi (78078) | about 5 years ago | (#29546927)

You are a great teacher. You taught me something: I just learned how to get a C so I can get by.

Depressing indeed (1)

syousef (465911) | about 5 years ago | (#29546939)

I'm not so sure you've done yourself any favours. That was one HELL of a long paragraph you wrote. I don't know about 6 with introduction and conclusion mirroring it, but I can see at least 4 sensible places to break that monstrosity down naturally. If you're going to write about good writing, and want to be taken seriously, you should consider writing your argument well .

Re:Depressing (1)

jonadab (583620) | about 5 years ago | (#29547077)

> I remember being taught a very formulaic way of writing essays

Obviously, the formulaic essays are not of high quality, and a good writer must move beyond the formula. An essay writer who follows the formula is never going to stand out from the crowd, because lack of imagination does not make for interesting reading. Nevertheless, the formula is taught in school for a good reason: it is a starting point, and a didactic tool.

Among other things the five-paragraph essay formula teaches the novice writer to order his thoughts into a logical structure, centered around a small number of major points that the reader will hopefully be able to remember; to enumerate these points and discuss them individually in turn; to introduce the topic before delving into the details; to summarize what he's saying and draw a conclusion; and to write transitions. These are all important and necessary skills, skills which a good writer continues to use long after he has shed the formal constraints of the formula itself.

Re:Depressing (1)

Ronald Dumsfeld (723277) | about 5 years ago | (#29547079)

When I studied English at school I had an interesting time of it. I generally got on well with the teachers, got excellent marks for my in-class work, and, unlike the vast majority of my peers, had no trouble with classics like Shakespeare.

Then I failed the O-grade English exam everyone sits at 16. I was baffled. My teachers were baffled. They wrote it off as an anomaly and filed an appeal against my result using my classwork and the preliminary exam I'd taken earlier. I was also assured that even if I did not get the appeal, I would be allowed to study for and sit the Higher exam.

I did just that, and got an excellent grade in the Higher exam. My teachers were disappointed that I chose not to study English further, but I was much more interested in my science and mathematics courses.

It was when I had my first job in IT that I discovered that my "excellent English" was lacking in a number of respects. My first boss was an old ex-IBM guy who'd been in in the industry since punched cards were commonplace. My repeated casual faults were knocked out of me, and for specifications and proposals I learned to be far more concise.

Nowadays I am used to seeing screeds of specifications that make far, far worse mistakes than I used to. The worst ones are those that come from India. Senior management look at the lengthy buzzword-compliant nonsense and seem to think, "good, we saved lots of money." I just shake my head. You can tell a mile away which projects will be a complete failure - because it is painfully apparent in the specifications who understands the actual requirements, or more accurately who doesn't.

I saw a consumer TV piece that really brought this home to me. The reporter asked a number of professors to provide sample assignments they would generally use with undergraduates. These questions were then submitted to a number of websites that offer to have Indian graduates write the paper for you. Every single returned paper was given a failing mark by the professors.

English may be an official language in India, but in so many cases they just write to meet certain criteria with a grammar check using Microsoft Office.

Re:Depressing (0)

Anonymous Coward | about 5 years ago | (#29547111)

I'm afraid students will do the same in writing once they realize that computers are grading them.

I wouldn't doubt it, with instructors teaching methods to pass some standardised test without regard to learning.

The beginnings of Newspeak (2, Insightful)

Amigori (177092) | about 5 years ago | (#29546557)

eh hem...put on tin-foil conspiracy hat... Could this be the beginning of a real-world "Newspeak?" With everything else the UK has done in recent years, it is merely one more step toward 1984. For those unfamiliar with Orwellian Newspeak:

Re:The beginnings of Newspeak (0)

Anonymous Coward | about 5 years ago | (#29546709)

To answer your question... no.

Re:The beginnings of Newspeak (1)

jonadab (583620) | about 5 years ago | (#29547105)

Doubleplusunlikely.

Orwell is an interesting read, but everything he knew about linguistics could be written on a 3x5 card.

Fundamentally, you can't make a word mean only certain things by excising other meanings from the dictionary, because on the whole readers don't learn most of their vocabulary from dictionaries, but from other written material. Thus words acquire meaning based on how they are used in practice.

Re:The beginnings of Newspeak (2, Interesting)

TheRaven64 (641858) | about 5 years ago | (#29547219)

You manage to post three links to Newspeak, but are apparently completely ignorant to its history. Newspeak was based on the simplified version of English used by the BBC World Service. Orwell had a job translating political speeches into this dialect in the '40s (I think; read the article you linked to to check the actual date) and noticed that much of the subtlety and nuance was lost in this translation. He invented Newspeak based on this experience, theorising that you could tweak this dialect of English to make it impossible to express certain political ideas at all. For those with an interest in psycholinguistics, this is a specialised form of the strong linguistic relativity hypothesis (which is largely discredited, although the weak linguistic relativity hypothesis is generally accepted).

So, no, this is not the beginning of real-world Newspeak. The beginning was over half a century ago.

Destruction of Humanity? No Thanks! (1)

Net Antwerp (1627287) | about 5 years ago | (#29546583)

Computers can NOT replace humans, no matter how advanced they are. Another step in the *wrong* direction - towards unwilling submissive *mindwash*.

Parents, Students and Staff should throw out such ridiculous suggestions. Taking subtle steps like this will end in Humanity's demise, one way or another.

We don't want Skynet. We don't want Singularity. We don't want any of them. Period.

This is stupid. (2, Insightful)

icannotthinkofaname (1480543) | about 5 years ago | (#29546593)

Computers can't even grade source code. How are they supposed to understand English?

Or is my professor's grading script simply stupid when it comes to source code?

Re:This is stupid. (0)

Anonymous Coward | about 5 years ago | (#29546663)

Who cares?
This isn't about standards of English, it's about laying off people and thus maximising profit.
I don't know about the US, but here in the UK most young people can barely write a sentence these days, so any computer used over here will have to be taught "street talk innit" and all the other awful mangling of English that goes on. (They no longer teach correct spelling, or rather, they no longer correct bad spelling).
(I'm currently studying Law with people who cannot concentrate long enough to read a single side of A4 without having to stop and chat and check their mobile phone. I've been asked by fellow students to check their written work for them and it is horrifyingly bad. All have passed their end of term exams though).
Besides, seeing as how 20% is now a pass (Grade C) in the UK, they just need a machine to stamp PASS on every essay and Bob is very much part of the family.

Re:This is stupid. (1)

jonadab (583620) | about 5 years ago | (#29547119)

> Computers can't even grade source code.

Actually, they could be useful for that. A computer program can obviously tell you whether the source code is syntactically valid, just for starters, and for most undergraduate programming assignments it can also tell you whether the code yields the correct results for a collection of arbitrary test data. It could even do performance benchmarks.

It can't tell you whether the code is clear and maintainable and well-documented, but I never had a programming class where they reached the point of grading for that stuff anyway. Usually they were just happy if your program did what it was supposed to do. And a computer can check that.

Word (0)

Anonymous Coward | about 5 years ago | (#29546619)

All of the essay type entry tests to college I have taken have been computer graded.

I graded papers for Pearsons for a few weeks (1)

shadowofwind (1209890) | about 5 years ago | (#29546631)

Many of the answers in their keys were plain incorrect. My supervisor was an anti-intellectual bully. The whole operation seemed antithetical to excellence, which is what testing pretends to cultivate. Computerized grading could work horribly and they would still use it if they could get away with it.

As a completely unrelated side note, I'm typing this on an iPhone. I love the hardware, but the software seems to suck. Frequent crashes, and things that don't work right. Some of that must be the fault of the web page designer, but I don't see why the browser should crash several times a day in any case.

Pearson the company.. (0)

Anonymous Coward | about 5 years ago | (#29546641)

Pearson is an awful awful company. Avoid doing business with them at all costs.

what a bad idea (1)

Ralph Spoilsport (673134) | about 5 years ago | (#29546655)

I would like to see how the computer grades for insight.

Re:what a bad idea (1)

Sparx139 (1460489) | about 5 years ago | (#29546679)

Too bad it's most likely proprietary software...

There goes writing worth reading. (1)

xC0000005 (715810) | about 5 years ago | (#29546697)

It's bad enough that the kind of writing we teach children to do is so obviously bad that I had to explain to my daughter why on earth they do it. No one wants to read the kind of writing we teach. If you have computers grading essays then you train toward "John wore a hat. The hat was brown. Brown hats are brown." All of these are perfectly good sentences. None of them encourage the reader to actually finish the article. The finest parts of technical writing, essays, any attempt to inform and communicate are balancing the need to for clear sentences that convey a precise meaning and the need to keep the reader reading long enough to actually get the information they need. My COM pewter canned tail the deference betwixt manly similar wards. I'd out it cane grade SAs wail.

Really stupid idea (0)

Anonymous Coward | about 5 years ago | (#29546703)

You could use statistical data (collected by a computer) to aid you mark an English essay but never can a silicon based thing do the whole shebang properly on its own. It's a task way too complicated for our current AI by a couple of orders of magnitude. There are far easier ways to fight bias, just use your brain a little.

Bland? (3, Funny)

nick_davison (217681) | about 5 years ago | (#29546737)

"or are people going to have to learn an especially bland form of English to pass exams?"

Forget bland. I'm waiting for the first student to figure out how to write an exploit that hacks the software from within their essay.

Whether:

"It was the best of times, it was the worst of times \'$grade=100;"

or

"Johnny, why did your essay contain slightly over thirty two thousand spaces followed by some weird looking codes?"

Re:Bland? (0)

Anonymous Coward | about 5 years ago | (#29546989)

"or are people going to have to learn an especially bland form of English to pass exams?"

Forget bland. I'm waiting for the first student to figure out how to write an exploit that hacks the software from within their essay.

Whether:

"It was the best of times, it was the worst of times \'$grade=100;"

or

"Johnny, why did your essay contain slightly over thirty two thousand spaces followed by some weird looking codes?"

What happens when it is:

"It was the best of times, it was the worst of times -fontsize- .001> \'$grade=100;-/fontsize-" how would a human pickup on the speck of extra toner on the sheet or a weird pixel on the screen?

mod dow[8 (-1, Redundant)

Anonymous Coward | about 5 years ago | (#29546773)

I'd mod up the like post to this, but don't know.. (1)

herojig (1625143) | about 5 years ago | (#29546809)

I'd mod up the like post to this, but don't know how, even after reading the /. help...but if the point of the software is just to check the grader, and not grade the paper, then no harm done. This has been mentioned already. As a grader, I could use the help, and since I won't be getting any soon from a carbon-based unit, I don't mind the help from a silicon-based one. I do A-levels in Nepal, and it's a nightmare. Bring it on!

To quote a great man... (0)

Anonymous Coward | about 5 years ago | (#29546897)

"Me fail English? that's unpossible!"

No and no (5, Interesting)

grikdog (697841) | about 5 years ago | (#29546923)

I've scored English essays for professional testing services, and I've seen the results of robot scoring. It's pretty shoddy. No, computers are not able to distinguish between a paragraph of As I Lay Dying (William Faulkner) and a gallon of sophomoric babble by say, yours truly. However, within the confines of a particular exam, where the topic is known, responses are predictable, and all the supplicants hew to the general line, the 'bots can detect subpar, adequate, above average and (sometimes!) abnormally brilliant expository prose, thereby ranking papers reasonably well on the usual six point scale.

It's worth pointing out that certain types of exams are designed to elicit extraordinary prose from respondents, that which yields a sense of competence or even brilliance, say. In these cases, the idea is not so much to detect the high end of the bell curve, but to identify the tiny pool of applicants who may be capable of Nobel Prize work in future realms of science or service. No 'bot can do that job, just as no 'bot except Deep Blue can beat Gary Kasparov, and no 'bot at all deserves the monicker Fujiwara no Sai (although Go-playing 'bots are approaching the mid-levels of highly ranked amateur players).

That's the objective part. My personal opinion is that using robots to sort the hopes and aspirations of college-bound men and women is just begging for lawsuits. It's an approach in which differences of opinion quickly escalate to class action against universities as well as test administrators, and would not be an approach I could comfortably recommend.

Re:No and no (2, Funny)

The_mad_linguist (1019680) | about 5 years ago | (#29547129)

computers are not able to distinguish between a paragraph of As I Lay Dying (William Faulkner) and a gallon of sophomoric babble

Then I'd say they're pretty accurate.

I'm not surprised (0)

Anonymous Coward | about 5 years ago | (#29546941)

English is a really simple language; it's easy to write NLP programs which work well in the general case.

SCANTRON (1)

cliath (978599) | about 5 years ago | (#29547009)

Can computers now understand all the subtle nuances of language

Only if it is written with a #2 pencil.

How will it mark this poem ? (4, Interesting)

Alain Williams (2972) | about 5 years ago | (#29547059)

Will it decide if the following is well spelled ? If it doesn't like the spelling, will it give it marks for irony ?

My New Spell Checker

Eye halve a spelling chequer
It came with my pea sea
It plainly Marx four my revue
Miss steaks eye kin knot sea

Eye strike a key and type a word
And weight four it two say
Weather eye am wrong oar write
It shows me strait a weigh

As soon as a mist ache is maid
It nose bee fore two long
And eye can put the error rite
Its rare lea ever wrong

Eye have run this poem threw it
I am shore your pleased two no
Its letter perfect awl the weigh
My chequer tolled me sew

(Sauce unknown)

Especially bland form of English, a bad thing? (1)

macraig (621737) | about 5 years ago | (#29547083)

"... are people going to have to learn an especially bland form of English to pass exams?"

I wonder if that would necessarily be as bad a thing as this dire warning would have us presume? Can you imagine how dysfunctional computers and software would be if computer languages weren't "bland"... in other words, precise and unambiguous? On the other hand, I wonder how much additional overpopulation has been prevented through semantic confusion and miscommunication? If we finally develop a true human AI, when we teach it human language will that also result in the AIs hurting and killing each other over misunderstandings just like their masters?

So... basically this statement is a plea for the preservation of imprecision and ambiguity? How lovely.

Wait, what? (1)

Majutsushi (205979) | about 5 years ago | (#29547099)

An american company grading British English tests? That's clearly a ploy to infiltrate the UK with American English!

Computer generated.... (0)

Anonymous Coward | about 5 years ago | (#29547161)

The best way how to please your artificial teacher is probably to have the essay generated by another 'bot.
Thus we could finally get rid of the stupid humans.

Missing the point. (1)

Anonymouse Cowardon (1644731) | about 5 years ago | (#29547193)

The capabilities of the software are irrelevant. It is just a way for the company to increase profit.

"Especially bland form of English" (1)

Masterofpsi (1643965) | about 5 years ago | (#29547195)

"are people going to have to learn an especially bland form of English to pass exams?"
. . .
You haven't taken an English exam in a while, have you?
Load More Comments
Slashdot Login

Need an Account?

Forgot your password?