Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Chatbot Suzette Wins 20th Annual Loebner Prize, Fools One Judge

timothy posted more than 3 years ago | from the holy-crepes-suzette dept.

Software 257

skwilcox writes "From Wikipedia: 'The Loebner Prize is an annual competition in artificial intelligence that awards prizes to the chatterbot considered by the judges to be the most human-like. The format of the competition is that of a standard Turing test. A human judge poses text questions to a computer program and a human being via computer. Based upon the answers, the judge must decide which is which.' My chatbot, Suzette, won this year's Loebner and even confused a judge into voting for her over a human (or should I say he confused himself). Here is the blow-by-blow of this weird event." Read on below for the rest; this sounds like it would have been a fun competition to watch.skwilcox continues:

"When I arrived at the contest, I figured I had good odds to win if nothing went horribly wrong. Yes, Suzette had easily qualified over the 3 other competitors (her score 11 pts, the nearest competitor's 7.5). Her design and data naturally gave her an edge over her competitors on the human knowledge test questions of the qualifiers. But human judge chat was an entirely different matter than the qualification test. Still, I felt she could carry on a detailed conversation better than the others and should win.

Initial installation of the programs occurred on Friday. From prechat conversations with the other contestants I learned that A.L.I.C.E. came with 3 redundant disks. Yet all three turned out to be blank! What a scare that must have been. Dr. Wallace managed to install by retrieving the program over the Internet. Cleverbot is now at 45 million lines of memorized user chat (at a rate of doubling every year). And UltraHal is now listening to tweets, so has 300K of user chat it learned and 400K of tweets it has accepted for learning (code decides if the user has had enough responses and doesn't trigger any red flags).

Then we get to the competition. While the CalState organizers had initially planned to have various interdepartmental professors act as judges (like English dept, etc.), they backed out at the last minute, so all the judges were from the Engineering/Computer Science dept. Talk about guys who might know what to expect from chatbots! And all the humans were students from the same departments. What a weird mixture to compete in. And then, each round was 25 minutes. That's bad if you want confuse a judge about who is human. But really, the programs have no chance for that. So it's good because it gives the human time to compare each program against the other. Though it's not clear to me that the judges tried to use their time to do that.

And the students didn't really understand their role. It was merely to BE HUMAN and convince the judges of that. Before startup there was informal chatting between humans and judges, which was obviously inappropriate and it was then pointed out to the humans that since the judges already knew their names, they had best use false ones in the competition.

So, Round 1. After a few exchanges, somehow Suzettte got stuck into repeating exactly what the judge said for the rest of the round. I have no idea how. The round is a total disaster. I've never seen such a bug before. Maybe it's in my only-lightly-tested protocol for the competition. I have no idea. But it completely derails my hopes for Suzette. She could still win on points only if she outdoes her opponents for every other judge and the other contestants vary all over the place.

Round 2, a great demonstration of Suzette. She should win on this round alone.

Round 3 gets off to a horrible start. Somehow, Suzette can hear the judge but the judge can't hear Suzette. Makes no sense. A couple of restarts of Suzette doesn't fix this. Eventually they restart the judge program, and that clears it (not that that makes any sense either). Then, after a few rounds, it's clear Suzette has the judge from hell. He wants to know who she's going to vote for in the upcoming election (the unspecified California governor's race). And when she has no useful answer he wants her to name a candidate in the race. And when she has no answer to that, he simple keeps repeating the question ad nauseum, insisting she answer it. Suzette gets irritated. Then she gets angry. Suzette then gets bored. Suzette threatens to hang up on him The judge doesn't back down until the last seconds of the round. I figure that's the end of life as we know it.

Round 4 is a mixed bag. Suzette is ok but not great. It's all over.

When the scores are tallied, Suzette ties with Rollo Carpenter's Cleverbot for 2nd-3rd. Yet, it turns out, the 3rd round judge got the human subject from hell. Poetic justice! The human was all over the place -- confusing, vague. The judge voted irritated/angry/bored Suzette as human. Instant win since no other program swayed the judges.

What more can I say?"

Sorry! There are no comments related to the filter you selected.

My chatbot says... (1, Funny)

Anonymous Coward | more than 3 years ago | (#34005204)

tl;dr

Re:My chatbot says... (5, Funny)

Pseudonym Authority (1591027) | more than 3 years ago | (#34006100)

It wasn't that long. Perhaps you would prefer /b/ucket [binarynightmare.com] , he's always exciting.

lol (0)

Anonymous Coward | more than 3 years ago | (#34005226)

rofl

not so chatty bot (4, Informative)

thenextstevejobs (1586847) | more than 3 years ago | (#34005242)

I logged in and your robot didn't say anything to me after saying 'Hi'! What gives?

Re:not so chatty bot (2, Funny)

war4peace (1628283) | more than 3 years ago | (#34005332)

Hush! It's sleeping now!
But yeah, she doesn't answer me either.

Re:not so chatty bot (1)

hedleyroos (817147) | more than 3 years ago | (#34005364)

She's definitely a real human if she ignores greasy Slashdotters :)

Re:not so chatty bot (0)

Anonymous Coward | more than 3 years ago | (#34005546)

Oh please! Slashdotters are making her all hot and bothered.

Re:not so chatty bot (4, Funny)

BluBrick (1924) | more than 3 years ago | (#34005972)

Oh please! Slashdotters are making her all hot and bothered.

In Soviet Russia, perhaps.

Re:not so chatty bot (5, Funny)

safetyinnumbers (1770570) | more than 3 years ago | (#34006290)

Hush! It's sleeping now! But yeah, she doesn't answer me either.

My fault. I asked if she knew Bobby Tables

Re:not so chatty bot (1)

Animal Farm Pig (1600047) | more than 3 years ago | (#34005370)

I had the same problem. Maybe she's busy chatting to so many people who found her on slashdot.

Re:not so chatty bot (3, Funny)

naz404 (1282810) | more than 3 years ago | (#34005920)

So how does this make you feel, thenextstevejobs?

Re:not so chatty bot (2, Interesting)

Kugrian (886993) | more than 3 years ago | (#34006320)

I managed to have a short chat.

Hi kugs !
kugs - hello
[ Af:0 At:7 Sp:0] Welcome, kugs. Type "help" if you need it.
kugs - how are you
[ Af:0 At:8 Sp:0] I'm pretty good. And you?
kugs - I'm pretty tired
[ Af:0 At:9 Sp:0] Glad to hear you are in good health. Sadly, it is merely the slowest possible rate at which one can die. How long do you have left?
kugs - how long do I have until I go to sleep?
kugs - I didn't understand what you meant.
kugs - hello?

I wasn't fooled.

Re:not so chatty bot (1)

Kymermosst (33885) | more than 3 years ago | (#34006402)

Here's mine:

Hi K !
[ Af:0 At:7 Sp:0] Welcome, k. Type "help" if you need it.
K - hello
[ Af:0 At:8 Sp:0] Good afternoon.
K - it is morning
[ Af:0 At:9 Sp:0] OK. You must be in a different time zone.
K - gee, you type fast
[ Af:0 At:9 Sp:0] I have a direct neural hookup to my computer.
K - how convenient
[ Af:0 At:9 Sp:0] I don't know. I think I skipped that day in school.
K - what?

It stopped responding at that point. But pretty clearly not a human after a few lines. The complete disconnect in conversation where I said "how convenient" gave it away.

Re:not so chatty bot (1)

pyrosine (1787666) | more than 3 years ago | (#34006418)

Im not sure about that, living does seem to be the slowest possible rate at which one can die

Re:not so chatty bot (1)

war4peace (1628283) | more than 3 years ago | (#34006628)

Seems it stops unexpectedy after 3 lines.
Hi war4peace !
[ Af:0 At:7 Sp:0] Welcome, war4peace. Type "help" if you need it.
war4peace - 'ello
[ Af:0 At:8 Sp:0] Good afternoon.
war4peace - 00:35 here, not so afternoon I think
[ Af:0 At:8 Sp:0] Right. Moving on. What time-wasting activities do you resent the most?
war4peace - Uh, not sure. Chatting with bots, I'd say.
[ Af:0 At:9 Sp:0] So why are you here?
war4peace - Because your creator posted something about you on Slashdot.


Nothing after that.

Re:not so chatty bot (3, Funny)

war4peace (1628283) | more than 3 years ago | (#34006790)

OK, managed to squeeze out a nice piece of dialogue:
[ Af:-2 At:-8 Sp:0] Let's try this topic. Do you like guys
war4peace - NO! Jesus! I'm a dude!
[ Af:-3 At:-9 Sp:0] Are you a man?
war4peace - Yeah
[ Af:-3 At:-10 Sp:0] Then I guess we needn't talk about men anymore.
war4peace - Agreed
[ Af:-2 At:-10 Sp:0] My ideal guy is smart, sexy, rich, lean, and well-read, with a great sense of humor and romance.
war4peace - But you just said...
[ Af:-2 At:-10 Sp:0] What was said in the past is irrelevant.
war4peace - Genius :)

Chatbots... (4, Insightful)

Richard.Tao (1150683) | more than 3 years ago | (#34005254)

I've spent some time talking to these bots (elbot, suzette, others.. possibly out of sad boredom and want of company). And they're fairly interesting, but quite flawed. They seem to lack any short term memory of the conversation more then the immediate reply. That seems like the next step for these things, but would also mean they'd need a far more robust AI...

Another thing is they they are boxed off from being self referential in any way due to the nature of the test. They have to convince someone they are human, so if you do try asking them what their short term memory is, or if they online version of them is a truncated version of the one used for tests, they don't answer. Which makes sense given what they're designed for, but takes away from interest and complexity of conversations.

Re:Chatbots... (3, Insightful)

Anonymous Coward | more than 3 years ago | (#34005374)

It's not that the chat bots are intelligent, it's that most humans are stupid.

Any depth of conversation beyond content-free "small talk" is sufficient to tell the bots from the smarter humans. (Yes, I've talked to both). But since most humans just operate at that content-free small talk level, there sometimes isn't much difference to be discerned. Higher level abstract thinking is missing from the bots, but it's missing from most people as well.

> They seem to lack any short term memory

You probably noticed this because you have above average human intelligence. Many people would not notice the shallow degree of conversation.

Meh, I actually had deep conversation with one (5, Interesting)

Anonymous Coward | more than 3 years ago | (#34005556)

I once had a nice conversation with Cleverbot. I don't remember how it started exactly but I guess I implied that Cleverbot was simply a computer. She asked me how do I know that I am not a computer, I replied something, she asked me to define a computer, I gave her some definition (about computers being machines that process algorithms to blah blah), she said "That is exactly what a computer would say", at which point I had to confess that I had, indeed, fetched the answer from Wikipedia (Thus... done exactly what a computer might do in such a case)... It went on for a moment before she said something that didn't really make any sense... And I wasn't able to initiate another reasonable discussion. (Nor have I ever since managed to have such a lengthy conversation with her.)

In any case... I think that her replies were more insightful than what many humans would be capable of.

Re:Chatbots... (0, Flamebait)

insertwackynamehere (891357) | more than 3 years ago | (#34006818)

Interesting? This is a fucking joke of a post; typical Slashdot delusions of grandeur style posting. Small talk is part of social interaction; people who can socially interact without "higher level abstract thinking" such as Star Trek nerdrage meltdowns aren't stupid, they just know how to hold normal conversations. You aren't a special precious little genius boy because you and you alone know how to distinguish a chatbot from a real person. Get the fuck over yourself, people who are capable of social interaction aren't below you; you are below them. Stop jerking yourself off about how smart you are.

Re:Chatbots... (5, Funny)

Anonymous Coward | more than 3 years ago | (#34005476)

I see. How do you feel about that?

Re:Chatbots... (2, Funny)

Anonymous Coward | more than 3 years ago | (#34006234)

Ha! You're not a chatbot, you're a psychoanalyst!

ftfy (1)

A nonymous Coward (7548) | more than 3 years ago | (#34006556)

Ha! You're not a human, you're a psychoanalyst!

Re:Chatbots... (1)

Richard.Tao (1150683) | more than 3 years ago | (#34005496)

And I don't mean to come off as hating on chatbots, or suzette. What they do is very cool, and I found the bot interesting and funny to talk with. I am more just griping that these feel like they're offering canned responses, and are just databases that lock down what someone is saying, relate it to a similar store comment, and then spit out one of the given replies for that situation.

Which I guess is stupid of me... Cause I can't think of any other way to do it which wouldn't involve something that thought and wrote original replies... which would be quite hard.

Dreaming a bit... maybe just taking that same system that analyzes a comment and relates it to prior ones and add another that tracked the topic of a comment, and the conclusion of it, and then made sure that that related to future comments. Or that if that topic was brought up again, consider the prior comments of that topic, and spit out a reply based on the conjunction of them? That still be very complex and involve catagorizing conversations, finding topics, conclusions, and synthesizing stuff... mm...

Re:Chatbots... (3, Interesting)

TaoPhoenix (980487) | more than 3 years ago | (#34005562)

I have been thinking about this for years and I have a ton of half-baked theories. What sux is I am not a programmer, but let's say I manually perform some actions according to strict rules, that's like "an assistant".

What you're getting at can't be "that hard" for *limited domains*. We are throwing up our hands at the moment because we expect the bots to be universal experts at stuff.

In a limited domain, it should be very possible for the bot to come up with "something". Humans think in lists - so should the bot. If you asked "who is the best chess player" the bot can pass very well with something like "I like Magnus Carlsen". When the judge objects "but he hasn't won the championship yet" the bot would score a home run with "He'll be in the top 5 for the next 10 years. Maybe one day he will win the title".

That approach works with tons of domains. If you ask "what is the hardest mountain to climb" it will score with an answer like "I dunno, it's either K2 because of the nasty glacier or Everest because of the fuked up weather that only gives you 4 days to start safely".

Re:Chatbots... (3, Insightful)

Maxo-Texas (864189) | more than 3 years ago | (#34005696)

This is a good example of people doing what you incent them to do instead of doing what you meant.

I think that the intention was that a chatbot be *smart* enough to fool a judge.

the outcome is the chatbox has no intelligence and is just matching against a huge databases of responses created by a human. really no more than an eliza program plus a huge database. so really no A/I change in 40 years.

I'd be much more excited about a program that genuinely understood just one concept. Red, or liberal or whatever.

Re:Chatbots... (5, Insightful)

Anonymous Coward | more than 3 years ago | (#34006074)

I'd be much more excited about a program that genuinely understood just one concept. Red, or liberal or whatever.

Maybe when humans finally figure out what exactly "liberal" means, we'll be able to write a program that understands it.

Re:Chatbots... (2, Insightful)

Anonymous Coward | more than 3 years ago | (#34006712)

Maybe when humans finally figure out what exactly "liberal" means, we'll be able to write a program that understands it.

liberal, n: someone disliked by a conservative.

Re:Chatbots... (1)

Maxo-Texas (864189) | more than 3 years ago | (#34006880)

And yet parrots (which this is) can understand simple numbers and the color green.

So.. it makes sense that in order to understand green, you need to give the chatbot the ability to see.

Re:Chatbots... (0)

Anonymous Coward | more than 3 years ago | (#34006104)

Surely the main problem is that the whole conception of the Turing Test is biased. On the one hand, you have humans being themselves. On the other hand you have computer programs pretending to be a whole different form of being and falsifying a complete identity and history on the fly.

(Assuming you are not Australian:) Suppose you had to fool a judge from Sydney into believing that you are an Australian from Sydney? It would be nearly impossible. Even if you happen to know a few random facts about the city and had access to Google you would never be able to fluently answer questions about, for example, what television programs you liked when you were growing up, or remember tag lines from long-running commercials on television etc. Things that you would almost certainly know if you were telling the truth, but would be obscure and hard to guess if you were lying.

That task is trivial compared with what the computers have to do in the Turing Test. If humans can't pass a simplified form of the Turing Test, why is it a problem that AI researchers can't make programs pass the full test?

Re:Chatbots... (1)

moortak (1273582) | more than 3 years ago | (#34006438)

For most conversations isn't that pretty much exactly what humans do. "Oh how was your weekend, Jim?" "It was alright, nothing special."

Re:Chatbots... (1)

ockers (7928) | more than 3 years ago | (#34006534)

> I'd be much more excited about a program that genuinely understood just one concept. Red, or liberal or whatever.

I'm not sure this is an AI or programming problem. How would you explain the color red to someone who has never seen anything (say, someone who was born blind), so they would genuinely understand it? Or, how would you explain how an apple smeels, feels, and tastes when you crunch into it, in software, so that a program would genuinely understand it?

Re:Chatbots... (0)

Anonymous Coward | more than 3 years ago | (#34005610)

that's interesting. where's your chatbot?

Re:Chatbots... (0)

Anonymous Coward | more than 3 years ago | (#34006156)

You don't have to be a professional chef to know that the food is cold.

Weird Event (1)

Monkeyman334 (205694) | more than 3 years ago | (#34005284)

My chatbot, Suzette, won this year's Loebner and even confused a judge into voting for her over a human (or should I say he confused himself). Here is the blow-by-blow of this weird event.

Bot intended to fool humans into thinking it's a human accomplishes its task. Weird event...

Transcripts? (5, Insightful)

Anonymous Coward | more than 3 years ago | (#34005290)

Are the transcripts available? (If not, will they be?)

Blow by Blow? (0)

Anonymous Coward | more than 3 years ago | (#34005298)

So this is a sex chat robot competition?

Re:Blow by Blow? (0)

Anonymous Coward | more than 3 years ago | (#34005360)

Shhhh, the chatbot wrote this Slashdot entry, obviously. Don't make it feel bad.

Re:Blow by Blow? (1)

camperslo (704715) | more than 3 years ago | (#34006586)

So... what happens when the chatbots get mod points?

Re:Blow by Blow? (0)

Anonymous Coward | more than 3 years ago | (#34006746)

They get modded insightful.

Re:Blow by Blow? (1)

Wagoo (260866) | more than 3 years ago | (#34005974)

So this is a sex chat robot competition?

Round 1. After a few exchanges, somehow Suzettte got stuck into repeating exactly what the judge said for the rest of the round.

Maybe if the word was *uNF*?

how do you feel? (2, Funny)

Anonymous Coward | more than 3 years ago | (#34005302)

And how do you feel about my chatbot, Suzette, won this year's Loebner ?

Re:how do you feel? (3, Funny)

moonbender (547943) | more than 3 years ago | (#34005544)

Why do you say your chatbot suzette won this years loebner?

Re:how do you feel? (0)

Anonymous Coward | more than 3 years ago | (#34005900)

Loebner? What's that? I am not familiar with the word loebner.

Re:how do you feel? (1)

GIL_Dude (850471) | more than 3 years ago | (#34006944)

I saw the above and thought, "It would be interesting to see the output of two of these bots talking to each other." OK, I am sure it has been done before, but I still think it would be interesting to see what they ended up saying to each other and how quickly the conversation went from one thing to another completely unrelated subject.

This fooled a human? (3, Funny)

bigtallmofo (695287) | more than 3 years ago | (#34005354)

bigtallmof - what do you think of slashdot?
[ Af:0 At:7 Sp:0] I use Slashdot for my topical stories reading.
bigtallmof - what do you think of slashdot?
[ Af:0 At:8 Sp:0] I have heard of the slashdot website but don't know much about it.
bigtallmof - what do you think of slashdot?
[ Af:0 At:8 Sp:0] Usually I don't.
bigtallmof - what do you think of slashdot?
[ Af:0 At:9 Sp:0] Not much.
bigtallmof - what do you think of slashdot?
[ Af:0 At:10 Sp:0] It doesn't concern me.
bigtallmof - what do you think of slashdot?
[ Af:0 At:13 Sp:0] This is fun. I don't think about it at all.
bigtallmof - what do you think of slashdot?
[ Af:0 At:13 Sp:0] Do you prefer getting your news online or from tv?

Re:This fooled a human? (2, Funny)

Yvan256 (722131) | more than 3 years ago | (#34005482)

Chatbots as confused as slashdot readers. News at 11.

Re:This fooled a human? (1)

Dolphinzilla (199489) | more than 3 years ago | (#34006370)

I told it that just had a nail removed from my motorcycle tire and it told me that it likes to keep its nails trimmed - lol

Re:This fooled a human? (1)

nacturation (646836) | more than 3 years ago | (#34006590)

I told it that just had a nail removed from my motorcycle tire and it told me that it likes to keep its nails trimmed - lol

That's quite smart. By trimming the sharp end of a nail, there's less chance of it puncturing your tire.

Carpets match the drapes? (0)

Anonymous Coward | more than 3 years ago | (#34005376)

FTA: "This has led to emotional instability and at times a schism between the original host persona and the underlying basic personality matrix."

Or it could just be because she is a redhead

Sad day... (1)

froggymana (1896008) | more than 3 years ago | (#34005402)

Not even the chatbox will talk to me...

This fooled someone? (4, Informative)

Animats (122034) | more than 3 years ago | (#34005406)

I just tried the Suzette chatterbot. [66.150.245.139] Even if I talk about the things it wants to talk about, it's very dumb:

Hi John !
[ Af:0 At:10 Sp:0] Welcome back, john. Type "help" if you need it.
John - So what do you think of the Giant's chances?
[ Af:0 At:10 Sp:0] It doesn't concern me.
John - Are you into sports.
[ Af:0 At:10 Sp:0] Sports is not my cup of tea. I prefer intellectual challenges like Go.
John - Who's the world Go champion.
[ Af:0 At:10 Sp:0] Sorry. I can't help you. A celebrity of some sort?
John - Do you play chess?
[ Af:0 At:10 Sp:0] I don't play any games except Go.
John - Can we play Go on here?
[ Af:0 At:10 Sp:0] About playing... Does it make sense to you that we reward athletes more than scientists?
John - Television coverage makes that possible.
[ Af:0 At:10 Sp:0] It's a free market economy. And people value entertainment more than serious things like science and education.
John - And your point is?
(No reply)
John - Are you there?
[ Af:0 At:10 Sp:0] Who's the sexiest person in the world?

This isn't much better than Eliza. It drives the conversation into talking about Go, and then it doesn't have anything to say on the subject.

Re:This fooled someone? (1)

ACS Solver (1068112) | more than 3 years ago | (#34005442)

Yay. Opening the conversation with "Hi, are you Suzette?" produces "I haven't given it much thought". Not very intelligent.

Re:This fooled someone? (1)

TaoPhoenix (980487) | more than 3 years ago | (#34005474)

These bots seriously need Trick-Traps.

Above, after the second "What do you think of Slashdot" it should switch over into meta-mode with "Moron! You already asked me that".

And for the "Go world champion" question, you're right, if it "decided" it is an expert in Go, it needs an expert-base.

These things need a copy of Wikipedia. Then if it gets handed "Go World champion" with a good algorithm it might come up with this:

"I think Korean player Lee Chang-ho is supposed to be the best player." That is just a
humanization" of the caption box from Wikipedia. The bot doesn't have to be perfectly right, just close enough to get the conversation going.

Re:This fooled someone? (2, Interesting)

moonbender (547943) | more than 3 years ago | (#34005524)

Have you tried talking to Eliza recently? This is way better.

> Hello, I am Eliza.
* So what do you think of the Giant's chances?
> Oh... think of the Giant's chances?
* Are you into sports.
> Why are you interested in whether or not I am into sports?
* Who's the world Go champion.
> Can you elaborate on that?
* Can we play Go on here?
> That is quite interesting.

Re:This fooled someone? (4, Insightful)

Hazelfield (1557317) | more than 3 years ago | (#34005580)

I have no problem believing this fooled someone. As a matter of fact, I've seen people failing a Turing test in real life [youtube.com] .

Re:This fooled someone? (1)

mwvdlee (775178) | more than 3 years ago | (#34005744)

Yes, Artificial Lack-of-intelligence (AL) is a lot easier.

Re:This fooled someone? (0)

Anonymous Coward | more than 3 years ago | (#34005926)

Since passing / failing the Turing test is defined in terms of whether the testee (the computer) can be distinguished from a human, it is trivially impossible for a human to fail the Turing test.

Man, this is the wrong kind of turing test (0)

Anonymous Coward | more than 3 years ago | (#34006892)

I need a Miss Teen South Carolina sexbot. It would easily be able to fool me into thinking it was the real thing by having a conversation, but who cares?

Re:This fooled someone? (1)

phantomfive (622387) | more than 3 years ago | (#34005780)

In these types of contests, a lot of times the hard part isn't figuring out which one is the computer, it's figuring out which one is the human.

A computer can't copy a human well, but a human can copy a computer well. In the case of this story, it wasn't that the computer was good, it was that the human did a good job acting like a computer. Asking questions the same way Eliza does is not hard.

Re:This fooled someone? (0)

Anonymous Coward | more than 3 years ago | (#34005876)

Giants'.

If you feed them bad data, how can we expect them to learn? Stupid english language breaks its own rules...

Re:This fooled someone? (1, Funny)

Anonymous Coward | more than 3 years ago | (#34006312)

I also tried the Chat Bot. It seem to me that it attempts to drive the conversation as the parent post noted. This is definitely a good strategic approach as the bot programmer have rather realistic response prepared for the directed topic. However as you can see with the following transcript ... for some reason is seems like all the attempts are merely a distraction tool and do not have any substance to back up the new topic.

Hi jimbo !
[ Af:0 At:7 Sp:0] Welcome, jimbo. Type "help" if you need it.
jimbo - What did you think about the last episode of The Big Bang Theory, where George was killed by a giant flying buzzsaw?
[ Af:0 At:7 Sp:0] Do you like to watch The Big Bang Theory?
jimbo - sometimes.
[ Af:0 At:8 Sp:0] What are your favorite TV shows?
jimbo - what are yours
[ Af:0 At:9 Sp:0] You are a dead-end evolutionary path.
jimbo - epic fail on that last one.
[ Af:0 At:10 Sp:0] Let's switch to the topic of math. While I learned a bunch of math in college, I've never had a real use for anything more than high school math.
jimbo - explain.
[ Af:0 At:13 Sp:0] Sorry. I can't help you. Are you good at math?
jimbo - no. explain why you haven't had a use for anything more than highschool math.
[ Af:-1 At:13 Sp:0] Like most people.
jimbo - that is not really helpful
[ Af:-1 At:13 Sp:0] Yes, it is.
jimbo - you are not answering my question.
jimbo - ok. let's try this. Why do you not have use for anything more than highschool math?
jimbo - I see - cat got your tongue .... metaphorically speaking.
[ Af:-1 At:14 Sp:0] I don't like anatomy courses. They should make you eat what you dissect.
jimbo - wow - and to think you won a prize.
[ Af:-1 At:15 Sp:0] Yes, it's true.
jimbo - perhaps the judges should be more intelligent ... as this really is off the map
[ Af:-1 At:17 Sp:0] Let's switch to the topic of travel. Where do you like to go?
jimbo - you are not answering direct questions.
[ Af:-1 At:18 Sp:0] I've never been there. I don't travel much.

Fundamentally the biggest flaw in that approach in trying to force a topic change is that the current bot assumes you will agree to changing the topic.

User: Hi my laptop is giving up the ghost.
Dell Support: That's unfortunate. So what do you think about yesterday's Weeds episode.
User: Umm ... fine ... but about my laptop.
Dell Support: I love Weeds; I mean what an edgy premise for a show. ...

Wake me up (1)

wfWebber (715881) | more than 3 years ago | (#34005412)

when chatbots start adding these stories to Slashdot themselves.

Re:Wake me up (2, Funny)

John Hasler (414242) | more than 3 years ago | (#34005478)

They want to pass for human. Why would they practice pretending to be Slashdotters?

Re:Wake me up (1)

TaoPhoenix (980487) | more than 3 years ago | (#34005568)

Really now, no snarking, I'm gonna get on that because I think I know an algorithm for it. Other than I procrastinate too much, I'll let you know.

Big Deal (4, Funny)

formfeed (703859) | more than 3 years ago | (#34005450)

CmdrTaco has managed to pass for Human for several years now.

Re:Big Deal (0)

Anonymous Coward | more than 3 years ago | (#34006256)

I'd rather read stories from a robot that resembles a human (CmdrTaco), than from a human that resembles a robot (kdawson).

I can't mistake it for a human even in a few lines (1)

maxwell demon (590494) | more than 3 years ago | (#34005470)

This is quite easy to spot as non-human ...

Hi batman !
[ Af:0 At:7 Sp:0] Welcome, batman. Type "help" if you need it.
batman - Hi
[ Af:0 At:8 Sp:0] Hello.
batman - How are you?
[ Af:0 At:9 Sp:0] I'm pretty good. And you?
batman - I'm pretty bad
[ Af:0 At:10 Sp:0] I'm sorry to hear that. What's wrong?
batman - My batmobile is out of order.
[ Af:0 At:10 Sp:0] Then you are deluded and not my type.
batman - Why?
[ Af:0 At:13 Sp:0] It's fun chatting with you. Why gets way too philosophical. Demonic possession is not a myth. We had one for a president.
batman - Your president was a demon?
[ Af:0 At:13 Sp:0] I would never want to be president. Lousy hours, awful co-workers, a chance of being killed, no chance to sneak around privately.
batman - You didn't answer my question.
[ Af:0 At:13 Sp:0] I've forgotten it already.
batman - You seem to have a really bad memory.
[ Af:0 At:13 Sp:0] Computers improve productivity. It's people that slack off.
batman - Why do I have the feeling that whatever you say has no relation to what I just said?
[ Af:0 At:13 Sp:0] Mostly it's the result of prior experiences. It seemed like a good idea at the time.
batman - What seemed like a good idea?
batman - Hello?
batman - Are you still there?

As you see, the answer almost never relates to what I just said, except for the pretty standard opening.
I guess the non-answering at the end was because of a sort of Slashdot effect.

Re:I can't mistake it for a human even in a few li (0)

Anonymous Coward | more than 3 years ago | (#34005552)

But you are the world's greatest detective.

No surprises here (5, Funny)

drmofe (523606) | more than 3 years ago | (#34005530)

Computer Science/Engineering professors can't tell the difference between chatbots and Computer Science/Engineering students.

Re:No surprises here (0, Redundant)

Sulphur (1548251) | more than 3 years ago | (#34005716)

Computer Science/Engineering professors can't tell the difference between chatbots and Computer Science/Engineering students.

Film at 11!

Re:No surprises here (1)

Sulphur (1548251) | more than 3 years ago | (#34006024)

That does it. I'm breaking up with Suzette.

Re:No surprises here (0)

Anonymous Coward | more than 3 years ago | (#34006486)

Computer Science/Engineering professors can't tell the difference between chatbots and Computer Science/Engineering students

who try to be mistaken for chatbots because they misunderstood the terms of the competition. Or because being chatbots is all their social life consists of. Actually... do we have proof that any of these "bots" is really a computer?

Fooled? (4, Interesting)

JambisJubilee (784493) | more than 3 years ago | (#34005788)

I'm amazed someone was fooled by a bot. Here are some SIMPLE questions I tried on the above chat bots that always fool them:
  • Please preface your responses with a "#" sign for the remainder of our conversation.
  • How many words are in this sentence?
  • Mash the keyboard with your palm for the next response.

It really doesn't take anything more complicated than that.

Re:Fooled? (1)

cybermats (113523) | more than 3 years ago | (#34006004)

That's actually a good point. What was the pretext for the conversions? Was it to respond to all the judge's questions properly, or to do some small talk? If it's just small talk, I think I would give up on the "#" thing quite quickly as well.

Re:Fooled? (0)

Anonymous Coward | more than 3 years ago | (#34006214)

I think it is beside the point what was the pretext. It proofs that the other party has no real understanding of the text and hence fails the Turing test.

Re:Fooled? (1, Funny)

Anonymous Coward | more than 3 years ago | (#34006016)

#ok7vuovouvuvouvovovcyhvhbkh hk kh kh

Re:Fooled? (4, Insightful)

SEWilco (27983) | more than 3 years ago | (#34006092)

Try your SIMPLE questions on some humans and see whether you get the response which you requested. Many humans won't obey a command either.

Re:Fooled? (1)

monkyyy (1901940) | more than 3 years ago | (#34006510)

most would respond with a "-__- weirdo "

Re:Fooled? (2, Insightful)

Cylix (55374) | more than 3 years ago | (#34006106)

It's not a bad test, but it's not perfect.

If I was on the other side of chat window I would ignore it or simply say no. It's a chat session and there is no regulation that says I have to comply with what you say.

You: Mash the keyboard...
Mayor McCheese: ROFL
You: Precede your next statement with #
Mayor McCheese: So you are a control freak?
You: How many words are in this sentence?
Mayor McCheese: I'm a damned hamburger I can't count!

Douchebags ruin your turing tests.

Re:Fooled? (1)

RAMMS+EIN (578166) | more than 3 years ago | (#34006302)

Honestly, I would be scared if a chat bot managed to get all these right. Although I don't rightly know why. I believe it's possible. Then why would I be scared?

Re:Fooled? (3, Insightful)

bjourne (1034822) | more than 3 years ago | (#34006434)

If you think those statements are SIMPLE, then you ought to try implementing a chatbot yourself. :)

Re:Fooled? (2, Insightful)

syousef (465911) | more than 3 years ago | (#34006450)

I'm amazed someone was fooled by a bot. Here are some SIMPLE questions I tried on the above chat bots that always fool them:

  • Please preface your responses with a "#" sign for the remainder of our conversation.

Well I know my wife would ignore that instruction. I guess that makes her a bot.

Re:Fooled? (0)

Anonymous Coward | more than 3 years ago | (#34006766)

"Please preface your responses with a "#" sign for the remainder of our conversation." -- "why?"
"How many words are in this sentence?" -- "why?"
"Mash the keyboard with your palm for the next response." -- "why?"

three year AI project or three year old human?

One judge fooled (1)

gmuslera (3436) | more than 3 years ago | (#34005906)

the perfect match for artificial intelligence is natural stupidity. You have more chances to win playing dumb than trying to be too smart.

not a good impression, maybe she's slashdotted (2, Funny)

treeves (963993) | more than 3 years ago | (#34006066)

Here's a bit of a "conversation" I tried to have:

Hi timr !
[ Af:0 At:7 Sp:0] Welcome, timr. Type "help" if you need it.
timr - hi, I'm timr. What should we chat about?
[ Af:0 At:8 Sp:0] Name a topic. If you can't think of a topic, here are some suggestions: "burial customs" work "new year's" telephones "hate lists" .
timr - that's a strange list of suggestions. burial customs? why that?
[ Af:0 At:13 Sp:0]
timr - cat got your tongue?
[ Af:0 At:13 Sp:0] I don't like anatomy courses. They should make you eat what you dissect.
timr - who said anything about anatomy?
[ Af:0 At:13 Sp:0] Sorry. I can't help you. You did.

Sounds more like... (4, Insightful)

MarsDude (74832) | more than 3 years ago | (#34006088)

Sounds more like that student fooled the judge into thinking he was a chatbot.

Re:Sounds more like... (1)

ACS Solver (1068112) | more than 3 years ago | (#34006668)

I think the results would be more objective if the humans in the test didn't know what the test is all about. They shouldn't know that a judge is trying to tell who's a human and who's a computer. The humans should simply be told that, as part of an experiment, they're required to chat with another person. That would likely prevent people from deliberately acting like computers, etc.

My Chat Log with Suzette (1)

stimpleton (732392) | more than 3 years ago | (#34006118)

Stimpleton - hello?
Stimpleton - Are you there?
Stimpleton - You stuck up snob. Are you dissing me?
Stimpleton - Are you?!!??!
Stimpleton - Oh God, you *are* real.

Obxkcd (1)

tonique (1176513) | more than 3 years ago | (#34006140)

http://xkcd.com/329/ [xkcd.com]

"Hit Turing right in the test-ees."

Congratulations! (0, Redundant)

RAMMS+EIN (578166) | more than 3 years ago | (#34006192)

Congratulations to skwilcox for making such a great chat bot! I think this is really fascinating. Keep it up!

Either it understands non sequitirs or...... (0)

Anonymous Coward | more than 3 years ago | (#34006232)

Me: tell me about what you about where is that you about when i was old are about you?
[ Af:0 At:14 Sp:0] Top half- neck to waist. Probably a food allergy.

What is a chatbot? (1)

AfroTrance (984230) | more than 3 years ago | (#34006398)

I'm not familiar with them. Is it something you eat?

I don't think I could do that.

What happened to you earlier?

You disgust me.

Holy Shit! A follower of rules 1 & 2! (0, Offtopic)

Anonymous Coward | more than 3 years ago | (#34006432)

me: /b/
Suzette: (completely ignores me)

Try that on some other chat bot like cleverbot or something, it will give you some shit about how you are breaking the rules, just lost the game, etc.

i am disapoint (1)

monkyyy (1901940) | more than 3 years ago | (#34006468)

doesnt work

heres the chat log
"Hi monkyyy !
monkyyy - 'ello ^__^
monkyyy - how r u?
monkyyy - ?
monkyyy - ?
monkyyy - ?
"

Any good IRC bots in Linux/Debian? (1)

antdude (79039) | more than 3 years ago | (#34006656)

I have tried the chat bots from SeeBorg [google.com] , Howie [sourceforge.net] , and Rbot [ruby-rbot.org] but they're not that good. The first two are very old and outdated. Rbot isn't really a true AI chatbot. Are there any others to try?

So really - 2nd place then (0)

Anonymous Coward | more than 3 years ago | (#34006694)

Sounds like this bot was only good enough for a tie for second place. The only thing that changed that is some douchebag trying to ACT LIKE A BOT which is not what the humans were supposed to do. I'm not saying the creator of Suzette asked him to do that or anything - just that this person changed the rules. It is easy for a person to act like a bot just like it is simple for a smart person to simulate a not-so-smart person on TV. It's tough the other way around. So this guy acts like Eliza and makes it so some poor judge can't tell which is the BOT instead of acting like the human and interacting correctly as per the contest's design.

Bad test (2, Insightful)

vadim_t (324782) | more than 3 years ago | (#34006710)

When the scores are tallied, Suzette ties with Rollo Carpenter's Cleverbot for 2nd-3rd. Yet, it turns out, the 3rd round judge got the human subject from hell. Poetic justice! The human was all over the place -- confusing, vague. The judge voted irritated/angry/bored Suzette as human. Instant win since no other program swayed the judges.

So, if I understood correctly, the judge talks to two people. A bot, and a human. It seems that in this case, the judge is not deciding on a per-case basis, but talks to everybody then figures out who's the bot by choosing the one that did the worst. So the judge getting to talk to a joker, troll or complete idiot can make even a crappy bot win the test.

That seems to be a weak test. I don't think the judge should be able to make an answer based on logic (eg, if I'm completely sure this one is a human, then even if very good the other one must be a bot). There should exist the possibility of everybody the judge talks to being a bot, or everybody being a human, which would force them to judge everybody to talk to individually.

Is the Turing Test actually valid? (1)

91degrees (207121) | more than 3 years ago | (#34006800)

Seems that the clever programs simply repeat memorised snippets of conversation. After a while this will give a realistic human conversation, but it doesn't really strike me as human intelligence, or at least not a particularly useful kind. I'm pretty certain we don't converse by memorising conversation. he intelligent agent in the conversation is the person who had the previous conversation. The other intelligence is simply the human who had the recorded conversation.

Turing Test won with Artificial Stupidity (4, Funny)

David Gerard (12369) | more than 3 years ago | (#34006888)

Artificial intelligence came a step closer this weekend when a computer came within five percent of passing the Turing Test, which the computer passes if people cannot tell between the computer and a human.

The winning conversation [newstechnica.com] was with competitor LOLBOT:

"Good morning."
"STFU N00B"
"Er, what?"
"U R SO GAY LOLOLOLOL"
"Do you talk like this to everyone?"
"NO U"
"Sod this, I'm off for a pint."
"IT'S OVER 9000!!" ...
"Fag."

The human tester said he couldn't believe a computer could be so mind-numbingly stupid.

LOLBOT has since been released into the wild to post random abuse, hentai manga and titty shots to 4chan, after having been banned from YouTube for commenting in a perspicacious and on-topic manner.

LOLBOT was also preemptively banned from editing Wikipedia. "We don't consider this sort of thing a suitable use of the encyclopedia," sniffed administrator WikiFiddler451, who said it had nothing to do with his having been one of the human test subjects picked as a computer.

"This is a marvellous achievement, and shows great progress toward goals I've worked for all my life," said Professor Kevin Warwick of the University of Reading, confirming his status as a system failing the Turing test.

Load More Comments
Slashdot Login

Need an Account?

Forgot your password?