Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Creating a Better Chatbot Through Crowdsourcing

samzenpus posted more than 2 years ago | from the people's-bot dept.

AI 49

An anonymous reader writes "MIT Technology Review reports on a chatbot built at the University of Rochester that is capable of high quality, human-level conversation — thanks to software called Chorus that turns to Amazon's crowdsourcing service Mechanical Turk to generate and evaluate replies to a human's statements and questions. No one person is ever acting as the bot, instead multiple workers suggest responses that are then voted on to select the best. The crowd workers contributing change frequently, but Chorus also has them keep a running list of important contextual information to give the bot a kind of memory of a conversation's history. The researchers say Chorus-style chat bots could out-perform fully automated assistants such as Siri, while being considerably cheaper than a true concierge service."

cancel ×

49 comments

Sorry! There are no comments related to the filter you selected.

BREAKING NEWS (5, Funny)

Anonymous Coward | more than 2 years ago | (#41293349)

Humans better at chatting than computers, and by replacing computers with humans, you can almost have a natural conversation.

Re:BREAKING NEWS (4, Funny)

RottenJ (2060834) | more than 2 years ago | (#41294047)

Personally, I am waiting for a passable vocal chatbot to troll telemarketers.

Re:BREAKING NEWS (3, Interesting)

Gordonjcp (186804) | more than 2 years ago | (#41294135)

Get a small child who is just about old enough to understand that telephones are for talking to people. They love to talk to *anyone* on the phone.

Did you know that if telemarketers hang up on a call, it counts against them in their call stats?

Re:BREAKING NEWS (1)

camperdave (969942) | more than 2 years ago | (#41294399)

Personally, I am waiting for a passable vocal chatbot to troll telemarketers.

Um... these chatbots are going to be the telemarketers.

Re:BREAKING NEWS (0)

Anonymous Coward | more than 2 years ago | (#41295925)

Good. The two of them can have a conversation and leave us alone.

i read a lot about that bot. (-1)

Anonymous Coward | more than 2 years ago | (#41293377)

i think it was here... http://amzn.to/Pc9LDE

Creating an earlier post through FrostPosting (-1, Troll)

Anonymous Coward | more than 2 years ago | (#41293383)

"AC Technology Review reports on a frostpost built at the University of Linux that is capable of high quality, human-level posting— thanks to software called Anus that turns to Goatse's crowdsourcing service Mechanical Penis to generate and evaluate replies to a human's posts and tardiness. No one person is ever acting as the poster, instead multiple posters suggest responses that are then voted on to select the widest anus. The posters contributing change frequently, but Anus also has them keep a running list of important rectal information to give the bot a kind of memory of a conversation's gaping asshole. The researchers say Anus-style posters could out-perform fully automated assistants such as Dr. Stephen Jobs, while being considerably cheaper than a true transexual hooker or iPhone."

crowdsourcing != AI (3, Insightful)

metageek (466836) | more than 2 years ago | (#41293393)

Good try, but even if this passes the Turing test, it is not AI in any way. Responses are by humans and there is no intelligence in it. So it will be the collective human engine behind it that will pass the test. Not really any big achievement.

Mod parent up. (3, Insightful)

khasim (1285) | more than 2 years ago | (#41293469)

And if this ever takes off you know that it will become a game of trying to submit/vote the most inappropriate responses to the questions. A never ending battle between the censors forbidding words and phrases and the people finding new ones.

Mod parent sideways (4, Funny)

Quiet_Desperation (858215) | more than 2 years ago | (#41293597)

User: Where's the nearest pizza place?
Chorus: Penis.
User: Excuse me?
Chorus: YOLO

Re:Mod parent up. (0)

Anonymous Coward | more than 2 years ago | (#41298063)

And if this ever takes off you know that it will become a game of trying to submit/vote the most inappropriate responses to the questions.

Only weak cranes do not fly.

Re:crowdsourcing != AI (1)

Anonymous Coward | more than 2 years ago | (#41293553)

The summary never mentions AI or the Turing test at all so you're attacking a strawman.

It's not like they're claiming to have made an AI breakthrough or something. They're just saying "I made a chatbot out of humans. Perhaps it may be of some use."

Re:crowdsourcing != AI (1)

metageek (466836) | more than 2 years ago | (#41293641)

If you follow the link on the summary you find an article with title "Artificial Intelligence, Powered by Many Humans"

Re:crowdsourcing != AI (0)

Anonymous Coward | more than 2 years ago | (#41293687)

Well, yeah, but why would I read the article when I get more value for my time by reading and participating in the comments?

Re:crowdsourcing != AI (1)

drcheap (1897540) | more than 2 years ago | (#41295993)

If you follow the link on the summary you find an article with title "Artificial Intelligence, Powered by Many Humans"

That about sums up my daily experiences...
Everywhere I go there are many humans with an overwhelming lack of real intelligence.

At least someone has finally found a use for their collective stupidity such that the whole is greater than the sum of the parts.

Re:crowdsourcing != AI (2)

Katatsumuri (1137173) | more than 2 years ago | (#41293969)

There can still be some value in this for advancement of AI and hybrid systems. They decompose the problem of keeping up a conversation into nice simple subtasks with clear interfaces. Some of these subtasks (suggesting replies, evaluating them, keeping notes) can then be further automated or assisted independently to a varying degree, gradually reducing the use of human brainpower. Also, there can be uses for adding such crowdsourced conversation support into otherwise automated systems. Perhaps entertainment robots or something like that.

Not even useful (1)

Okian Warrior (537106) | more than 2 years ago | (#41294033)

Good try, but even if this passes the Turing test, it is not AI in any way. Responses are by humans and there is no intelligence in it. So it will be the collective human engine behind it that will pass the test. Not really any big achievement.

Totally agree.

What's the point of this? Is it to make a better chatbot? Do chatbots have any use beyond annoying chatroom participants and surreptitiously inserting ads?

Is it to study human language? Will useful insights come from this? Will there be results? A publishable algorithm? Will this inform future software packages?

Does this tell us something about AI? Can this bring us closer to machines which actually think?

I'm not even sure there is a purpose here. I don't see any academic value in this project. Perhaps someone more familiar with mainstream AI could shed some light.

How much is a sentence of conversation worth? (1)

Iamthecheese (1264298) | more than 2 years ago | (#41293455)

I can't imagine anyone fluent in the language being willing to work for the prices that could be paid. We're talking pennies per hour.

Re:How much is a sentence of conversation worth? (1)

Anonymous Coward | more than 2 years ago | (#41293655)

But I your question can answer good! Pennies please do be sending to me. My starve childs you help?

Re:How much is a sentence of conversation worth? (3, Interesting)

CRCulver (715279) | more than 2 years ago | (#41293829)

Five or six years ago one could find decent work on Mechanical Turk. I used to do podcast transcriptions for around US$10/hour, and some of the podcasts were on topics of interest to me, so it was enjoyable. The ability to do such work remotely made for some good times sitting on beaches in various backpacker hideaways, where the money from a couple of hours of work a day was more than enough to pay one's travel costs. Eventually I got better, more dependable work and stopped logging into MTurk. When I visited it again a couple of years later, I noticed that the money now paid for such tasks is miniscule. It really became a race to the bottom. Even if the money offered was enough for people in the Third World, surely people with the English language skills required could find something better. It's unclear to be just what demographic MTurk is depending on now.

Re:How much is a sentence of conversation worth? (1)

stephanruby (542433) | more than 2 years ago | (#41297483)

Even if the money offered was enough for people in the Third World, surely people with the English language skills required could find something better. It's unclear to be just what demographic MTurk is depending on now.

They're mostly doing it for ratings these days, not actual money.

I'm surprised a highly rated Slashdot user such as yourself didn't pick up on that.

This is dumb (0)

Anonymous Coward | more than 2 years ago | (#41293471)

Read the subject line

OK, let's test this... (0)

Anonymous Coward | more than 2 years ago | (#41293475)

What are you wearing?

It's been done...Cleverbot is a fake (1)

ilikenwf (1139495) | more than 2 years ago | (#41293483)

Cleverbot isn't AI as it's claimed, and I think it's interesting how they try and pass it off as such. Maybe there's a bit of AI there, but from what I've seen it just connects two random users and then reshuffles every so often so you're getting responses from some other real person than the one before...

Re:It's been done...Cleverbot is a fake (1)

Verdatum (1257828) | more than 2 years ago | (#41293911)

It feels that way since it has only like 2 lines of conversation memory/context, but to be sure, it is recording answers. If you repeatedly ask the same question, you can get the same answer twice. You can also check out things like its extensive knowledge of Pokemon. Random users are only going to occasionally know how to respond to Pokemon queries. Cleverbot pretty much always does.

Re:It's been done...Cleverbot is a fake (1)

someones (2687911) | more than 2 years ago | (#41294901)

oh, come on!
EVERYONE played pokemon when we were young, when there were only 151...

Re:It's been done...Cleverbot is a fake (0)

Anonymous Coward | more than 2 years ago | (#41295769)

152. MissingNO.

Re:It's been done...Cleverbot is a fake (1)

Klinky (636952) | more than 2 years ago | (#41296455)

I don't know about that...

Me:What are the names of team rocket?
Cleverbot: Sufian Stevens and Elvis Prestly.

Kinda Like ThinkMosaic chatbot (1)

Anonymous Coward | more than 2 years ago | (#41293521)

I haven't made mine live as I keep tweaking the code but I've made many generations of chatbots. The concept I wanted (and still want) to open up to all is ThinkMosaic. It doesn't have the vote for the best response.. More like, chat with it and then if you don't like the response then edit the logic by which it chose that response... Basically, there is a user statement that is best-matched to a "recognizer" (a text pattern with named wildcards optionally in it). This identifies a "meaning". Each meaning may have multiple "reactions". Each reaction has a "condition", some and, or, not, with parenthesis logic and an "action sequence". The action sequence is a stream of commands including:
* say ".." // this obviously responds with a textual (and/or html subset) message.
* remember ".." // this stores a text message in "memory" which is a mainly just an array of strings
* forget ".." // this deletes memories entered via the remember command
* interpret as ".." // this acts as if the text (..) was entered as a user statement (can have multiple of these, of course)
* expect ".." as ".." // this sets up so if the first ".." is matched in the next user statement then it is interpreted as if the user says the second ".."

Wildcards are named by putting them in brackets (.e.g. I love [food] for dinner). And those can be then used in the command strings listed above (including the forget one).

So it's easy to use and works well.. I've made many iterations of different command sets and this one seems very practical and powerful. But it can also be extended with plugins to add new commands, like one I made to grab sections from wikipedia pages.

Cool.. huh... I think so.. I'll have it up maybe in a month or two.. I development version is up but has quirks right now at http://thinkmosaic.com

Re:Kinda Like ThinkMosaic chatbot (1)

msauve (701917) | more than 2 years ago | (#41293635)

The requested response format ("A system error occured. No response is possible until the error has been fixed.") is not supported.

Hey, it's working great! I have people use that line as a conversation starter all the time.

Re:Kinda Like ThinkMosaic chatbot (0)

Anonymous Coward | more than 2 years ago | (#41296661)

Having to be edited by the user is NOT works well in any meaning of the word. The idea of a chatbot is that it should be seamless to the end user, the person chatting with it. If you remove that constraint then the only use for a chatbot is to play with for the programmer himself. I.e, it's your own programmatical masturbation. Which is fine, but don't expect anyone else to care or want to see.

Replacement for Siri? (1)

Plumpaquatsch (2701653) | more than 2 years ago | (#41293619)

"I feel like some Mexican food." - 10 minutes later: "40% of us say that there should be a Taco Bell somewhere around. Just look for the bell sign"

Not a bot (1)

Hentes (2461350) | more than 2 years ago | (#41293633)

Of course chatting with real people will feel like chatting with real people. How is this different from, for example tech support where you get connected to lots of random people?

Re:Not a bot (1)

houstonbofh (602064) | more than 2 years ago | (#41293831)

How is this different from, for example tech support where you get connected to lots of random people?

You don't have an unintelligible accent when you type?

Re:Not a bot (1)

drcheap (1897540) | more than 2 years ago | (#41296001)

You don't have an unintelligible accent when you type?

Um, hefe-a yuoo nut met zee Svedeesh Cheff? Hees vreettee vurds steell hefe-a qooeete-a un eccent. Bork Bork Bork!

Better than a fake chatbot: (0)

Anonymous Coward | more than 2 years ago | (#41293677)

A fake bot that can do maths [slashdot.org] !

Sounds like a business meeting (1)

Megahard (1053072) | more than 2 years ago | (#41293719)

Probably gets similar results.

Re:Sounds like a business meeting (1)

arth1 (260657) | more than 2 years ago | (#41294723)

Yes, I think so too.
Remember the old Demotivator(TM)? "Because none of us is as stupid as all of us".

I also think the idea that this is cheaper than a concierge service is ridiculous. Instead of having to pay one person to answer your question, you have to pay many.

And even worse, there's no "conversation" as such, because the people answering aren't necessarily the ones who provided the previous output. There's bound to be erroneous answers or WTFs due to a lack of continuity.

Think of the possibilities! (1)

MobileTatsu-NJG (946591) | more than 2 years ago | (#41293793)

Just think: If this technology lives up to its promise, Slashdotters fighting in the great OS Flamewar could create their own automated sentries to fight back with phrases like "reality distortion field' and "rounded corners". Maybe we can finally end all this bloodshed!

Already being done in IRC (0)

bmo (77928) | more than 2 years ago | (#41293879)

Over in IRC, one of the denizens of the channel I frequent has neen running a chatbot that snarfs conversation from all over IRC.

It is like Siri, but drunk and insane, and knows all your secrets.

00:18 ? ascaris
00:18 Ascaris lumbricoides, or "roundworm", infections in humans occur when an ingested infective egg releases a riot on facebook

20:57 bmo, what is the survival length of cats in yogurt
20:57 I don't know, I never tried

--
BMO

Re:Already being done in IRC (1)

someones (2687911) | more than 2 years ago | (#41294925)

I guess they did not want to use a bot fed with IRC chatlogs.

too many penises.

Re:Already being done in IRC (1)

bmo (77928) | more than 2 years ago | (#41295413)

>too many furries.

Fixed.

--
BMO

HIve Mind (1)

NicknamesAreStupid (1040118) | more than 2 years ago | (#41294387)

This may be a good example of a hive mind implementation. The first being the church, and the second was the corporation. The difference is that a machine might be the parser, instead of another person. There was talk of this kind of machine logic being used by companies in the form of schedulers, automated personal assistants, and management analytic engines -- all based on business policies. Once interconnected, they form a collective intelligence that drives the workers, probably like DNA drives a colony of ants. One might think something sinister, but the results are likely to be unpredictable.

Re:HIve Mind (0)

Anonymous Coward | more than 2 years ago | (#41294521)

Woah, dude. That's like totally matrix plus shit right there. We could be inside the matrix, but the machines could be using us to power their brain, so we could be like totally inside the matrix and running it at the same time. Ask not for whom the Neo looks, he looks for thee.
Yeah, I realize this was stupid. Stupid enough to mask my true identity.

couple of cents a question. (0)

Anonymous Coward | more than 2 years ago | (#41294477)

here is my 2 cents.
you are now paying for 7 people to do the job of 1 person, that should know your product or service better that 7 Indian strangers.

In other words your customer is in the hands of strangers paid in pennies, that took you dollars to acquire.

More dumass shit.

It's called a Cyborg. (2)

VortexCortex (1117377) | more than 2 years ago | (#41294727)

That which has both Cybernetic and Organic components... Their "Chatbot" is a Cyborg.

on human value (1)

Hazel Bergeron (2015538) | more than 2 years ago | (#41297551)

while being considerably cheaper than a true concierge service

A "true concierge" is an experienced individual who knows his clients and can perform all manner of complex tasks requiring intelligence.

Neither Siri nor choose-from-random-trollish-human-responses form a concierge service.

It's like pointing out that a vacuum cleaner and a washing machine are cheaper than a butler.

DARPA (0)

Anonymous Coward | more than 2 years ago | (#41299433)

The government is creating logic for chatbots that are run by NGOs to influence the news cycle in places like egypt, syria, russia, to create a buzz that forshortened the news cycle and would influences geopolitical outcomes, Putin and Mubarak would certaily agree.

I have no proof but is no mystery that government capabilities are frequently 20 years ahead of civilian and I participated in a study to this effect (to help tune the math and logic) that anticipated by several days the news cycle. So this is just a logical extrapolation...

Something to consider when conversing on 4Chan, Pastebin, etc.

Diamond Age (1)

bigmo (181402) | more than 2 years ago | (#41302133)

I guess it's old news, but this sounds exactly like what was being described in Neil Stephenson's Diamond Age. Actors there were paid to read/act short pieces of text/commands to reply to a young girl's questions. In the story, the girl was asking a book to explain a concept to her. Not much different from what happens with a chatbot.

I guess this might also relate to the earlier post on online math courses. Presumably grad students could be given micropayments to answer specific questions for an online course that the teacher doesn't have the time/inclination to answer.

My real concern in using this sort of thing for important information/decisions is how the answers get moderated quickly. Wikipedia has a pretty good, though certainly not perfect, way to deal with this, but it's not necessarily fast enough for real time issues.

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?