Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Microsoft Research Uses Kinect To Translate Between Spoken and Sign Languages

samzenpus posted about 10 months ago | from the what'd-you-say dept.

Microsoft 79

An anonymous reader writes in with a neat project Microsoft is working on to translate sign language with a Kinect. "Microsoft Research is now using the Kinect to bridge the gap between folks who don't speak the same language, whether they can hear or not. The Kinect Sign Language Translator is a research prototype that can translate sign language into spoken language and vice versa. The best part? It does it all in real time."

cancel ×

79 comments

Sorry! There are no comments related to the filter you selected.

P2TC signs: "I really like this" (2)

Press2ToContinue (2424598) | about 10 months ago | (#45287207)

Kinect translation with new autocorrect-for-ASL: "I really like your tits"

Thanks Microsoft!

Re:P2TC signs: "I really like this" (-1)

Anonymous Coward | about 10 months ago | (#45287393)

that should be "you're tits"

Re:P2TC signs: "I really like this" (0)

Anonymous Coward | about 10 months ago | (#45287515)

"I really like you are tits"? It doesn't make any sense but somehow makes sense for an auto-correct.

Re:P2TC signs: "I really like this" (0)

Anonymous Coward | about 10 months ago | (#45287755)

That's really funny :)

Guess not too many /.ers know ASL.

Re:P2TC signs: "I really like this" (1)

SleazyRidr (1563649) | about 10 months ago | (#45291229)

As a /.'er unfamiliar with ASL, could you elaborate? I learned a couple of phrases for some volunteer work I did a few years back, but nothing to do with tits. I feel like I missed the most important part of the lesson.

Re:P2TC signs: "I really like this" (1)

fatphil (181876) | about 10 months ago | (#45292907)

Everyone on the internet should know that ASL is short for "age / sex / location". Clearly the sex has already been ascertained, presumably from the nick in use, if complements about tits are already being made.

Re:P2TC signs: "I really like this" (1)

xSander (1227106) | about 10 months ago | (#45289969)

This little joke actually raises an interesting topic, as sign language usually does not follow the same grammar as spoken/written language. Sign languages can differ from country to country or from region to region, similar to dialects. The hands aren't the only things used to convey meanings either: facial expressions or body movements are used too.

An example of sign language using the line above could be to represent the words "breasts", "you" and "like", in that order, with expressing the meaning "very much" by widening the eyes or exaggerating the sign for "like". If a software program can only recognize signs and nothing else, the "really" gets lost in translation. And then you have to put the words in the right order as well.

Re:P2TC signs: "I really like this" (0)

Anonymous Coward | about 10 months ago | (#45290069)

This comment you've made Press2ToContinue is incredibly childish and hurtful. You should apologize. I'm frankly angry that Slashdot would mod up this kind of comment. The people working on this are deaf and for them this kind of real-time translation would be transformative. Not to mention the potential benefit for the rest of us as we try to talk to others around the world who do not speak our native language. Man up and apologize for this hurtful comment.

Re:P2TC signs: "I really like this" (0)

Anonymous Coward | about 10 months ago | (#45290287)

...not sure if serious?

Re:P2TC signs: "I really like this" (0)

Anonymous Coward | about 10 months ago | (#45291595)

The user Press2ToContinue is in the middle of a gender transition, and your comment about "manning up" is quite hurtful to her.

P2TC signs: "I really like this" (1)

Press2ToContinue (2424598) | about 10 months ago | (#45296465)

nods and looks sad, grabs cap brim with left hand and runs right thumb along chin

And vice versa? (1)

narcc (412956) | about 10 months ago | (#45287245)

How do you sign: Dear aunt, let's set so double the killer delete select all?

Re:And vice versa? (0)

Anonymous Coward | about 10 months ago | (#45287499)

That was 7 years ago dude, let it go. How far have _you_ come in the last 7 years?

Does it translate... (0)

Anonymous Coward | about 10 months ago | (#45287263)

... flipping the bird?

It probably wouldn't, because... (1)

Press2ToContinue (2424598) | about 10 months ago | (#45287283)

some signs need no translation.

Re:Does it translate... (0)

RedHackTea (2779623) | about 10 months ago | (#45287711)

Yes, but currently "mooning" (def., pants down and butt cheeks slapping together) is a problem and translates it as "moonfuckingyou!"

Re:Does it translate... (1)

wonkavader (605434) | about 10 months ago | (#45297373)

In ASL the bird is directional. Not sure about CSL. But you don't point it upward in ASL, you point it as the moron you're insulting.

IF ... !! (0)

Anonymous Coward | about 10 months ago | (#45287285)

... I flip the bird does is say FUCK YOU ??

Sign Language Is Obsolete (-1)

TrollstonButterbeans (2914995) | about 10 months ago | (#45287313)

Sign language isn't actually much relevant these days.

Almost no one understands sign language and it is quicker and more convenient for the disabled to send a text message.

Mobile phone technology is to thank for this --- it is making the world far more fairer every day.

Re:Sign Language Is Obsolete (1)

MrEricSir (398214) | about 10 months ago | (#45287359)

I'd love to see a blind person try to use touch screen phone.

Re:Sign Language Is Obsolete (3, Insightful)

hyfe (641811) | about 10 months ago | (#45287463)

I'd love to see a blind person use sign language.

Re:Sign Language Is Obsolete (1)

Anonymous Coward | about 10 months ago | (#45288027)

Have... Have you heard of Helen Keller?

Re:Sign Language Is Obsolete (5, Funny)

Anonymous Coward | about 10 months ago | (#45288783)

No, and neither has she.

Re:Sign Language Is Obsolete (1)

Sarten-X (1102295) | about 10 months ago | (#45288251)

You'd see them hold their hands around the person signing to them, so they can feel the gestures. This is one of several techniques [aadb.org] .

Re:Sign Language Is Obsolete (1)

Fri13 (963421) | about 10 months ago | (#45289327)

They do. My friend did a article about that. She spend one week with a couple who both are deaf and the man had become blind few years before.
They both communicate with sign language but it is done by holding each other hands when doing it. There were three couples registered to be in same situation that they need to communicate with physical sign language.

You can be amazed what people can do who can not hear, talk and don't have anymore capability to see. They live normal live just together.

ps. My friend is deaf as well, can speak pretty well if really a required but she doesn't like to do it because it is really hard to her to speak more than few words.

Re:Sign Language Is Obsolete (0)

Anonymous Coward | about 10 months ago | (#45291673)

Ask and ye shall receive: http://www.youtube.com/watch?v=qoClCdxaUCk

Re:Sign Language Is Obsolete (4, Informative)

petsounds (593538) | about 10 months ago | (#45287497)

They do. They use iPhones (and iPads). iOS has an accessibility feature called VoiceOver which changes the input paradigm from a touch initiating a 'click' action to one where a touch reads out the description of the UI element with text-to-speech. Two taps on the item will send a tap to the UI, and a three-finger swipe will initiate scrolling. So you can basically drag your finger across the screen and find things with your ears instead of your eyes, then navigate and interact as a sighted person would once you get your bearings.

Re: Sign Language Is Obsolete (1)

Anonymous Coward | about 10 months ago | (#45288003)

Can't tell if that's intended to be a troll or not. Since this is the internet I have to assume it is.

As far as trolling goes, being married to a deaf women I'm pretty pissed off at it. It's so rude and dead wrong I'm left incredulous, but that is the point of trolling isn't it.

Re:Sign Language Is Obsolete (2)

Jah-Wren Ryel (80510) | about 10 months ago | (#45288269)

I'd love to see a blind person try to use touch screen phone.

Touch screen phones may well be the best tech to come along for helping blind people ever.

http://bits.blogs.nytimes.com/2013/09/29/disruptions-guided-by-touch-screens-blind-turn-to-smartphones-for-sight [nytimes.com]

Since that might be pay-walled, here's a copy:

Disruptions: Visually Impaired Turn to Smartphones to See Their World
September 29, 2013, 11:00 am

In recent years, many smartphone apps that are aimed at blind people have appeared.

Luis Perez loves taking photographs. He shoots mostly on an iPhone, snapping gorgeous pictures of sunsets, vintage cars, old buildings and cute puppies. But when he arrives at a photo shoot, people are often startled when he pulls out a long white cane.

In addition to being a professional photographer, Mr. Perez is almost blind.

"With the iPhone I am able to use the same technology as everyone else, and having a product that doesn't have a stigma that other technologies do has been really important to me," said Mr. Perez, who is also an advocate for blind people and speaks regularly at conferences about the benefits of technology for people who cannot see. "Now, even if you're blind, you can still take a photo."

Smartphones and tablets, with their flat glass touch screens and nary a texture anywhere, may not seem like the best technological innovation for people who cannot see. But advocates for the blind say the devices could be the biggest assistive aid to come along since Braille was invented in the 1820s.

Counterintuitive? You bet. People with vision problems can use a smartphone's voice commands to read or write. They can determine denominations of money using a camera app, figure out where they are using GPS and compass applications, and, like Mr. Perez, take photos.

Google's latest releases of its Android operating systems have increased its assistive technologies, specifically with updates to TalkBack, a Google-made application that adds spoken, audible and vibration feedback to a smartphone. Windows phones also offer some voice commands, but they are fewer than either Google's or Apple's.

Among Apple's features are ones that help people with vision problems take pictures. In assistive mode, for example, the phone can say how many heads are in a picture and where they are in the frame, so someone who is blind knows if the family photo she is about to take includes everyone.

All this has come as a delightful shock to most people with vision problems.

"We were sort of conditioned to believe that you can't use a touch screen because you can't see it," said Dorrie Rush, the marketing director of accessible technology at Lighthouse International, a nonprofit vision education and rehabilitation center. "The belief was the tools for the visually impaired must have a tactile screen, which, it turns out, is completely untrue."

Ms. Rush, who has a retinal disorder, said that before the smartphone, people who were visually impaired could use a flip-phone to make calls, but they could not read on the tiny two-inch screens. While the first version of the iPhone allowed people who were losing their vision to enlarge text, it wasn't until 2009, when the company introduced accessibility features, that the device became a benefit to blind people.

While some companies might have altruistic goals in building products and services for people who have lost their sight, the number of people who need these products is growing.

About 10 million people in the United States are blind or partly blind, according to statistics from the American Foundation for the Blind. And some estimates predict that over the next 30 years, as the vast baby boomer generation ages, the number of adults with vision impairments could double.

Apple's assistive technologies also include VoiceOver, which the company says is the world's first "gesture-based screen reader" and lets blind people interact with their devices using multitouch gestures on the screen. For example, if you slide a finger around the phone's surface, the iPhone will read aloud the name of each application.

In a reading app, like one for a newspaper, swiping two fingers down the screen will prompt the phone to read the text aloud. Taking two fingers and holding them an inch apart, then turning them in a circle like opening a padlock calls a slew of menus, including ones with the ability to change VoiceOver's rate of speech or language.

The iPhone also supports over 40 different Braille Bluetooth keyboards.

On all the mobile platforms, people with vision loss say, the real magic lies in the hundreds of apps that are designed specifically to help people who are blind.

There are apps that can help people see colors, so pointing their phones at an object will yield a detailed audio description of the color, like "pale yellow green" or "fresh apricot." People who are blind say these apps open up an entirely new way of seeing the world. Light detection apps can emit a sound that intensifies when someone approaches a light source. This can be used to help people find a room's exit, locate a window or turn off a light. There are apps that read aloud e-mails, the weather, stock prices as well as Twitter and Facebook feeds.

In the United States, one of the biggest challenges for blind people is figuring out a bill's denomination. While coins are different sizes, there is no such differentiation between a $1 bill and a $100 bill. In the past, people with impairments had someone who could see help them fold notes differently to know which was which, or they carried an expensive third-party device, but now apps that use the camera can identify the denomination aloud.

"Before a smartphone was accessible we had to carry six different things, and now all of those things are in one of those devices," Ms. Rush said. "A $150 money reader is now a $1.99 app."

She added: "These devices are a game-changer. They have created the era of inclusion."

While some app makers have made great efforts to build products that help people with impairments, other developers overlook the importance of creating assistive components.

Mr. Perez said what he could do now with his smartphone was inconceivable just a few years ago. But even well-known apps like Instagram, which he uses to share some of his photos, do not mark all of their features.

"When some developers design their apps, they don't label all of their buttons and controls, so the screen reader just says, âThis is a button,' but it doesn't say what the button actually does," Mr. Perez said. "That's an area where we need a lot of improvement."

Re:Sign Language Is Obsolete (0)

Anonymous Coward | about 10 months ago | (#45291693)

Ask and ye shall receive: http://www.youtube.com/watch?v=c0nvdiRdehw

Re:Sign Language Is Obsolete (0)

Anonymous Coward | about 10 months ago | (#45292621)

Blind people can (usually) speak and hear just fine so they aren't particularly relevant to a discussion about sign language which is a solution for deaf or mute people.

For blind people we have speech to text and text to speech technology. Not much point involving sign language in there as it'll juts make things more complicated.

Re:Sign Language Is Obsolete (1)

SJHillman (1966756) | about 10 months ago | (#45287367)

Not just text messages, but also online college courses, Facebook messages, etc. Especially for people like me who are deaf enough that suffer a lot - socially, academically and in the workplace, but not so deaf that we qualify for any sort of assistance. It's a shame Obamacare doesn't address hearing aids and glasses, it would have likely gotten a lot more support.

Re:Sign Language Is Obsolete (2, Interesting)

narcc (412956) | about 10 months ago | (#45287557)

It's a shame Obamacare doesn't address hearing aids and glasses, it would have likely gotten a lot more support.

That's funny. Not only is the deaf community a tiny minority (unlikely to have any impact at all), the issue of hearing aids is enough to divide them! If they found out you could get a cochlear implant with insurance purchased through an exchange, you'd see little other than opposition. (Yeah, they're that crazy.)

Not that they're likely to be aware of the issue, as illiteracy is so prevalent. Still, if they found out, they'd oppose it.

Re:Sign Language Is Obsolete (2, Insightful)

Anonymous Coward | about 10 months ago | (#45287597)

Yeah, trippy that the Deaf want to protect being Deaf so much that they don't want other deaf people to hear!

Re:Sign Language Is Obsolete (1)

SJHillman (1966756) | about 10 months ago | (#45289609)

That's generally a vocal minority that's mostly perpetuated by the older population that has always been deaf. Deaf people who aren't exposed to the older deaf generation are much more open minded. Unfortunately, some colleges aimed at the deaf (I'm looking at you, RIT NTID) continue to perpetuate this - as a result, many students there come to equate deafness with being an asshole. And I don't blame them.

Re:Sign Language Is Obsolete (0)

Anonymous Coward | about 10 months ago | (#45289395)

Because that's not so much about being "deaf" per se, but really about ASL culture....Like many linguistic minorities, they often perceive themselves as under attack.

Re:Sign Language Is Obsolete (1)

SJHillman (1966756) | about 10 months ago | (#45289595)

People who are functionally deaf are a minority (less than a million in the US), and hearing aids will likely never help them. People who are partially deaf and would benefit from hearing aids are thought to number at least 10 million in the US, and that may be a low estimate. To put it in comparison, there are only 7 states with more than 10 million residents (and three of those aren't too much higher).

There's also a lot of people with partial hearing loss who do not identify with any "deaf community". I recognize that it exists, but equating hearing loss with being part of the deaf community is just plain wrong.

Re:Sign Language Is Obsolete (3, Insightful)

mythosaz (572040) | about 10 months ago | (#45287375)

You clearly haven't seen two or more deaf people in public.

Plenty of people sign. Heck, plenty of hearing people sign.

Conversations (deaf-deaf, hearing-hearing, hearing-deaf) are all much more natural face-to-face where you can interrupt, show expressions, assign nuanced gestures or tones to words -- all things that are not well suited for text.

Re:Sign Language Is Obsolete (2)

TheloniousToady (3343045) | about 10 months ago | (#45287639)

Spot on. Also, sign language is quite important culturally as the native language of deaf people. All cultures view their native language as essential to their cultural identity. The deaf are no different (or so I hear.)

Re:Sign Language Is Obsolete (1)

narcc (412956) | about 10 months ago | (#45287821)

Except sign is a con lang -- strike that -- a bunch of different con langs incompatible with one another.

Re:Sign Language Is Obsolete (0)

Anonymous Coward | about 10 months ago | (#45288451)

No, it isn't. Some (all?) modern sign languages may have begun as attempts as constructed languages for the deaf but they quickly creolize just like any other natural language. In fact there's evidence that sign languages are older than spoken languages. For one, young children can acquire sign language younger than they can acquire spoken languages partially because the hand movements are a lot easier for young children than learning to use their vocal cords.

Re:Sign Language Is Obsolete (0)

Anonymous Coward | about 10 months ago | (#45290021)

(or so I hear.)

Now that was fucking funny.

Re:Sign Language Is Obsolete (1)

kermidge (2221646) | about 9 months ago | (#45301841)

Sign comes in handy in very noisy environments, or conversely where silence is golden. Then there's job-specific signing, as for crane operators and their spotters.

Re:Sign Language Is Obsolete (4, Informative)

CRCulver (715279) | about 10 months ago | (#45287565)

Sign language isn't actually much relevant these days. Almost no one understands sign language and it is quicker and more convenient for the disabled to send a text message.

This is one of the stupidest things I've read all day. You think that pupils at a school for the deaf are sending each other text messages as they stand next to each other? Signing is still very popular among the deaf, and is even a part of the distinct culture that has arisen in deaf communities.

Re:Sign Language Is Obsolete (1)

narcc (412956) | about 10 months ago | (#45287825)

Deaf culture is destructive.

Re:Sign Language Is Obsolete (1)

Anonymous Coward | about 10 months ago | (#45288753)

Well... sending each other texts while standing next to each other works for non-deaf teens....

Re:Sign Language Is Obsolete (0)

FuzzNugget (2840687) | about 10 months ago | (#45288801)

This is one of the stupidest things I've read all day.

Awfully generous of you to use "day."

Re:Sign Language Is Obsolete (2, Informative)

Anonymous Coward | about 10 months ago | (#45287575)

Your comment shows the sign of ignorance. Plenty of deaf AND hearing people sign anywhere. It's easily 10 times faster than texting. Even faster than speech itself.

Re:Sign Language Is Obsolete (1)

Cro Magnon (467622) | about 10 months ago | (#45290515)

Yeah, I've found that presenting the proper (or improper) finger while driving is much faster than yelling "F- off" out the window.

Re:Sign Language Is Obsolete (1)

Krishnoid (984597) | about 10 months ago | (#45287669)

Possibly obsolete, but it's spawned at least one mini-celebrity [buzzfeed.com] .

Re:Sign Language Is Obsolete (3, Informative)

puto (533470) | about 10 months ago | (#45287875)

I have two deaf cousins who would beg to differ, and all of us in the family who learned to sign can communicate much quicker by signing than grabbing a smart phone. And my hands do not need electricity. Fucking douchebag.

Re:Sign Language Is Obsolete (2)

pne (93383) | about 10 months ago | (#45288777)

it is quicker and more convenient for the disabled to send a text message.

I would have thought it would be more convenient for someone to speak/be spoken to in their native language - sign - rather than send/receive a text in a foreign language - English or whatever.

ASL? Siglish? (1)

mythosaz (572040) | about 10 months ago | (#45287317)

There's a lot of contextual clues necessary to understand sign language. Most conversations would seem "faux caveman" like to the outsider - a lot of Noun Verb Noun going on...

I'm going to have to watch the video from another machine, but I'm more interested in the bumper at the bottom that has realtime English/Chinese translation in your own voice...

Re:ASL? Siglish? (1)

Blakey Rat (99501) | about 10 months ago | (#45292231)

They (I assume the same group) demoed the real-time English/Chinese translation in your own voice last year. It's really impressive, and the results were surprisingly good.

I do wonder how it deals with phonemes that are present in one language but not in another, maybe there's a "training process" you have to do initially to make sure it has enough recorded samples to get full-coverage of the target language.

Hardly revolutionary (3, Interesting)

afidel (530433) | about 10 months ago | (#45287327)

We were doing real-time ASL translation to text using the webcam on the Indy2 workstation back in 1997, success rate was about 85% and most of the misses were from hidden object problems which the Kinect does nothing to help with.

Re:Hardly revolutionary (0)

Anonymous Coward | about 10 months ago | (#45288143)

That sounds like extremely interesting work, especially given that it was done in 1997. If I understand correctly it's still difficult to get computers with webcams to track individual fingers accurately in 2013. Do you have a link where we can see more information about your project?

Re:Hardly revolutionary (2)

Miamicanes (730264) | about 10 months ago | (#45289285)

Believe it or not, the video compression codec we now know as h.264 had its origins about 20 years ago as a compression scheme to allow the transmission of video good enough to allow sign language over a dialup modem. Its evolution into h.264 (along with some very powerful ASICs that jumpstarted h.264 to allow realtime hardware compression and decompression whose development costs were subsidized by the federal government) perfectly demonstrates an assertion I've long made -- the best way for deaf people to make their lives easier is to find creative ways to commercialize the technology they depend upon and make it useful to NON-deaf people as well, so it goes from being an expensive accommodation to a normal & common business practice. When closed captions first came out, the only shows that had them were shows that few deaf people really wanted to watch. Ten years later, once the decoding hardware was built into new TVs by default and gyms & sports bars realized they were useful for people to watch over the background noise, captions were almost universal... even for things like live newscasts.

Coming soon: a method for you to drive to the drive-through window of a restaurant, launch your 'restaurant-order' app, take a pic of the 2-d barcode (or maybe swipe the phone over a rfid tag) identifying the restaurant, then place your order without having to talk to the person inside over an unintelligible, noisy audio link. Deaf people are an awesome test market for things involving high-end mobile phones, because even elderly deaf people are overwhelmingly likely to have a fairly recent iPhone or best of breed Android phone. There are also quite a few deaf software developers... partly, because it's a career where being deaf is almost inconsequential as long as you can arrange your life to work with others who are also deaf or know sign language.

Re:Hardly revolutionary (1)

jeffb (2.718) (1189693) | about 10 months ago | (#45289585)

Hmm. I know Morton Nadler at Virgina Tech was working on a system to image-process, recognize, compress and transmit sign language (I think it may have been fingerspelling) back in the early- to mid-1980s. I think he was using racks of custom hardware; this may have pre-dated even FPGAs. It was expensive, but everybody knew that hardware would keep getting cheaper...

Re:Hardly revolutionary (1)

CowboyBob500 (580695) | about 10 months ago | (#45289043)

And for my masters thesis in 2000 I worked on a thing called TESSA [uea.ac.uk] . Specifically my part was on the compression algorithm that reduced the data points for the sign language half from 15 dimensions (1 dimension for each joint in the fingers and thumb) down to 3 dimensions as computing power was too limited back then to deal with any more in real time. In the end I used a combination of hidden Markov models and active shape models to do this. Was really interesting stuff.

Re:Hardly revolutionary (1)

wonkavader (605434) | about 10 months ago | (#45297385)

And these people are NOT doing real time translation. They're having people sign each sign stilted isolation.

As

Though

They

Talked

Like

This

Edutainment (1)

FalleStar (847778) | about 10 months ago | (#45287365)

As far as ways to communicate online goes, I'm not sure how useful of a tool this would be. I can definitely see how this could easily become the best way to learn sign language though if paired with Rosetta Stone-like tutoring software. My wife has been planning to learn sign language soon, I'm sure she'd love to have something like this as a learning aide.

Re:Edutainment (1)

TheSeatOfMyPants (2645007) | about 10 months ago | (#45288401)

It would probably be invaluable for video-chatting/conferences (or even IRL discussions) where there's a mix of Deaf people &hearing ones that don't know the specific sign language they use, as it'd mean the two groups could interact directly/independently without requiring a translator. That'd be great for private sensitive conversations between two individuals, relatives meeting for the first time (as it'd mean they could communicate directly), and might have a good impact on Deaf people's employability.

There's an interesting problem with this approach (1)

Edis Krad (1003934) | about 10 months ago | (#45287387)

I tried to tackle the same thing a couple of months back using OpenCV and a smartphone. Before starting I consulted with people that knew sign language and the problem is not so much recognizing hand gestures, but facial expressions. They say that most of the conversation happens with a a given look, a frown, or the movement of the lips.

Needless to say, I though it was too hard to solve the problem on a smartphone so I postponed the project. I don't think Kinect can do a much better job at picking up the small details of facial expressions, but let's wait an see.

Re:There's an interesting problem with this approa (0)

Anonymous Coward | about 10 months ago | (#45287423)

If the kinect is tracking your arms/legs/fingers/head etc. I'm fairly certain it can be programmed to start using facial expressions to add the missing conversations.

Re:There's an interesting problem with this approa (1)

wonkavader (605434) | about 10 months ago | (#45297389)

UNlikely.

Much subtler changes for facial work.

Re:There's an interesting problem with this approa (1)

mythosaz (572040) | about 10 months ago | (#45287483)

Yup. Many signed phrases are just noun, verb, point, noun .

Re:There's an interesting problem with this approa (0)

Anonymous Coward | about 10 months ago | (#45288839)

Not sure how ASL compares to BSL (I'm a BSL n00b), but facial expressions are one thing that has bothered me about sign-to-text/speech systems. Without facial expressions: I can't express the magnitude of certain things; I can't ask questions; I can't answer some questions; I can't distinguish between things like uncle, aunt, nephew, niece or battery; yadda yadda. Non-manual features are really important grammatically and expressively.

There are also signs that don't really have an equivalent in spoken/written language unless you start incorporating diagrams into the translation.

Don't get me wrong, I think work like this is a great idea but the projects I've seen so far either ignore the problem or grossly simplify it - losing the power and expressiveness of sign language.

Awesome (0)

Anonymous Coward | about 10 months ago | (#45287421)

Goodbye Apple and Google. This technology is set to put you OUT. OF. BUSINESS.

slow (0)

Anonymous Coward | about 10 months ago | (#45287445)

Looks like about the quality of verbel speech recognition...

I have never seen anyone sign that clearly or slowly except to very new beginners.

Re:slow (1)

RightSaidFred99 (874576) | about 10 months ago | (#45288655)

They didn't immediately make it able to understand full-speed sign language including that from a man wearing black gloves in a dark room?!!!eleven!

Curse you, Microsoft Research, curse you to the depths of fucking hell!!!!

Very creative (1)

uncomformistsheep (2950041) | about 10 months ago | (#45288471)

There are lots of research being done with kinect, by BS and masters students, mostly around physiotherapy. This is one of those creative applications that everybody says after hearing about it .. "damn, why didn't I think of that". Very creative use of the kinect.

Real Time? (1)

pne (93383) | about 10 months ago | (#45288793)

Looking at the video in the article, it seems that "in real time" means "at about 1/4 of the speed of regular signing".

Imagine. Having. To. Speak. Like. Kirk.

Re:Real Time? (0)

Anonymous Coward | about 10 months ago | (#45288929)

Imagine having to speak like this: - - - -

Re:Real Time? (1)

pne (93383) | about 10 months ago | (#45288967)

OK, you have a point.

Being understood definitely beats not being, even if you have to slow down to achieve that. And the translation was happening reasonably quickly if you were slow enough.

I was just a bit disappointed since from the "real time" I was expecting the translation to keep up with someone speaking at normal conversational speed. (Like the asker at the information kiosk, for example.)

Most impressive use of technology I've ever seen (4, Interesting)

hakioawa (127597) | about 10 months ago | (#45290275)

Disclaimer: yes I work for Microsoft. No not on these projects.

This was demo'd live in front of 30K MSFT employees at our annual company meeting. It nearly brought me to tears. Yes, I can see through demoware and and yes it's highly imperfect, but honestly it was the single most impressive use of technology I've ever seen. It was both novel and simple. It combined hardware, algorithms, user experience, and cloud scale. I don't know if it will ever go anywhere though I expect that it will. The key point here is that these are off the shelf components. Kinect and gesture APIs combined with machine translation and text to speech. It's important that these are, all or nearly all public production APIs. Such a system 10 years ago even if possible, would never make it to market because of the tiny user base. Today we can build such apps for the 0.01% of the population that need Mandarin Sign Language translated to English. And it can be cost effectively. That is the point. Technology being used to address real problems for under served communities. So yes, maybe people researched automated sign language recognition years ago, but bringing it to market and enabling a scenario for real people is a wholly different beast

Universal Translator (1)

Hamsterdan (815291) | about 10 months ago | (#45290811)

I knew it! the Federation runs on Microsoft technology. Guess that explains all the exploding consoles and Transporter accidents.

Way too many Dialects (1)

Kagato (116051) | about 10 months ago | (#45291197)

The ASL interpreters I know do a lot of on-call work for medical, mental health and educational purposes. One thing they mentioned is ASL has regional dialects.

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>