Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Designing a Practical UI For a Gesture-Based Interface

timothy posted about a year and a half ago | from the ok-but-I-definitely-am-sticking-with-steering-wheels dept.

Software 44

An anonymous reader writes with a link to an intriguing account of the challenge of designing a close-range, hand and finger-based gesture recognition interface using 3D cameras. Things like this look good in science-fiction, but it's hard to create a gesture-based system that makes sense to the user and rejects gestures not meant for the computer.

cancel ×

44 comments

Sorry! There are no comments related to the filter you selected.

Test it on Italians (5, Funny)

Overzeetop (214511) | about a year and a half ago | (#42785387)

You're going to have to test it on Italians for gesture rejection. I can't imagine what kind of havoc could be created if the interface over-saw even the most mundane of conversations.

Re:Test it on Italians (0)

Anonymous Coward | about a year and a half ago | (#42785687)

Italians are amazing people; they can change gears, smoke a cigarette, and flip you off all at the same time.

Re:Test it on Italians (1)

K. S. Kyosuke (729550) | about a year and a half ago | (#42788603)

You're going to have to test it on Italians for gesture rejection. I can't imagine what kind of havoc could be created if the interface over-saw even the most mundane of conversations.

We already know where this is going to lead:

For years radios had been operated by means of pressing buttons and turning dials; then as the technology became more sophisticated the controls were made touch-sensitive - you merely had to brush the panels with your fingers; now all you had to do was wave your hand in the general direction of the components and hope. It saved a lot of muscular expenditure of course, but meant that you had to sit infuriatingly still if you wanted to keep listening to the same programme.

We need to discuss the Superbowl here (-1)

Anonymous Coward | about a year and a half ago | (#42785449)

And why the power went out? Have they figured that out yet?

I hate gestures. (0)

Anonymous Coward | about a year and a half ago | (#42785481)

The beauty and the promise of devices like Kinect, and the power of modern computers, is the potential for an interface with almost no gestures. I know skeuomorphism has taken a lot of hits here lately, but this is the worst yet. They've made a model of a bookshelf in beautiful 3D, and the first thing you'll want to do is pluck a book from the shelf and peel the covers apart. Instead, you'll have to poke, and jab, and pinch, and drag. If you want a 3D gesture-based interface, bang it out Kinect style, or iOS style, with beautiful-looking icons to double-tap, or double-punch, or whatever. Make it look like a traditional interface, because doing the opposite will only make me sad when I discover that it is still just a traditional interface.

need virtual keyboards (0)

Anonymous Coward | about a year and a half ago | (#42793255)

The gesture stuff is nice, but we still need to type words to communicate and I'd like to see some kind of new creative keyboard interface that will make it more portable. Why not use this 3D gesture recognition to recognize how we move our fingers, to allow us to type virtually without a physical keyboard?

If Microsoft is involved... (2)

TWX (665546) | about a year and a half ago | (#42785485)

...I can think of a particular gesture that will be very common, but most likely won't be recognized as it'll be provided after the system isn't working...

Re:If Microsoft is involved... (2)

vlm (69642) | about a year and a half ago | (#42785509)

American Sign Language? Oh wait you mean something else.

Seriously whats wrong with ASL? Large installed base, lots of training, could be pretty darn useful as a translator when someone without ASL "talks" to someone with ASL.

My guess is MS would try to embrace extend and extinguish ASL... make it just like ASL but upside down and backwards, just to make it hard for everyone.

Re:If Microsoft is involved... (0)

Anonymous Coward | about a year and a half ago | (#42785693)

Patent each and every gesture!

Re:If Microsoft is involved... (0)

Anonymous Coward | about a year and a half ago | (#42786019)

My guess is MS would try to embrace extend and extinguish ASL... make it just like ASL but upside down and backwards, just to make it hard for everyone.

Australian Sign Langauge?

Re:If Microsoft is involved... (1)

JasterBobaMereel (1102861) | about a year and a half ago | (#42786669)

If you mean the finger then it is not as widely used or recognised outside the USA ... ...many gestures are very localised and do not travel very well to new cultures

Re:If Microsoft is involved... (1)

SuricouRaven (1897204) | about a year and a half ago | (#42789583)

It wasn't. Due to the international success of US-made entertainment media, their expressions and gestures have been spreading for some time. The finger is certainly recognized here in the UK.

Take a look at the Leap (2)

gentryx (759438) | about a year and a half ago | (#42785491)

I just got my development kit from Leapmotion [leapmotion.com] . Take a look at their 1m video [youtube.com] . Pretty impressive, I would say. Also, the resolution of their sensor (1mm) seems to be much higher than that of what cameras can do today.

Looked good? (0)

Anonymous Coward | about a year and a half ago | (#42785505)

The exaggerated gestures looked completely silly in Minority Report. Who in his right mind would want a computer like that?

Re:Looked good? (0)

Anonymous Coward | about a year and a half ago | (#42788613)

Agreed. Only Tom Cruise can make those "gestures" look good.

That being said, your typical alpha-male MBA or ad agency guy would look (think he does) like a cool guy using it. Or Chuck Norris.

what finger am i holding up (0)

Anonymous Coward | about a year and a half ago | (#42785545)

take a big guess

more challenging...practicality. (2)

argStyopa (232550) | about a year and a half ago | (#42785817)

The biggest challenge is explaining who/how/why anyone would NEED a gesture-based interface.

Sure, we all thought Minority Report looked cool with the flying window-thingy, but honestly, Mr Cruise could have 'flipped through' his data far, far more easily sitting at a desk with trivial motions of a mouse instead of giant arm-wavings and calisthenics.

Plus, then he'd get to SIT, which is a little more conducive to surviving the 10+ hour days most of us spend staring at a little computer screen.

Re:more challenging...practicality. (1)

Jmc23 (2353706) | about a year and a half ago | (#42786397)

Perhaps these UI's aren't meant for low-level troglodytes like yourself? Perhaps they're made for people who can move their bodies and spend most of their time not being worker slaves?

Re:more challenging...practicality. (0)

Anonymous Coward | about a year and a half ago | (#42787051)

I'm trying to imagine the support calls.

"All right. Now put up two fingers and wave them at the screen to move your file to the other monitor."
"Oh, it rebooted? No, you were supposed to point your hand away from the screen, not at it."

Gestures are a retarded interface for anything that can easily be fitted with a keyboard and mouse.

Re:more challenging...practicality. (2)

hey (83763) | about a year and a half ago | (#42786663)

I dislike gesture UIs. Not just because they are new. The main reason is that if you are a newbie there are no hints about what to do. With shortcut keys you could always use the menus instead if you don't know the keystrokes. But Windows, Blackberry, Android, iPhone, all have different gestures its easy to get lost if you use more than one platform.

Re:more challenging...practicality. (1)

citizenr (871508) | about a year and a half ago | (#42787035)

Plus, then he'd get to SIT, which is a little more conducive to surviving the 10+ hour days most of us spend staring at a little computer screen.

This is actually a minus, and a big one at that. Standing up desk is a LOT healthier.

Re:more challenging...practicality. (1)

Richy_T (111409) | about a year and a half ago | (#42788091)

Agreed. The mousewheel has become ubiquitous because it out-competed the effort required to twitch the mouse to move the pointer over to the scroll bar.

Aside: Have any comedies spoofed gesture interfaces? It seems like it would be a good target for a physical comedian (Rowan Atkinson, Lee Evans or Ryan Stiles for example).

Re:more challenging...practicality. (0)

Anonymous Coward | about a year and a half ago | (#42789909)

I would learn to use it since my right hand is experiencing significant pain from mousing and typing all day.

Standing up to use gestures would help to relieve that pain and I could get more exercise, if not cardio. Just getting to move around more would be a relief.

Re:more challenging...practicality. (0)

Anonymous Coward | about a year and a half ago | (#42795129)

Yo, you must have never played with a Kinect/Xtion, Bee...
Try processing [processing.org] with SimpleOpenNI lib for a quick entry, or
see here for some examples: http://www.openprocessing.org/search/?q=kinect (or look online for tuts).
That's nice for prototyping as well, btw.

C U in minority report 2.0

As an ASL Interpreter... (4, Informative)

bigattichouse (527527) | about a year and a half ago | (#42785911)

I interpreted ASL in educational settings (High School, Freelance, University, Public, and even elementary.) for something like 6 or 7 years.

My arms were ripped, and you could expect to burn several hundred calories (EASY) during a day of doing that. Also, I had learned the stretch properly thanks to some Aikido training.. and I still had some bad habits that caused me repetitive stress problems.

Gestures are a novelty, and a lot of work for the user... I think there will be many blind alleys before they become natural.

Some problems/ideas I see:
1. Exhaustion - you waste a lot of energy
2. "Namespaces" - you can make two gestures at once - geez... so you have a left hand gesture that tells the computer to listen (the ASL "Attention" one handed would work) + a command - maybe even "against" that hand. Its like a salute with your left hand vertical moving away from your face.
3. Facial expressions are a HUGE part of ASL, probably not even considered. "WH" questions get eyebrows scrunched, other queries eyebrows up, puffed cheeks and all kinds of things...
4. Security - I defy you to sign EXACTLY like someone else... It's possible, and easy in a mocking sense (High schoolers) - but I imagine a door that could see you carrying groceries and unlock combined with voice recog., or other simple things would be useful.

Re:As an ASL Interpreter... (1)

bigattichouse (527527) | about a year and a half ago | (#42785927)

Well.. ok.. attention would be both hands... but I meant it in context of using it to communicate with a computer.

Re:As an ASL Interpreter... (0)

Anonymous Coward | about a year and a half ago | (#42786323)

I interpreted ASL in educational settings (High School, Freelance, University, Public, and even elementary.) for something like 6 or 7 years.

My arms were ripped, and you could expect to burn several hundred calories (EASY) during a day of doing that. Also, I had learned the stretch properly thanks to some Aikido training.. and I still had some bad habits that caused me repetitive stress problems.

Gestures are a novelty, and a lot of work for the user... I think there will be many blind alleys before they become natural.

Some problems/ideas I see:
1. Exhaustion - you waste a lot of energy
2. "Namespaces" - you can make two gestures at once - geez... so you have a left hand gesture that tells the computer to listen (the ASL "Attention" one handed would work) + a command - maybe even "against" that hand. Its like a salute with your left hand vertical moving away from your face.
3. Facial expressions are a HUGE part of ASL, probably not even considered. "WH" questions get eyebrows scrunched, other queries eyebrows up, puffed cheeks and all kinds of things...
4. Security - I defy you to sign EXACTLY like someone else... It's possible, and easy in a mocking sense (High schoolers) - but I imagine a door that could see you carrying groceries and unlock combined with voice recog., or other simple things would be useful.

5. When in doubt, applying nikkyo is sometimes impractical.

Re:As an ASL Interpreter... (1)

Jmc23 (2353706) | about a year and a half ago | (#42786471)

It's a computer. Your interaction with it is limited. You do not tell the cursor to move by giving it directions in english to move so and so pixels in the x direction and in the y direction. So why the hell would you communicate in ASL with it???

All of your problems exist only because you are trying to use the wrong tool for the job.

The HUGE advantage this would have is that it could be an avenue for teaching people how to move their bodies properly. RSI happens when you don't know how to move properly, that's why the damage occurs.

Anyways, they still are coming about it the wrong way. I won't say why, I still have some hope that my system will be finished before someone comes up with the same idea. It'd be nice to be known for something and I'm extremely suprised nobody has implemented it before.

Re:As an ASL Interpreter... (1)

JasterBobaMereel (1102861) | about a year and a half ago | (#42786819)

Don't use ASL it is unlike English specific to USA/Canada and West Africa is not mutually intelligible with most other sign languages and is largely unknown outside the Deaf community

There is no deaf sign language that is reasonably universal like English is in spoken languages so using ASL as a base will alienate many of your potential users ...

The biggest issue is "Gorilla Arm" if you are making a lot of small precise gestures with arms out you get tied extremely quickly, which is why most sign languages avoid this, but most mockups of gesture systems show exactly this type of gesture

 

Too bad, so sad. (1)

AndyKron (937105) | about a year and a half ago | (#42785913)

Sorry, but whatever idea you have was patented already.

How do you do it when dreaming? (1)

TheRealMindChild (743925) | about a year and a half ago | (#42785975)

I would imagine the most intuitive interface is what you do to move objects when dreaming. For instance when I want to pull an object toward me, I focus on it, then tug an invisible rope quickly as to whip it back. I also push my hand outward holding my palm out as to say "stop" to push it away. Somewhere in our heads we already know what will work

Re:How do you do it when dreaming? (1)

Jmc23 (2353706) | about a year and a half ago | (#42786511)

Ah young padawan, do you not realize that the energy and the ritualistic movement that can generate it are two seperate things? There is no spoon.

Re:How do you do it when dreaming? (1)

MickLinux (579158) | about a year and a half ago | (#42788423)

Aah, young Jmc, do you not recognize that Matrix and Star Wars are two different things?

There is no force. No matrix either.

Re:How do you do it when dreaming? (1)

Jmc23 (2353706) | about a year and a half ago | (#42793417)

Force is the nerve impulse. Matrix is the mind construct.

Never heard of metaphors?

Any one who has enough control in their dreams to do what the parent can would understand. Perhaps you need to educate yourself in lucid dreaming?

Re:How do you do it when dreaming? (1)

MickLinux (579158) | about a year and a half ago | (#42841119)

I've done t before, but it's not as interesting as uncontrolled dreaming. I've also done multi-path dreaming: multiple dreams at the same time; but only once. I've also dreamed specifically in color once or twice.

Re:How do you do it when dreaming? (1)

Jmc23 (2353706) | about a year and a half ago | (#42841605)

Not as interesting? Lucidity is separate from control. Control in the hypnogogic state can be as boring as your imagination. Lucidity in deep rem is anything but un-interesting though requires extreme emotional control to maintain

Lucidity at night is directly correlated to awareness during the day.

Re:How do you do it when dreaming? (0)

Anonymous Coward | about a year and a half ago | (#42842473)

Oh, in that case the multipath dreaming was also lucid dreaming, but it was tied to me being extremely exhausted, iirc.

Re:How do you do it when dreaming? (0)

Anonymous Coward | about a year and a half ago | (#42791207)

I have never, ever done this in a dream. On the other hand, an interface based around stomping on the brake pedal and having it not work would be pretty ideal for me. Or punching really slowly.

What about intent recognition (1)

DrXym (126579) | about a year and a half ago | (#42786247)

I'm wondering when the first device will appear which shows GUI elements even before the user has touched the screen. i.e. it senses the user moving their hand close to some portion of the screen and a hidden GUI associated with that area pops up in time to be there when they touch it. It'd have to be done carefully to avoid frustration but I think it'll come in time.

Re:What about intent recognition (0)

Anonymous Coward | about a year and a half ago | (#42793121)

hand jerking up and down == next picture ?

Optional (1)

gmuslera (3436) | about a year and a half ago | (#42786429)

As extra, optional, and not intrusive input device could be good, games are a good example, information consuming activities could be other. But for most content creation (music, and paint excluded) probably would be bad.

I think there is a good solution to this problem (1)

Omnifarious (11933) | about a year and a half ago | (#42787805)

It's the solution humans use in gesture-based communication. Eye contact. The computer needs an obvious eye, and the software needs to be able to tell when the user is looking at it. Bonus points if the eye can blink or move or otherwise express the computer's understanding of the gesture.

Re:I think there is a good solution to this proble (0)

Anonymous Coward | about a year and a half ago | (#42790261)

That would probably work. Face tracking is doable now and should be a subset of the capabilities of the hardware needed for a gesture interface in general.

Hitchhiker's Guide to the Galaxy. (1)

tragedy (27079) | about a year and a half ago | (#42792543)

Have to quote this from _The Hitchhiker's Guide to the Galaxy_ :

A loud clatter of gunk music flooded through the Heart of Gold cabin as Zaphod searched the sub-etha radio wave bands for news of himself. The machine was rather difficult to operate. For years radios had been operated by means of pressing buttons and turning dials; then as the technology became more sophisticated the controls were made touch-sensitive--you merely had to brush the panels with your fingers; now all you had to do was wave your hand in the general direction of the components and hope. It saved a lot of muscular expenditure, of course, but meant that you had to sit infuriatingly still if you wanted to keep listening to the same program.

The fundamental problem of gesture recognition has been recognized for quite some time now.

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?