Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Transforming Any Flat Surface Into a Control Panel With Sound

timothy posted more than 2 years ago | from the scritcha-scritcha dept.

Input Devices 53

New submitter brunozamborlin writes "I just published a short video that shows how a very cheap contact microphone can be used to recognize different types of fingers touch and transform any surface into an interactive board. In the video we put the microphone over different surfaces such as kitchen tables and balloons and through realtime gesture recognition we show how we can play different virtual music instruments using a technique called physical modeling . A mobile version would be definitely possible." The project's Web page shows several more examples. Update: 12/31 15:17 GMT by T : Bruno Zamborlin points out that the surfaces don't need to be flat; instead, they simply need to be rigid.

Sorry! There are no comments related to the filter you selected.

Huh? (1)

frnic (98517) | more than 2 years ago | (#38547742)

I don't see that "recognition" is achieved beyond being able to tell that touches were made and were "different" from each other - what those differences were was not demonstrated.

Re:Huh? (0)

flyneye (84093) | more than 2 years ago | (#38547814)

Following the links leads me to a place that says it runs on Mac. (Insert dissapointed/disapproval emoticon of your choice in this space)

Re:Huh? (0)

brunozamborlin (2542970) | more than 2 years ago | (#38548034)

The code is actually standard c++, hope to be able to share it soooon....

Re:Huh? (1)

flyneye (84093) | more than 2 years ago | (#38548116)


Re:Huh? (1)

ZeroExistenZ (721849) | more than 2 years ago | (#38548140)

How do you recognize the sound ?

Pitch? volume? Or really "patterns" (sequence of sound signatures) ?

Fe. can you measure the distances from the microphone of multiple sound signals ?

this is as bad as when they said a new star wars (1)

decora (1710862) | more than 2 years ago | (#38547896)

was coming out... then we go to the theatre and its muppets in space crossed with some kind of saturday morning cartoon reject.

Re:Huh? (1)

datavirtue (1104259) | more than 2 years ago | (#38547980)

This is just a novel button.One that requires a lot more processing power than even a touch screen that are available cheaply. In addition it is not as flexible as a touch screen in that with a touchscreen you can change the interface and location of the "buttons" at will.

Re:Huh? (2)

houghi (78078) | more than 2 years ago | (#38548366)

Depending on how precise it can measure where your 'mouse' is I could see the following happening:
1) You build this into a smart phone or tablet PC
2) Calibrate by selecting the 4 corners of your surface.
3) use the whole surface you selected as an extended touch screen.
This could be a bar top, a table at your local fast food, a whiteboard, your desk, tray in a plane, ....

It would allow the screen to be smaller yet the input to be bigger without the need to drag the keyboard with you. You could use a paper with a pre-printed design on it, like a keyboard or a map or whatever. Make it in silk and it is indestructable (well almost) and yet very small. []

Re:Huh? (1)

Amyntas (1774358) | more than 2 years ago | (#38549724)

Look for Acoustic Pulse Recognition. It's pretty well exactly what you just said.
It uses a series of microphones, usually four of them on a glass pane, to locate contact.

I would personally love such a setup for a glass desk top or keyboard tray, with the letters and buttons lightly etched into the glass. I'm sure creating imperfections in the glass would require a more rigorous calibration, but I believe it would be well worth the trouble.

I can also imagine using more than four microphones would allow for vastly more detailed input, but I really have no clue.

Re:Huh? (4, Interesting)

brunozamborlin (2542970) | more than 2 years ago | (#38548020)

In the video we see that different gestures are mapped to different sounds. For example the coin enables a certain abstract sound whist the bare fingers enables the bass sound and the nails enables the pad... this is done thanks to gesture recognition techniques (a modified version of Hidden Markov Models to work in realtime). Apologies whether it is not clear enough in the video though...

Training? (5, Interesting)

Mabbo (1337229) | more than 2 years ago | (#38547788)

Do you have to give the system training for the specific purpose and gesture? And could the microphone be on the opposite side of the surface? If so, I think I've got an awesome new way to unlock my door.

Re:Training? (0)

Anonymous Coward | more than 2 years ago | (#38548004)

That could be pretty epic.
Add an elfish desing and a coupe of leds to freak out everyone

Re:Training? (3, Informative)

brunozamborlin (2542970) | more than 2 years ago | (#38548050)

Yes you can train the system with your own gestures :-)

Re:Training? (1)

OzPeter (195038) | more than 2 years ago | (#38548182)

If so, I think I've got an awesome new way to unlock my door.

*knock* *knock* Mabbo?
*knock* *knock* Mabbo?
*knock* *knock* Mabbo?

Re:Training? (1)

newcastlejon (1483695) | more than 2 years ago | (#38548528)

Please, tell me you're not thinking about the barber's knock.

would it work without earbuds (0)

Anonymous Coward | more than 2 years ago | (#38547876)

how would you prevent sound feedback loops

Re:would it work without earbuds (1)

flyneye (84093) | more than 2 years ago | (#38548026)

Looks like it just triggers some keyboard/synth program. Presumably you could get some annoying feedback this way by using a contact speaker on the sensing surface. speaker-----O o____contact mic arranged thusly. Perhaps moving the speaker over the area with a bit of pressure could produce cataclysmic glitching as well. How damn cool is that? Maybe it could be ported to *nix so all the modules of the program could be maybe linked up in Jackd. This is O0ober cool and would make a nice performance controller with midi and pd.

Flawed, or useable? (1)

zAPPzAPP (1207370) | more than 2 years ago | (#38547890)

I assume that things that sound the same in this demo, would be recognized as the same gesture.
It is shown that doing the same gesture produces the same sounds over and over again, so that gesture would be reliably recognized.

But, this would be extremly dependant on the type of surface and the spot on the surface, that the gesture is performed on. Changing either the spot on the surface, or to another surface entirely, would alter the outcome of the gesture, even though it is actually the same. That is also shown in the video.
How would they work around that?
Maybe by only looking at signals in relation to each other, not at the total signal?

Re:Flawed, or useable? (3, Interesting)

Andy_R (114137) | more than 2 years ago | (#38547944)

It depends what you want to use it for - as a replacement for a touch screen, it's probably not very useful because location information isn't going to be as easy to capture as timbral information, but as a musical instrument (or as a way of adding more sensitivity to existing electronic instruments, it's amazing.

I wonder if accurate position information could be captured by triangulating with 3 of these contact mics?

Re:Flawed, or useable? (1)

Andy_R (114137) | more than 2 years ago | (#38548008)

Quick back of an envelope calculation... speed of sound 340m/s, sampling rate of most D/A converters 44,100 samples/sec... 7mm accuracy (or better if the speed of sound in the solid object is lower)? Not great, but good enough to turn any solid object into a virtual keyboard.

Re:Flawed, or useable? (2)

Sleepyhead5 (1246752) | more than 2 years ago | (#38548418)

The speed of sound happens to be much higher in solids. The more rigid the material, the faster sound will travel through it.

Re:Flawed, or useable? (2)

CSMoran (1577071) | more than 2 years ago | (#38548438)

Quick back of an envelope calculation... speed of sound 340m/s,

In vacuum. In solid objects sound is typically an order of magnitude faster -- e.g. sound travels at approx 3400 m/s in wood.

Re:Flawed, or useable? (1)

Anonymous Coward | more than 2 years ago | (#38548534)

Speed of sound in vacuum? Fail.

Re:Flawed, or useable? (2)

CSMoran (1577071) | more than 2 years ago | (#38548620)


Yeah, I definitely need another coffee :D.

Re:Flawed, or useable? (0)

Anonymous Coward | more than 2 years ago | (#38548548)

In vacuum.

So, in space, people can hear you scream? I guess physicists have been wrong this whole time!

Re:Flawed, or useable? (1)

PPH (736903) | more than 2 years ago | (#38548596)

The speed of sound is much higher in rigid objects. But this has already been done (to a certain degree) with high end electronic drum kits that can vary their response based on the location (radially) at which the pad is struck.

They probably do some hardware DSP at much higher sample rates than 44K samples/sec. But this pushes the solution out of the realm of using a simple mic or two and a PC audio card.

Re:Flawed, or useable? (1)

White Flame (1074973) | more than 2 years ago | (#38549454)

It doesn't appear that the point of this project is to deal with positional information at all, but that it's a musical project where the type of contact the person does with the arbitrary surface is simulated as perturbing some physics-based audio source.

Multiple-mic positioning has been done before in other projects.

Re:Flawed, or useable? (0)

Anonymous Coward | more than 2 years ago | (#38550854)

Maybe I'm missing something because this isn't an area I've done any research in, but if you can use multiple microphones to capture position information, why would they have to be far apart, e.g. at the corners of the surface? Why couldn't they all be in the same small quarter-size device? Especially if you used echo information?

Re:Flawed, or useable? (1)

aaronb1138 (2035478) | more than 2 years ago | (#38552078)

I'm pretty sure with just two mics a fixed or known distance apart, you could get 2d coordinates. Just have to look at the timing offset of similar waves and calculate the radial intersection. On the flip side though, small CCD elements with sufficient resolution for a Wii/Smoothboard or Kinect style setup are more than cheap enough to use and relatively compact. A single mic as demonstrated might be useful for broadening the number of unique touch actions though.

Re:Flawed, or useable? (1)

ResidentSourcerer (1011469) | more than 2 years ago | (#38556698)

Two mikes give you the difference in distance to the source, if you are doing it just with timing. The solution to a point that is x cm from one mike and x+y cm from another is a hyperbola. Done with radio, and the other way around, you get Loran navigation.

Three mikes can intersect two hyperbolas for a location.

Positional accuracy is limited. Human hearable frequencies top out at about 18 KHz, which in air has a wavelength of about 1.9 cm. At CD audio speed, you can get 44,000 samples per second, which corresponds to about 0.7 cm. To do better than this you have to sample fast enough to get phase information. And to work with the speed of sound in most solids, you have to have another factor of 10 in sampling speed.

However you don't necessarily need high fidelity. I suspect that even 3-4 bit sampling at a megahertz or so would be sufficient.

Re:Flawed, or useable? (1)

brunozamborlin (2542970) | more than 2 years ago | (#38548090)

In this system the user can define a gesture by performing it just once. So users can easily define their gestures for every surface in few seconds. The next step would be to automatically recognise that the surface has changed so to avoid to re-train the system for every surface. Thanks for the interest!

cool (0)

Anonymous Coward | more than 2 years ago | (#38547928)

Anytime someone does something that expands the ways humans and machines interact, I think it's pretty cool.

Re:cool (2)

PPH (736903) | more than 2 years ago | (#38548518)

Mic your PC's case so it can respond appropriately when you smack it out of frustration.

So.. (1)

Anonymous Coward | more than 2 years ago | (#38547952)

Someone hooked up a microphone... Come on... Now 3 of these to track position may be news...

Theremin by a different technique? (1)

Psychofreak (17440) | more than 2 years ago | (#38547962) []
Seems like a different input technique...


Re:Theremin by a different technique? (1)

brunozamborlin (2542970) | more than 2 years ago | (#38548130)

:-) A (very) old project I've done in the past was actually "augmenting" a DIY theremin using it as a controller: []

Re:Theremin by a different technique? (1)

kcitren (72383) | more than 2 years ago | (#38549850)

Not even close.

cheers (0)

Anonymous Coward | more than 2 years ago | (#38548080)

cheers. and thank you for showing computer power + (single) sensor input = impressive. as far as i concern. this is the best example of what 20 years of computer era has given us. and no, that is not ironic. amazing. massive. and practical implementations on the fly. i'm speechless.

Nothing new (-1)

Anonymous Coward | more than 2 years ago | (#38548128)

Please refer to Microsoft Surface and ReacTable.

Don Cherry has one of these (4, Funny)

cangrande (199946) | more than 2 years ago | (#38548132) []

[spoiler: it's an amusingly-dubbed video of eccentric Canadian hockey announcer Don Cherry, who wears really loud clothes and makes vigorous table thumping gestures]

New? (0)

Anonymous Coward | more than 2 years ago | (#38548134)

I'm sure I saw someone do this years ago.

very cool (0)

Anonymous Coward | more than 2 years ago | (#38548212)

It looks like mapping of sounds to actions. A very cool idea indeed. Like those "corded" keyboards that had physical buttons (you "type" like playing a guitar). I wonder how many (realistically) different cords can be defined - would it be possible to "type?" Imagine sending text messages without having to look where your fingers are on the virtual keyboard! You could type in your pocket (okay - bad idea). The "keyboard" could be the back of the device - more screen for one to view and maybe a more natural grip.

Or maybe not a keyboard, but a menu / action selection device. Video editing (rewind, step whatever) without the need for expensive touch pads. Maybe it isn't practical to turn your table into an input service - but it could be a cheaper "touch" surface.

Cool, but perhaps flawed for most uses (1)

wanderfowl (2534492) | more than 2 years ago | (#38548238)

Acoustical modeling to determine the point of origin of sounds is nothing new, and although it's a wonderful idea to implement it as they are here, it'll go to hell the moment there are other vibrations on the surface. A low-pass filter should stop most environmental noise bouncing off the table/surface from triggering it, but if you put down your coffee mug on your desk, or bump your leg to the table, you'll likely get false input. Not to mention, as others have pointed out, the processing costs. This could be another of those technologies which is great for allowing input on an inert and durable psuedo-sterile and wipe-clean surface in a quiet, controlled room, but likely won't be worth much outside of bizarre use cases like that. But it's still amazing research.

I was looking for a video like this (1)

mshenrick (1874438) | more than 2 years ago | (#38548448)

Years ago, before youtube was really popular, I saw a video of a man doing a similar thing but couldn't find it recently!

I see, its a less useful version of this (3, Informative)

james_van (2241758) | more than 2 years ago | (#38548890)

saw stuff like this years ago. cant find the exact tech demo that i originally saw that used 3 mics for triangulation, but this video is pretty close. even mocked up a rough version (low precision, slow response time, but pretty damn good for an afternoon worth of work and very little experience with arduino) tied to a projector in my shop. customers had a blast with it, thought it was the coolest thing in the world. []

Source code or binary release? (0)

Anonymous Coward | more than 2 years ago | (#38548954)

Very cool idea!

Was there a source code or binary release planned? I'd certainly like to play with this software!

How much of this is real? (1)

Animats (122034) | more than 2 years ago | (#38550384)

I have some suspicions about this. They're getting more information out of one microphone than is usually possible. You might be able to extract some positional information by picking up the echoes off the edges of the object.

Keylogger? (1)

Dwonis (52652) | more than 2 years ago | (#38554740)

Hmm. I wonder how accurate of a keylogger could be made with something like that...

Saw this before (0)

Anonymous Coward | more than 2 years ago | (#38556770)

I saw this (although for a different purpose) years ago here [] .

piezo mic application (0)

Anonymous Coward | more than 2 years ago | (#38564246)

Six months ago, available now, free.


Scratch Input? (0)

Anonymous Coward | more than 2 years ago | (#38572938)

Seems very similar to Harrison's Scratch Input, shown at SIGGRAPH in 2009...

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?