Beta

Slashdot: News for Nerds

×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

How Seeing Can Trump Listening, Mapped In the Brain

timothy posted about a year ago | from the you're-just-envious-of-the-sensors dept.

Science 51

cortex writes "University of Utah bioengineers discovered our understanding of language may depend more heavily on vision than previously thought: under the right conditions, what you see can override what you hear. In an article published in PLOS One, 'Seeing Is Believing: Neural Representations of Visual Stimuli in Human Auditory Cortex Correlate with Illusory Auditory Perceptions,' the authors showed that visual stimuli can influence neural signals in the auditory processing part of the brain and change what a person hears. In this study patients were shown videos of an auditory illusion called the McGurk Effect while electrical recordings were made from the surface of the cerebral cortex."

cancel ×

51 comments

These processes helped us survive... (0)

Anonymous Coward | about a year ago | (#44783091)

... and there are always trade offs.

Of course there are going to be bugs, especially for a process that is mostly driven by blind forces. I'd be more interested to see how often in natural settings these systems work properly and/or are beneficial. Instead of engineering unnatural input to find flaws/bugs or sometimes just to mock our wetware and say "See our brains work poorly".

When I See Trump? (2)

Jeremiah Cornelius (137) | about a year ago | (#44783451)

I can't listen, either..

Kind of expected this... (4, Interesting)

Thantik (1207112) | about a year ago | (#44783117)

Ever happen to be looking for a street in an unknown area, and you end up turning down your radio? This actually increases your visual acuity slightly even though many may question your sanity when doing it. Many blind people have increased auditory capacity, and this has been known for a very long time. It doesn't seem all that far fetched that one (or more) of those senses could override the other. Maybe that meal that you hate tastes absolutely amazing...but looks so terrible that you taste it differently.

And conversely (1)

justthinkit (954982) | about a year ago | (#44783215)

And conversely, when an advertiser has to tell you the bad news about their product, the visuals crank up/switch to something compelling so you won't register the bad news.

Re:Kind of expected this... (0)

Anonymous Coward | about a year ago | (#44785109)

Visual acuity? Do you mean concentration? Or are they same thing?

Re: Kind of expected this... (1)

gzuckier (1155781) | about a year ago | (#44794095)

Similarly all the studies that even hands-free phone conversations interfere with visual attention when driving.

Trump listening? (1)

Anonymous Coward | about a year ago | (#44783139)

I think the Donald only listens to other right-wing conspiracy nutters

Re:Trump listening? (1)

Anonymous Coward | about a year ago | (#44783849)

But for a mere $45K you'll get to pose next to a lifesize cardboard cutout of Trump, and get a piece of paper certifying why nobody should let you get anywhere near their money.

Re:Trump listening? (0)

Anonymous Coward | about a year ago | (#44783869)

Please don't insult right-wing conspiracy nutters like that!

Narrator annoying (2)

I'm New Around Here (1154723) | about a year ago | (#44783145)

Why can't the woman shutup for 10 seconds so I can concentrate on the effect she keeps prattling on about? It makes it difficult to study when her voice is over top of the illusion half the time.

Re:Narrator annoying (1)

Anonymous Coward | about a year ago | (#44783327)

Why can't the woman shutup for 10 seconds...

 
Did you read your question? She's a woman. Duh.

Re:Narrator annoying (0)

Anonymous Coward | about a year ago | (#44784485)

Careful, I heard there was a woman on slashdot...

Re:Narrator annoying (1)

Behrooz Amoozad (2831361) | about a year ago | (#44784727)

Somebody Mod parent Insightful.

Re: Narrator annoying (1)

gzuckier (1155781) | about a year ago | (#44794107)

Mod this funny (intentionally I believe)

Did not notice effect at all... (1)

Rich0 (548339) | about a year ago | (#44783149)

After googling for the McGurk Effect and watching a bunch of videos I have concluded that I can't really sense this effect at all. I'll take their word that most people can.

Probably what is more interesting is some of the info on Wikipedia about the sorts of things that make the effect more or less pronounced. I'd be interested in the results for the average Slashdotter - I suspect that things like mild autism are much more common here.

Re:Did not notice effect at all... (1)

Qzukk (229616) | about a year ago | (#44783183)

Hadn't heard of this McGurk effect before. I was thinking it was something related to how I can still hear the noisy user interface elements in apps and games in my head when I have the sound muted and mouse over those menus that go "tink".

As for the McGurk video I saw of a man saying baa, it worked on me until the voice over told me the man was saying baa. Then I could look at the guy's lips and still hear baa.

Re:Did not notice effect at all... (1)

mikeabbott420 (744514) | about a year ago | (#44783209)

I heard the different sounds, bar and far, quite clearly and distinctly.

Re:Did not notice effect at all... (2)

ShanghaiBill (739463) | about a year ago | (#44783445)

I heard the different sounds, bar and far, quite clearly and distinctly.

Me too. The effect works on me. If I watch the video, I hear "fa". If I close my eyes I hear "ba". Even when I know he is saying "ba", I hear "fa". So I am normal.

Re:Did not notice effect at all... (1)

Trax3001BBS (2368736) | about a year ago | (#44786549)

I heard it as well...

When the culling comes I figure we're safe as we can "play the game".

Re:Did not notice effect at all... (0)

Anonymous Coward | about a year ago | (#44787159)

Basically, hearing is imperfect. For one thing, it doesn't work well over long ranges. For another, background noise. And for another: Accents

Because of these things, we tend to use more then just hearing to know what is being said. We also use our other senses, such as sight. We HEAR "aaa", we SEE them make a "B" sound, so we 'know' they said "Baaa". If we HEAR "aaa", and SEE "F", we 'know' they said "Faaa".

Re:Did not notice effect at all... (1)

Fallen Kell (165468) | about a year ago | (#44783231)

I too did not suffer from the "effect". I believe it has to do with the fact that I had 8+ years of vocal performance training and sang in various choirs and madrigal groups in many euro-classical forms and languages (German, Italian, French, multiple Latin dialects (yes, there are at least three that I am aware of, German Latin (used in many pieces, especially those of Bach and Mozart), Liturgical Latin (this is the Latin handed down and used through the Roman Catholic Church), and Italian/Roman Latin) and it does make a significant difference as to how the piece if performed as many of the rhythmic and poetic points are lost if using the dialect).

I have personally spent thousands of hours performing and listening to myself and others in the various groups, needing to develop an extremely critical ear for vow sounds and the phonetic makeup of words, especially the IPA (International Phonetic Alphabet). And in a group, especially in roles such as section leader (used in many larger groups to hear and fix issues) you need to hear when just one person made a mistake out of the group so you can point it out to be worked on and corrected.

I believe as a result of all this training, I do not suffer from this effect (at least on the linked video, to me it was very clear that the sound was still bah).

Re:Did not notice effect at all... (1)

Fallen Kell (165468) | about a year ago | (#44783271)

Oh, as a side note, 80-90% of all people are tone deaf and can not tell the difference between notes even as far apart as a septimal quarter-tone, unless they are played at the same time (causing a harmonic dissonance wave).

Re:Did not notice effect at all... (1)

Fallen Kell (165468) | about a year ago | (#44783283)

Oh, one more aside, many tuning systems do not even allow for notes of that different in tone to even be played (such as a piano, harp, trumpet, clarinet, flute, sax, etc., etc.,). Only stringed instruments (which are fretless or which the string can be bent to change the tension) and the trombone have this ability. Some instruments can tune their notes to use a scale which does include which uses septimal quarter-tone intervals as a part of it, but once so tuned, it then can not play the standard tuning notes and a separate instrument would need to be used to do so.

Re:Did not notice effect at all... (1)

Rich0 (548339) | about a year ago | (#44783787)

Hmm - that might or might not be the cause. I do play the cello and can sing reasonably well. I wouldn't put myself among the masters when it comes to intonation, but in general I would say that my intonation is quite good.

On the other hand, I have a friend who can sing harmony with just about anything by ear with excellent intonation and she definitely noticed the effect. This friend also has anomic aphasia, so who knows - there are a bunch of different neurological anomalies between us...

Re:Did not notice effect at all... (1)

the_other_chewey (1119125) | about a year ago | (#44784481)

Oh, one more aside, many tuning systems do not even allow for notes of that different in tone to even be played (such as a piano, harp, trumpet, clarinet, flute, sax, etc., etc.,).

That's mostly untrue for wind instruments. I can modulate a note on the
trumpet more than a halftone up and down easily. There's - mostly modern - music
for winds that notates quarter tones, and it's played with standard instruments.

Flutes seem to be especially easy to modulate, but I know many people capable of
doing this, playing all kinds of winds.

Very well known example: The opening of Gershwin's Rhapsody in Blue [youtube.com] ...

Re:Did not notice effect at all... (0)

Anonymous Coward | about a year ago | (#44784645)

Vowel sounds.

Re:Did not notice effect at all... (0)

Anonymous Coward | about a year ago | (#44783397)

I did not notice the effect at all while watching directly, but in one of the videos, the narrator talks over it, and in those instances, it did occur to me. Maybe it only happens if you're being overloaded with information?

Re:Did not notice effect at all... (1)

Jmc23 (2353706) | about a year ago | (#44783447)

It's all about ability to take cues from another persons face and be able to replicate them. If you can't put yourself 'inside' another person to learn a physical motion, then this 'illusion' won't work on you. Illusion isn't the correct term since it's in fact just a more dominant sense replacing input of a less dominant sense. They should actually be able to turn this 'illusion' on its head by testing people whose auditory senses are more dominant than their visual ones.

Re:Did not notice effect at all... (1)

Half-pint HAL (718102) | about a year ago | (#44788711)

It's all about ability to take cues from another persons face and be able to replicate them. If you can't put yourself 'inside' another person to learn a physical motion, then this 'illusion' won't work on you.

There is a theory that all language relies on reconstructing the speaker's movements, employing the mirror neurons. The most sophisticated model of this is V S Ramachandran's notion of language being a combination of mirror neuron action and synaesthesia (where stimulus from one sense maifests itself in another sense -- eg seeing colours tied to musical notes). The important thing about synaesthesia in Ramachandran's work is that he considers it a normal state of affairs, and demonstrates this with examples of metaphors embedded into language -- food tasting "sharp" and sounds being "bright". This study starts to support similar conclusions.

Re:Did not notice effect at all... (1)

Jmc23 (2353706) | about a year ago | (#44789875)

Well, very easy to disprove that theory. There are, after all, many unfortunate cases of people, like me, very deficient in these sympathetic mirror neurons that are/were totally unable to learn from others. Yet, i still had language. Granted, i did not understand words where the energetic purpose behind them created a specific subjective kinesthethic 'feeling', e.g., looking down, stand tall, chin up, stiff upper lip,etc...

I'm actually working on a system that breaks down subjective human experience into the myriad energetic possibilities, names those energetic precursors and allows translation between different languages through the energetic precursors and not the somewhat arbitrary language specific words we use. For example, the current colloquial 'sweet' has replaced prior words that carry the same energetic experiential rote such as 'bitchin' or 'rad', yet softened it for more general wide spread use.

Though it is only one aspect of it and it seems a major aspect when dealing with colloquial conversation, it may not be the be all and end all of langauges as he tends to think. Then again, it usually takes the broken people to notice where theories fall apart. Studying people with only 2d eyesight and why they can't see 3d illusions would have advanced the field significantly faster than current stuff put forth by people with 3d eyesight.

Re:Did not notice effect at all... (1)

Half-pint HAL (718102) | about a year ago | (#44799221)

Well, very easy to disprove that theory. There are, after all, many unfortunate cases of people, like me, very deficient in these sympathetic mirror neurons that are/were totally unable to learn from others. Yet, i still had language. Granted, i did not understand words where the energetic purpose behind them created a specific subjective kinesthethic 'feeling', e.g., looking down, stand tall, chin up, stiff upper lip,etc...

"Very deficient" is still very different from "totally deficient" -- the amount required for any base function is unclear. Besides, the role of the mirror neuron system in ASDs (which I'm assuming you're referring to) is still very much open to debate.

Re:Did not notice effect at all... (2)

the_other_chewey (1119125) | about a year ago | (#44784403)

After googling for the McGurk Effect and watching a bunch of videos I have concluded that I can't really sense this effect at all. I'll take their word that most people can.

Doesn't work for me either, not even a bit. "Ba" all the way.

I have made an observation that is semi-related to this though:

Watching a subtitled movie where the spoken language is totally opaque
to me, fast-paced dialogue can be tricky to follow. In this case, turning the
sound up helps me read the subtitles.

Brains are really weird.

Re:Did not notice effect at all... (1)

Carewolf (581105) | about a year ago | (#44788807)

The first video posted didn't work for me either, but the second one from BCC did.

No way... (0)

Anonymous Coward | about a year ago | (#44783203)

A 20Gb sensory connection having more power than a 20mb connection? Who would have thought that making sense of the world depends on having the most data possible?

Re:No way... (1)

INT_QRK (1043164) | about a year ago | (#44783645)

Mostly the opposite. Too much data is more likely to obscure needed information, and much more likely to not add anything over jut enough data. Anyone who has operated in a command/information center knows that for optimal performance you want just enough of the right information flowing to the right people and no more.

Hmmm (2)

slick7 (1703596) | about a year ago | (#44783273)

One picture, one thousand words, who'd a thunk it?

Seeing is believing (1)

udachny (2454394) | about a year ago | (#44783363)

Eventually we will all have cameras, audio and other recording devices integrated into our daily lives to the point, where it will no longer matter what you believe you saw, you'll be just able to go back to the actual recording and reproduce the fact of what you saw, that's how we'll get around this biological impediment, the same way we got around our physical impediment, which prevents us from flying on our own - we built air-planes.

Of-course NSA will be thrilled....

what comes around goes around (0)

Anonymous Coward | about a year ago | (#44783449)

Nice to see a new generation learning things I studied in school 20 years ago. No, this isn't news, but y'all are children so it's all good.

Re:what comes around goes around (1)

Half-pint HAL (718102) | about a year ago | (#44788741)

Nice to see a new generation learning things I studied in school 20 years ago. No, this isn't news, but y'all are children so it's all good.

Oh sorry... I forgot that McGurk had access to brain-activity-detecting electrodes and pinpointed the parts of the brain that experienced his illusion. Oh wait, he didn't, did he. The summary isn't brilliant, but this is news... they've shown that the McGurk effect feeds into the auditory channel, rather than simply overriding the audio signal at one of the later, more specialised language-processing areas.

Eye contact? (1)

PPH (736903) | about a year ago | (#44783769)

There was an article a few days ago about software to create the illusion of eye contact [slashdot.org] with video calls. As some people pointed out, continuous eye contact is actually somewhat disconcerting. When I'm listening to someone, I tend to watch their lips, particularly in high background noise environments.

Noticeable with Sarcasm (0)

Anonymous Coward | about a year ago | (#44783817)

When I'm joking around with a friend, and I say something like "ahh go fuck yourself" and I'm smiling, they take it by my facial expression that I'm kidding around and not by my language which could otherwise be construed as insulting.

a shocker (0)

Anonymous Coward | about a year ago | (#44783913)

I never would have thought that people would fall for the visual appeal of something rather than listen to what is actually being said!
Next breaking news: Vanity results in bad decision making.

Lip position (1)

sopwath (95515) | about a year ago | (#44784013)

How can he physically produce the "Baa" sound when his bottom lip is tucked behind his teeth? His lips don't press together when the illusion is supposed to make it sound like he's saying "Faa" instead. I think he's actually making two different sounds.

Isn't the "Baa" sound impossible to make without the lips pressing together? Isn't the "Faa" sound impossible to make without blowing on the lip-teeth connection with the top and bottom lips separate?

Re:Lip position (3, Informative)

GreyWanderingRogue (598058) | about a year ago | (#44784477)

How can he physically produce the "Baa" sound when his bottom lip is tucked behind his teeth? His lips don't press together when the illusion is supposed to make it sound like he's saying "Faa" instead. I think he's actually making two different sounds.

Isn't the "Baa" sound impossible to make without the lips pressing together? Isn't the "Faa" sound impossible to make without blowing on the lip-teeth connection with the top and bottom lips separate?

This is the perfect description of what your brain is doing. Unfortunately, it appears you're misunderstanding what is actually going on. There is only one audio recording. It is dubbed over both clips. Check the section where there is a side by side with both videos. What you hear depends on what side of the screen you look at, even though there is only one audio track.

Re:Lip position (2)

the_other_chewey (1119125) | about a year ago | (#44784509)

How can he physically produce the "Baa" sound when his bottom lip is tucked behind his teeth?

He can't...

Isn't the "Baa" sound impossible to make without the lips pressing together? Isn't the "Faa" sound impossible to make without blowing on the lip-teeth connection with the top and bottom lips separate?

...that's why it's the same sound dubbed over different videos.

another example (taken down) (1)

Black Parrot (19622) | about a year ago | (#44784339)

There was a supposedly entertaining example of this on one of the astronomy blogs a while back, but they got a takedown notice before I saw it. Changed what you heard in a music video, IIRC. Does anyone know where we can find that?

Music competitions (1)

ath1901 (1570281) | about a year ago | (#44785219)

It reminds me of this:
http://www.economist.com/news/science-and-technology/21583974-top-musicians-are-judged-much-their-movements-their-melodies [economist.com]

In short, people were bad at guessing who won the competition when they could only hear the music and not see the performers. Professional judges were just as bad as novices (worse when they could see the performance but just as bad when they could only hear it).

Interesting stuff.

Oh dear, another Slashdot summary (0)

Anonymous Coward | about a year ago | (#44785887)

University of Utah bioengineers discovered our understanding of language may depend more heavily on vision than previously thought

If understanding of [spoken] language depended on vision, blind people wouldn't be able to understand it.

And the McGurk effect is nothing new. It was first described by Mr McGurk and Mr McDonals in 1976.

Finally Explains Sounds of the Northern Lights! (1)

deviated_prevert (1146403) | about a year ago | (#44788111)

Being a Canuck that has lived north of 60 I have seen incredible instances of the Northern Lights. What I always thought was strange was when the lights were in certain patterns and intensities I heard a strange crackling like static electric charges. Knowing that the lights originated from well beyond the atmosphere I just dismissed this as a fantasy. UNTIL one day having a beer in a pub in Watson Lake Yukon I overheard several people claiming to hear the lights the same way as I do!

THIS article finally confirms my suspicion that it is a normal response within some individuals to easily perceive light in an auditory fashion. Perhaps this is why mood lighting is so effective and disco balls were such a hit though they are essentially useless.

All though this explanation [space.com] from some scientists defies this logic, I have been with other people in the pitch dark watching the lights at the same time and they did not hear the sounds I was experiencing and the sound was not at all as described in the article on Space. The sounds that happen to me are definitely like an electrical spark crackling not at all like applause!

Not really news... (1)

neminem (561346) | about a year ago | (#44797659)

They "discovered" that? I remember a demonstration of this in a phonetics class, which I took in like 2002. Though my guess is that it's only the summary that's wrong, and the researchers themselves didn't claim to "discover" anything, but were simply doing new neurological research on the effect, given that the summary itself mentions the McGurk effect. Quoth Wikipedia, "The McGurk effect is sometimes called the McGurk-MacDonald effect. It was first described in 1976 in a paper by Harry McGurk and John MacDonald entitled 'Hearing Lips and Seeing Voices'."

Check for New Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Create a Slashdot Account

Loading...