Multisensory Development

Multisensory Development

1. Goals and outline

   • Understand developmental research methods and findings. Interpretation of some of these in terms of : 1. neural mechanisms: - Animal models - EEG measures. 2. models of information processing: race model (response times), Bayesian model (precision)

2. Revision - unisensory development

   • Vision
     • At birth: Eye control: poor accommodation (focus), poor vergence (both eyes converging on same target), ‘jerky’ eye movements. Orients to patterns and faces, but limited visual acuity (ability to see detail) and contrast sensitivity. At 1-3 months: Development of better accommodation, vergence, and smooth eye movements for tracking moving targets. Better acuity and contrast sensitivity, emerging abilities to distinguish motion, orientations, patterns, and stereo (binocular) depth. At 4-8 months: Acuity and contrast sensitivity getting closer to adult levels. Global organisation of stimuli, e.g. coherent form and motion. Discriminate different faces reliably. Depth from pictorial cues. Biological motion.
   • Audition
     • At birth: Recognise sounds and voices heard in utero. Prefers mother’s voice to stranger’s. Localise sounds in space. At 1-3 months: Distinguish different speech sounds. At 4-8 months: Increasing range of sensitivity to pitch (high and low). Distinguish auditory patterns, e.g. different pitches, rhythms, melodies. At 9-12 months: Being to lose the ability to discriminate speech sounds not used in the language (--> perceptual narrowing)
   • Single senses develop post-natally. Most events in the world are signalled to more than one sense. What can we gain by combining information across senses (multisensory processing)? How does multisensory processing develop?

3. Use of multisensory information

   • What can we gain by combining information across senses (multisensory processing)?
     • Complementary - e.g taste and touch for an apple.
     • Redundant - e,g, vision and audition for approaching cars
     • Uses of redundant information (=same info across more than one sense) - Property = location of car, sensory cues are vision and audition that both link to the property!
       • Advantages of redundancy --> Flexibility - use one cue when other not available. Correlations between cues allow us to separate and attend to important objects. When more information is available, it is possible to make faster or more accurate decisions
         • Uses of redundant information in multisensory development
           • 1. Flexibility - use one cue when other not available - need to know "cross-model correspondence". When are these learnt? 2. Correlations between cues allow us to separate and attend to important objects. Ability to spot and use these correlations in infancy: "intersensory redundancy" hypothesis. 3. When more information is available, it is possible to make faster or more accurate decisions - information processing models - comparison of infants and children with predictions of these!

4. Flexibility - use one cue when other not available. Need to know cross-modal correspondences. When are these learnt?

   • Cross-modal correspondences
     • Molyneux’s Problem
       • Letters from William Molyneux to John Locke, 1688 and 1693 - Suppose a Man born blind, and now adult, and taught by his touch to distinguish between a Cube, and a Sphere of the same metal, and nighly of the same bigness, so as to tell, when he felt one and t'other; which is the Cube, which the Sphere. Suppose then the Cube and Sphere placed on a Table, and the Blind Man to be made to see. Quaere, Whether by his sight, before he touch'd them, he could now distinguish, and tell, which is the Globe, which the Cube.
       • Are cross-modal correspondences innate or learnt? Locke’s view: learnt. (the blind man would not be able to do it). More recently – experimental approaches to the same question. Studies with infants, including newborns. Studies with congenitally blind patients who had their sight restored
     • Studies with infants
       • 1-2 month old infants: At 1 mo, transfer of texture (smooth vs granular) and substance (hard vs soft) from mouth to vision (Gibson & Walker, 1984; Meltzoff & Borton, 1979). But some criticisms and results not always reproduced (Maurer et al, 1999). At 2 mo, visually recognise shape of an object previously held in the hand (Streri 1987). But: could they have learnt the correspondences in the first 2 months? Also: texture and material not “geometric” properties (not Molyneux’s Q).
     • Studies with newborns
       • Newborns: Newborns can visually recognize the shape of an object that they have previously manipulated with their right hand, out of sight (via “intersensory paired-preference procedure”) – Streri, 2003. haptic familiarization phase - newborns given an object to explore manually without seeing it (cylinder or prism). visual test phase – infants shown the familiar object paired with a novel one for 60 seconds. Not so with the left hand (Streri, 2004), and for shape transfer is uni-directional (touch to vision but not vision to touch); Streri, 2007.
     • Studies with blind patients
       • Blind patients after removal of congenital cataract - Project Prakash. Touch to vision matching is at chance just after operation, but is learnt within a few days (Held et al, 2011)
     • Innate or learnt -conclusions
       • Note that there are many different cross-modal correspondences. Molyneux’s problem concerns SHAPE, quite an abstract property which presents itself to vision and touch in very different ways. contrast visual-haptic texture or material; audio-visual location – more on this shortly
         • For SHAPE in any case, the results are not entirely clear… NEWBORNS seem to be above chance at matching in some situations - suggests some innate ability to match shape across senses. But note newborns’ very basic visual perception (vision lecture) – in acuity as well as ability to process contours/edges. May be doing the judgment via different mechanisms to adults (adult shape perception mechanisms have not yet developed). BLIND ADULTS with sight restored seem to be at chance, but learn in 1-2 days. Direct answer to Molyneux’s problem. But NB long period of visual deprivation could have caused them to lose some abilities – including in theory even those that newborns have. Note also more difficult stimuli than those presented to newborns
     • What about other correspondences?
       • Increasing pitch - ball going up or down? High pitch - light or dark. Spikey shape (kiki) vs squishy shape (bouba). Which is which? There is evidence that young infants are sensitive to many of these. Perhaps even more so than older children or adults – “Infantile synesthesia” hypothesis – Maurer. Reviews: Maurer & Mondloch, 2004; Spector & Maurer, 2009
     • Conclusions
       • Molyneux’s problem is a classic problem in philosophy and perception concerning the origin of cross-modal correspondences – innate or learnt? Evidence from recent experimental research with infants, newborns and blind patients. Note that SHAPE is one kind of correspondence – there are many others

5. Correlations between cues allow us to separate and attent to important objects - ability to spot and use these correlations in infancy - "intersensory redundancy" hypothesis.

   • Correlations between cues - "Intersensory redundancy hypothesis"
     • The problem: Many visual, auditory, tactile events going on all the time. Which ones belong together? Solution? Timing. Events that go together have “amodal” features in common such as temporal synchrony, rhythm, tempo.
     • Bahrick, Lickliter - e.g. Bahrick et al, 2004; Bahrick & Lickliter, 2012. Internsensory redundancy = “overlapping information across auditory, visual, tactile, and/or proprioceptive stimulation for properties of objects and events”. Can alert infants to what to attend to – “selective attention”. Selective attention in turn guides perception and learning.
     • Contrast
       • Visual vs. haptic shape: The two cues work quite differently. Making links between them requires some prior knowledge (whether learnt or innate). Vision and audition from regular events: Timing cues such as synchrony are a strong indication that they go together. Can use to segregate from other visual and auditory events going on at the same time.
     • The proposal (see Bahrick & Lickliter, 2012): Young infants tune in to redundantly specified, amodal properties of stimuli. An account of how “young infants, with no prior knowledge of the world, rapidly come to perceive unitary events and attend to stimulation that is relevant to their needs and actions”. Example: audio-visual speech. The rhythm/timing of the speech is amodal (common to both senses). Prediction: infants first allocate attention to this. Visual features of the speaker and sound of the voice are unimodal. Prediction: infants allocate attention to these later
       • Some evidence - Some evidence (review, Bahrick & Lickliter, 2012 – this study, Bahrick & Lickliter, 2000). 5 month-olds see and/or hear a hammer tapping a surface to a rhythm. After habituation looking time goes down (they get bored). They change the rhythm. Will they show more interest (longer looking)? Prediction: will have noticed/learnt the rhythm better in the two-cue (AV) condition than given single cues. --> Redundant information across senses guides attention and learning
     • Also for more complex stimuli and social cues – e.g. face/voice/emotion. Role of amodal properties in guiding attention. “Attention can be ‘educated’ to amodal properties of events in unimodal stimulation by pre-exposing infants to salient intersensory redundancy, which serves to highlight amodal properties. It appears that infants continue to detect those same amodal properties in familiarized events, even when redundancy is eliminated”. Review: Bahrick & Lickliter, 2012
     • Relations to classic account of perception and learning
       • Piaget – perception is difficult – need to learn which cues to use and how they go together. JJ Gibson – perception is easy – the information is out there (e.g. strong correlations between cues that go together). The IRH account is in keeping with Gibson – for simple cues. But more complex or less well correlated cues do need to be learnt..
     • Shifts in infants' attention to sensory cues while learning language – Usefulness of visual information for understanding speech
       • Lewkowicz and Hansen-Tift 2012 - Eye tracking infants watching videos of faces speaking. Switch eyes to mouth to benefit from redundant cues while learning to process speech. Switch back once understand speech well in order to use social cues. This happens later with non-native speech (still trying to understand it – keep attention to mouth.
     • Conclusions
       • While some correlations between sensory cues may need prior knowledge and/or learning, others are readily available in the environment. Especially those provided by timing (events happening in sync). “Intersensory redundancy hypothesis” – this information guides attention and learning in infancy. (Bahrick, Lickliter). Application to language learning – infants use multimodal (visual and auditory) information flexibly to get better information (Lewkowicz)

6. When more information is available, it is possible to make faster or more accurate decisions - Information processing models - comparison of infants and children with predictions of these.

   • Multiple cues for faster or more accurate decisions
     • Example: audio-visual localisation. Neural substrates and post-natal development (animal model). How having two cues can make decisions faster (information processing model). Behavioural measures of audio-visual localisation in human development and comparison with model. Will return to ACCURACY later
     • Audio-visual localisation: neural substrates and post-natal development
       • Important substrate for audio-visual localisation: superior colliculus – linked to “automatic” orienting – eye and head movements. Animal (cat, monkey) model - review, Wallace & Stanford, 2008.
         • 1. Neurons in cat SC combine multisensory (incl visual and auditory) information – “superadditive” property. Meredith & Stein, 1986
         • 2. Neurons are not multisensory at birth - this develops post-natally. Wallace & Stein, 1997
         • 3. Development depends on experience. Cats raised in an altered environment (sights and sounds come from opposite directions) - Wallace & Stein, 2007. Multisensory neurons develop that respond to these spatially separated cues in the same way that neurons in normally raised cats respond to co-localised cues. Plasticity of multisensory integration (of this kind at least) – depends on what the animal experiences during development
       • Based on the animal model: We expect that human audio-visual orienting will also develop post-natally based on experience. But we cannot record from single neurons in human SC. EEG is also not a reliable way to record from SC. --> Need to assess integration using a behavioural test
     • Audio-visual localisation: information processing model
       • Behavioural test for multisensory integration. Can we just ask whether behaviour is faster / more accurate given two cues vs one? No: not all kinds of improvements are evidence for multisensory integration. Need not just to ask whether there is an improvement, but to compare performance with a model predicting effects of integration on behaviour.
       • Question: How can having two cues rather than one make responses faster or more accurate? Will consider just speed for now. Raab (1962): having two info sources in parallel can speed up response times even if they are not integrated, via a “race”. How the “race” works (audio-visual example) - comparing performance in single-cue trials (A or V) and combined-cue trials (AV). Processing times vary randomly trial to trial (hence reaction times are variable). So on some trials A is faster, on some trials V is faster. Given A+V together, a subject could process them independently but always respond using the fastest single cue (always pick the winner of a “race”). By doing this over a whole set of trials they would reduce their average reaction times in AV trials compared with either – but without having integrated the signals
       • Miller (1982) proposed a now widely used model that predicts the most benefit subjects could get just by processing cues in parallel (picking winners of races). Any speed gains greater than this are usually taken as evidence that the cues are being integrated (although see Otto & Mamassian, 2012). For reasons explained in the paper, this prediction comes simply from the sum of the cumulative distributions of single-cue reaction times
     • Audio-visual localisation: Human development?
       • To recap: We expect audio-visual integration for spatial orienting to develop post-natally, based on an animal model. Can we test for this behaviourally, in humans? Measure: reaction time. But some reaction time gains could be obtained without combining signals. So we cannot simply use reaction time gains as evidence for integration - but we can ask if these gains exceed those predicted by parallel processing (a “race”). This is what Neil et al (2006) did… Note – intended for prospective parents/participants. But provides a nice overview of some of the stimuli, including those in the 2006 paper (this starts at around 3m:40s). Also provides some practical info for you about how baby studies are run. Q: from the video – how were they testing for audiovisual integration in the task with the lights?
         • Neil et al (2006) – compare auditory (A), visual (V), and audiovisual (AV) orienting. 0- to 10-month-olds and adults. Centre attention via central light and sound. Play sound and/or flash light at the side, 25 or 45 degrees from centre (conditions: A, V, AV). Record how soon an eye movement happens – faster for AV than A or V? All conditions get faster with age; AV tends to be faster than A or V at all ages; “Race” between independent cues or multisensory integration? Race model: 8-10 mo and adults (but not younger groups) faster than theoretical maximum via a race, evidence for integration. Interpretation: sensory integration for audiovisual orienting also develops post-natally in humans – late in the first year of life. Link with experience-dependent development of superior colliculus (animal models).
       • So, is audiovisual integration mature by the first year of life? Some conflicting results: in later childhood, for participants pressing buttons, there seems to be much later and possibly “u-shaped” development – e.g. Barutchu et al 2008. Note – age is now in YEARS. AV better than A or V. But not better than race prediction at all ages
       • Interpretation - There are multiple sites of sensory convergence in the brain - increasing evidence for multisensory interactions even in “traditionally” unisensory areas such as visual cortex. Superior colliculus for spatial orienting is one dedicated, perhaps early developing mechanism. Other kinds of audiovisual integration need not use the same mechanisms or develop in the same way. Barutchu et al – detection of a central target and voluntary button-pressing (not “automatic” orienting). Future studies should distinguish which of these aspects of the task make a difference (e.g. being automatic or not, need to localise or not). See Nardini et al (2015)
       • Audio-visual localisation: Conclusions
         • Neural substrates and post-natal development (animal model). How having two cues can make decisions faster (information processing model). Behavioural measures of audio-visual localisation in human development and comparison with information processing model. Evidence for post-natal development of A-V integration for spatial localisation – consistent with animal model. But also seemingly much later and “u-shaped” development in a slightly different task. Multiple mechanisms for sensory integration – it may not always be clear which one a task is tapping into
     • This is all about faster. What about accuracy?
       • Quick intro to speed vs accuracy. Which should we measure in a task? It is always possible to make a trade-off (sacrifice speed for accuracy and vice-versa). So someone who has sped up need not be doing “better” – they may be making more errors. Someone who is making fewer errors may have done this just by slowing down. Approaches – Measure both speed and accuracy. Make the task very easy and just measure speed (like audiovisual orienting). Make the task very difficult and just measure accuracy (as we will see here)
       • Precision gains via cue combination - E.g. Judging where the bird is in the tree. Information available from audition and vision. The optimal strategy for making precise estimates is taking an average*. This averages out some of the random noise. *in which each cue is weighted by its reliability (inverse variance). See Ernst (2006).
         • This is the same principle we use when running experiments – collect data on multiple trials and subjects. We average over these to get rid of random noise and so get closer to the true answer. Here the perceptual system is playing the role of experimenter/statistician. This approach is additionally taking into account that not all the measures need be equally reliable (--> weighted averaging). This is a theoretical model (Bayesian cue combination) of how to improve precision given multiple uncertain sources of information. Do humans actually do this?
           • Do humans actually do this? Answer: in the case of adults, very often, yes – they combine cues, weighting them exactly according to their reliability. Challenge: Subjects must “know” the reliabilities of different cues in order to do this. Vision and touch - Ernst and Banks 2002. Vision + audition. (Alais & Burr, 2004). Multiple depth cues within vision (Hillis et al, 2004). Challenge: Subjects must “know” the reliabilities of different cues in order to do this. Question: When and how does this ability develop? In difficult perceptual or motor tasks, will children combine multiple cues, taking their reliability into account, to be more precise?
             • Gori et al (2008) tested children’s abilities to combine visual and haptic (touch) information in order to judge block size (expt 1) or line orientation (expt 2). Which of two blocks is taller? Conditions: Haptic only; Vision only; Crossmodal [Haptic+Vision], agree. Test for precision gain, compare with model prediction (“MLE”); Crossmodal [Haptic+Vision], small conflict. Determine Ss’ relative weighting for (reliance on) the two cues, compare with model prediction (“MLE”). Results - precision: Before 8 years children did not improve their precision when given both cues together vs either one alone. Results - weighting: For size judgments they over-rely on haptic information, even though it is unreliable.
             • Nardini et al (2008) tested children’s abilities to combine visual and nonvisual (e.g. vestibular) information in order to navigate to a spatial location. Results: Similarly, to age 8 years children did not improve precision by combining cues. Pattern of results best explained by switching trial to trial in which cue they use, but not averaging them
             • Across a range of different kinds of tasks using different sensory modalities, little evidence for precision gains via cue combination until after 8 years of age - Visual-haptic task Gori et al, Curr Biol 2008; Navigation Nardini et al Curr Biol 2008; Visual slant estimation Nardini et al, PNAS 2010; Vis-aud localisation Gori et al, Fr Int N’sci 2012; Pointing Nardini et al, JEP:HPP 2013; Haptic-auditory size Petrini et al, Dev Sci 2014
             • Conclusion – while some multisensory abilities develop early, the ability to improve precision by combining multiple estimates develops very late in childhood. Why? Some challenges for learning to make precision gains via cue combination: Know which cues go together; Be able to average estimates across modalities into a single value; Know the reliabilities of the cues. How do children learn to do this? Very long process of perceptual learning during childhood (see vision lecture); Brain changes, including connectivity, that could support better cue combination; One sense is needed to initially calibrate others? See Gori et al (2010)
   • Conclusion
     • Strengths of modelling approach – can conclude how information processing changes during childhood – examples: Race model: test for multisensory integration in a speeded task; Bayesian model: test for multisensory integration in a difficult perceptual task. In each case, expts have to be planned in order to provide data that can be analysed in this way (e.g. combinations of single- and multiple-cue conditions)

7. Summary

   • Senses provide different, complementary information about the world. Benefits of redundant information (one event --> multiple sensory cues) - Flexibility, separate and attend to objects, be faster or more accurate .Development of aspects of multisensory processing: Some cross-modal correspondences from birth – others learnt later; Amodal information across senses guides attention and learning in infancy; Audiovisual speed gains - post-natal development (animal model, race model); Precision gains – very late development (Bayesian model). Significant sensory learning underlying development of multisensory abilities throughout infancy and childhood