Please enable JavaScript.
Coggle requires JavaScript to display documents.
Multimodal (first 3 chapter) (Multimedia & Multimodality (Medium vs.…
Multimodal (first 3 chapter)
Multimedia & Multimodality
Medium vs. modality
Multimedia systems vs. multimodal systems
Modality relations
Speech communication as multimodal interaction
Benefit of multimodal systems
Hearing & Speech
Hearing
Ear
Outer ear
Directional hearing it consist of head, 'ear'
Pinna : individual shape, inter-ear distance 18cm
Ear canal : length ≈ 25 mm, diameter ≈ 7 mm.
Ear drum : Pressure sensor & Excitation of oscillation due to sound waves
Main function : Directional filter
Middle ear
impedance matching air, Ear drum, ossicles
low sound pressure & high particle velocity at the eardrum
Inner ear
pre signal processing cochlea, auditory
Fourier analysis: Decomposition of complex sounds into components: frequency to place transformation
cochlea
2.5 windings, length ≈ 34 mm (unrolled)
Inner haircells (IHC), appr. 3500
Outer haircells (OHC), appr. 25000
IHC are responsible to transmiting the information toward the brain.
OHC kind of amplifier, recieve the information from brain
Membrane system
Psychoacoustics
Speech
Function
Phonetics
Phoneme
Smallest sequencial unit of sound
Phone: smallest phonetic unit of speech
morpheme: smallest unit of speech which carry meaning
Vision
Visual perception
Gestalt perception
Depth perception
Motion perception
Object moves, observer follows object with the eyes
Observer moves through stationary environment
Object moves, observer is stationary
Reading