It’s a well-accepted fact that speechreading helps people (in all hearing-deficit situations) radically improve their speech perception. But how much do we really know about the cognitive capabilities of clients/patients who are using speechreading with various levels of success? This paper provides an overview of the cognitive research that a distinguished group of researchers in Sweden has carried out in the areas of visual speechreading, speech understanding with cochlear implants and tactile aids, and speech recognition with hearing instruments.

About 20 years ago, researchers at Linköping University in Sweden hypothesized that visual speechreading—and everyday practice from being a deaf or profoundly hearing-impaired speechreader—was sufficient to establish superior speechreading skills compared to a normal-hearing, age- and education-matched group. After a number of studies during the 1980s and early 1990s, we finally accepted the null hypothesis and conclusion that visual speechreading abilities do not improve spontaneously as a function of long-term, everyday exposure to, and demands on, visual language communication.

Compensation (ie, better performance or selective enhancement in certain speech conditions for the deafened group compared to a matched control group) was not found. This result held true irrespective of the type of presentation (video vs. real-life),1 type of materials (digits vs. discourse),2 for just-follow conversation tasks,3 for different durations of impairment, and for different levels (in dB) of hearing loss.1,2,4,5 In general, it seems that spontaneous improvement or compensation is a cherished myth rather than a definite reality.6,7

Acquired and Congenital Hearing Loss
Despite this apparent generality, there is an important constraint to be noted: In our studies, we are referring to the situation for adults with acquired hearing loss, not to individuals who have a congenital loss, or to those who have experienced a very early onset of the hearing impairment. Referring to the latter cases, Bernstein, Tucker & Auer8 have found signs of compensatory effects for early and profoundly hearing-impaired users of hearing aids on vibrotactually mediated speech perception (this work was further supported by the research of Bernstein et al9).

More recently, Bernstein, Demorest & Tucker10 have demonstrated that experience from speech perception without hearing in the severely hearing-impaired individual can result in enhanced visual phonetic perception which, in turn, is associated with such abilities as visual sentence perception. This possibility is also obvious from one of our own studies by Tillberg et al,11 where we clearly found that early and profound hearing loss contributed to improved visual speech perception.

It might be added, in this context, that one should perhaps not be too surprised that compensation exists as a more general possibility relative to brain and language interactions. The growing cognitive literature on early acquisition of signed language and its facilitative impact on cognitive functions (eg, attention, spatial functions, and memory) also testifies to the possibility of cortical plasticity, neural reorganization, and concomitant perceptual-cognitive changes in the deaf (for a review, see Rönnberg, Söderfeldt & Risberg12).

In view of this, our generalization about compensation obviously was a bit too pessimistic. However, within the limitations of this generalization, the combined empirical picture—across groups with normal and impaired hearing, as well as normal variations of skill—suggests that important parts of the variation in speechreading skill can be explained by individual differences in cognitive and perceptual subskills (for an overview, see Rönnberg et al.13).

Predictors of Sentence-Based Speechreading
Briefly, direct predictors of sentence-based speechreading are represented by skill at word decoding tests (lipped words, as opposed to mere discrimination of lip movements),14 and certain aspects of the speed of information processing, such as lexical and semantic access speed, as opposed to physical and name-matching speed.5 When verbal inference-making is assessed by a sentence completion test rather than a word completion test, it predicts successful sentence-based lipreading.4 Indirect predictors of sentence-based lipreading are:

  1. VN 130/P200 peak-to-peak amplitude measure in the visual evoked potential15;
  2. Complex information processing and storage tasks, measured by the reading span task4; and
  3. Verbal ability.16

Skilled Speechreaders
On the basis of a set of case studies of extremely skilled speechreaders, the importance of these cognitive abilities seems to be unvarying across communicative forms and onset of impairment. That is, cognitive functions add to the general capacity of communicating with poorly specified signals. However, there is one qualification to be made: We assume—and have shown in the case studies of extreme speechreading skill17-19– that bottom-up processing is important up to a certain threshold only,20 but is generalizable to both children21 and adults.22 The existence of such a threshold is presumed to be due to the efficiency of visual, phonetic, and lexically supported perception being constrained by visual-neural and lexical access speed, as well as by the quality and speed with which phonological representations are accessed.23,24

To reach proficiency, well-developed, more compound cognitive skills are also necessary. Published case studies (eg, GS18, MM19, and SJ17) all demonstrate this point. Speed of lexical decision making, decoding of lip-spoken words, and phonological functions are relatively normal or intact, whereas top-down, compound information-processing skills are superior. It is these skills that allow speechreaders to surpass the hypothetical threshold of low-level perceptual information extraction, and hence attain extreme speechreading skill.

Framework for a Working Memory System
A framework for a general working memory system for the processing of poorly specified language input has been sketched.25 In its general form, the framework is an expression of the continuous interaction between perceptual input channels, working memory, and long-term memory. In other words, information retrieved from long-term memory about a particular communicative situation or topic has a top-down influence (eg, in the form of schema or semantic knowledge) on the processing of a particular perceptual input. The more degraded, distorted, or unintelligible the signal, the greater the contribution from long-term memory and, conversely, the more specified the perceptual input (sometimes conveyed by several channels in concert), the less the need for perceptual enrichment (see Samuelsson & Rönnberg26 for a model).

The balance between top-down and bottom-up processing in speech understanding is also related to the general capacity for processing and storage of information in an on-line working memory system. A larger on-line capacity will allow for more inference-making and intelligent guesswork during speech processing. Phonological processing skills—part and parcel of a working memory system for poorly specified input— draw partly on the phonological representations of words that exist in long-term memory, as well as on the capacity of the phonological loop in working memory.24 The mapping of perceptual input to phonological representations—processed and mediated via the loop in working memory—is thus also important to consider. Andersson24 has shown that the loop per se remains intact in working memory, but that phonological deterioration takes place in long-term memory for longer durations of deafness or hearing loss.

The framework consists of three general parts:

  1. Multimodal language input channels;
  2. A semi-abstract phonological processor—allowing for information about phonology from nonsound sense modalities; and
  3. A set of general, amodal processing constraints. The amodal constraints are: early lexical access, processing speed, predictive inference-making, retrospective disambiguation, and simultaneous processing and storage capacity—cognitive constraints which are crucial to the understanding of poorly specified language signals.25 The hypothesized and detailed interactions among the component skills that underlie these constraints in working memory (eg, during a dialogue or within the delineation of a process model) is beyond the scope of this article. In our research, we have primarily used an individual difference approach to assessment of phonological and amodal processing subskills.

Application of the framework has demonstrated that roughly similar predictors play a significant role when it comes to efficiency of tactile aid use, cochlear implant functioning, and to hearing aid use. For example, the type of tactile display does not have any major effect in a short-term tactile-training speech-tracking program (10 weeks), whereas cognitive predictors (especially word decoding and rhyme tasks using long-term representations of phonology) account for major portions of the performance.13

Further systematic work also underscores the importance of cognitive predictors for transfer of tactile benefit and for different types of speechreading criteria.27,28 Cognitive predictions of speech understanding with cochlear implants are tapped by both rhyme tasks and by the reading span task, which assess simultaneous information processing and storage capacity.29 The ability to take advantage of different noise signal processing modes in hearing aids also —and very interestingly—seems to capitalize on high capacity for simultaneous information processing and storage (again assessed by the reading span task).30

It is quite clear to our research group that cognitive psychology yields new insights into speech communication relative to impoverished signals. Explicit, theory-driven cognitive tests seem to be useful prognostic tools in aural rehabilitation and alternative communication. It is easy to see that this type of approach can be implemented for screening, assessment, and rehabilitative purposes in the clinic. Such work is currently under way in Sweden.

This research is supported by a grant from the Swedish Council for Social Research (30305108). The information in this article was originally presented at the International Adult Aural Rehabilitation Conference held in Portland, Maine, in May 2001.

Jerker Rönnberg, PhD, is a researcher at the Department of Behavioural Sciences and the Swedish Institute for Disability Research (SIDR), Linköping University, Linköping, Sweden.

Correspondence can be addressed to HR or Jerker Rönnberg, PhD, Department of Behavioural Sciences, Linköping University, S-581 83 Linköping, Sweden; email: [email protected]

1. Rönnberg J., Öhngren G, Nilsson L-G. Hearing deficiency, speechreading and memory functions. Scand Audiol. 1982;11: 26l-268.
2. Rönnberg J, Öhngren G, Nilsson L-G. Speechreading performance evaluated by means of TV and real-life presentation: A comparison between a normally hearing, moderately impaired and profoundly hearing-impaired group. Scand Audiol. 1983; 12:71-77.
3. Hygge S, Rönnberg J, Larsby B, Arlinger S. Normal and hearing-impaired subjects´ ability to just follow conversation in competing speech, reversed speech, and noise backgrounds. J Speech Hear Res. 1992;35, 208-215.
4. Lyxell B, Rönnberg J. Information-processing skills and speechreading. Brit Jour Audiol. 1989;23, 339-347.
5. Rönnberg J. Cognitive and communicative function: The effects of chronological age and "handicap age". European Jour Cognitive Psychol. 1990;2, 253-273.
6. Rönnberg J. What makes a skilled speechreader? In: Plant G & Spens K, eds. Profound Deafness and Speech Communication. London: Whurr Publ; 1995:393-416.
7. Rönnberg J. Perceptual compensation in the deaf and blind: Myth or reality? In: Dixon RA & Bäckman L, eds. Compensating for Psychological Deficits and Declines: Managing Losses and Promoting Gains. Mahwah, NJ, USA: Lawrence Erlbaum Associates;1995:251-274.
8. Bernstein LE, Tucker PE, Auer ET. Potential perceptual bases for successful use of a vibrotactile speech perception aid. Scand Jour Psychol. 1998;39:181-186.
9. Bernstein LE, Demorest ME, Coulter DC, O´Connell MP. Lipreading sentences with vibrotactile vocoders: Performance of normal-hearing and hearing-impaired subjects. J Acoust Soc Amer. 1991;95: 2971-2984.
10. Bernstein LE, Demorest ME, Tucker PE. Speech perception without hearing. Perception & Psychophysics. 2000;62: 233-252.
11. Tillberg I, Rönnberg J, Svärd I, Ahlner B. Audio-visual tests in a group of hearing-aid users: The effects of onset age, handicap age, and degree of hearing loss. Scand Audiol. 1996;25:267-272.
12. Rönnberg J., Söderfeldt B, Risberg J. The cognitive neuroscience of signed language. Acta Psychologica. 2000;105:237-254.
13. Rönnberg J, Andersson U, Lyxell B, Spens K. Vibrotactile speechreading support: Cognitive prerequisites for training. Jour Deaf Studies & Deaf Education. 1998;3:143-156.
14. Lyxell B, Rönnberg J. Visual speech processing: Word decoding and word discrimination related to sentence-based speechreading and hearing-impairment. Scand Jour Psychol. 1991;32:9-17.
15. Rönnberg J, Arlinger S., Lyxell B, Kinnefors C. Visual evoked potentials: Relation to adult speechreading and cognitive function. J Speech Hear Res. 1989;32:725-735.
16. Lyxell B, Rönnberg J. The relationship between verbal ability and sentence-based speechreading. Scand Audiol. 1992;21:67-72.
17. Lyxell B. Skilled speechreading – a single case study. Scand Jour Psychol. 1994; 35:212-219.
18. Rönnberg J. Cognitive characteristics of skilled tactiling: The case of GS. European Jour Cognitive Psychol. 1993;5:19-33.
19. Rönnberg J, Andersson J, Samuelsson S, Söderfeldt B, Lyxell B, Risberg J. A speechreading expert: The case of MM. J Speech, Lang Hear Res. 1999;42:5-20.
20. Rönnberg J, Samuelsson S, Lyxell B. Conceptual constraints in sentence-based lipreading in the hearing impaired. In: Campbell R, Dodd B, Burnham D, eds. Hearing by Eye: Part II. Advances in the Psychology of Speechreading and Audiovisual Speech. London: Lawrence Erlbaum Ass; 1998:143-153.
21. Lyxell B, Holmberg I. Visual speechreading and cognitive performance in hearing-impaired and normal hearing children (11-14 years). Brit Jour Educational Psychol. 2000;70, 505-518.
22. Rönnberg J, Samuelsson E, Borg E. Visual cognitive tests, central auditory function and auditory communication. Scand Audiol. 2000;29:196-206.
23. Pichora-Fuller K. Processing speed: Psychoacoustics, speech perception, and comprehension. Paper presented at: Örebro-Linköping workshop on Assessment of Auditory Communication: From Lab to Real Life. Scandinavian Audiology, 2001.
24. Andersson U. Cognitive Deafness. The Deterioration of Phonological Representations in Adults with an Acquired Severe Hearing Loss and its Implications for Speech Understanding [dissertation]. Studies from the Swedish Institute for Disability Research No. 3, 2001.
25. Rönnberg J, Andersson J, Andersson U, Johansson K, Lyxell B, Samuelsson S. Cognition as a bridge between signal and dialogue: Communication in the hearing impaired and deaf. Intl Jour Audiol. 1998; [suppl] 27 (49):101-108.
26. Samuelsson S, Rönnberg J. Implicit and explicit use of scripted constraints in lipreading. European Jour Cognitive Psychol. 1993;5, 201-233.
27. Andersson U, Lyxell B, Rönnberg J, Spens K-E. Cognitive predictors of visual speech understanding. J Deaf Studies and Deaf Educ. 2001; 6:103-115.
28. Andersson U, Lyxell B, Rönnberg J, Spens K-E. A follow-up study on the effects of speech tracking training on visual speechreading of sentences and words: Cognitive prerequisites and chronological age. J Deaf Studies and Deaf Educ. 2001; 6:116-129.
29. Lyxell B, Arlinger S, Andersson J, Bredberg G, Harder H, Rönnberg J. Phonological representation and speech understanding with cochlear implants in deafened adults. Scand Jour Psychol. 1998;39:175-179.
30. Lunner T. Cognitive functions in relation to hearing aid use. Paper presented at: the Örebro-Linköping workshop on Assessment of auditory communication: From Lab to Real Life. Intl Jour Audiol. 2001.