Research | Hearing Review Online-May 2020
Six research studies for creating outcome measures that are more ecologically valid when assessing hearing aid performance
By Sophia E. Kramer, PhD, Tanveer Bhuiyan, PhD, Lars Bramsløw, PhD, Lorenz Fiedler, PhD, Carina Graversen, PhD, Lauren V. Hadley, PhD, Hamish Innes-Brown, PhD, Graham Naylor, PhD, Michael Richter, PhD, Gabrielle H. Saunders, PhD, Niek J. Versfeld, PhD, Dorothea Wendt, PhD, William M. Whitmer, PhD, and Adriana A. Zekveld, PhD
In today’s aging population, hearing impairment is an increasing concern for public health and social participation, affecting one in three older adults. In the long term, problems in understanding speech result in a cascade of cognitive and psycho-social declines. The constant challenge to effectively communicate can result in mental fatigue, poorer mental health and increased sick leave from work. This indicates a great disease burden.
It is crucial to use outcome measures that accurately reflect the challenges experienced in real life and to provide rehabilitation techniques that adequately address these problems.1 Current laboratory tests lack ecological validity2 in that many conduct testing at unrealistically low signal-to-noise ratios (SNR),3 use materials that do not reflect daily communication,4 and lack spatial separation of signals,5 again, not reflective of real-world communication. Other factors involved in real-life communication settings that are not generally considered in speech recognition tests are an individual’s motivation, the social setting, characteristics of the acoustical environment (eg, reverberation and visual information), and dynamics like head movement.
Also, recent research has identified new avenues that hold great promise for the identification of additional aspects sensitive to the listening experience in daily life. For example, there is a large body of literature showing that listening effort may still vary between conditions where speech performance is optimized.6 This is particularly the case in situations with high (positive) SNRs which are most common in everyday life.3 Physiological measures indexing listening effort [eg, pupillometry, electroencephalography (EEG), and cardiovascular measures] can be used to provide complementary information to speech perception scores.
The H2020 EU Marie-Curie European Industrial Doctorate project “Innovative HEaring Aid Research–Ecological Conditions and Outcome Measures” (HEAR-ECO) addresses the issues mentioned above by developing and combining new tools and outcome measures for realistic communication. HEAR-ECO includes translation of these tools into innovative developments and evaluations of new technology for those with hearing loss.
HEAR-ECO covers the training of six early-stage researchers. They are each enrolled in a doctoral program and jointly supervised by the academic and non-academic institutions participating in HEAR-ECO. The academic institutions in HEAR-ECO each hosting 3 students are the VU University medical center in Amsterdam (The Netherlands) and the University of Nottingham (United Kingdom). Associated academic partners providing training opportunities and co-supervision are the Liverpool John Moores University and the University of Manchester, both in the United Kingdom. The non-academic (industrial) partner completing the project is Eriksholm Research Centre (part of Oticon) in Denmark.
Given that HEAR-ECO is an interdisciplinary innovation project, the six researchers recruited for this project comprise a heterogeneous group in terms of knowledge and skills they bring to the project. They come from around the world:
- Patrycja Książek is an acoustics and computer sciences engineer, originally from Poland;
- Bethany Plain is a neuroscientist and clinical scientist in audiology from Southampton (UK);
- Hidde Pielage is a cognitive psychologist who completed his masters in Amsterdam;
- Defne Alfandari is a cognitive neuropsychologist originally from Turkey;
- Tirdad Seifi Ala is a biomedical engineer from Tehran, Iran, and
- Sergio Luiz Aguirre is an acoustics engineer from Florianopolis, Brazil.
The students spend time at their academic institution (VU University medical center in Amsterdam or University of Nottingham) and a minimum of 18 months at Eriksholm Research Centre (part of Oticon) in Denmark. In HEAR-ECO, they are working at the nexus of technology, (social) psychology, physiology and audiology. Whereas they each have their own focus, their common goal is to work together towards ecologically valid outcome measures of hearing.
HEAR-ECO currently celebrates its halfway point. Groundwork studies by each of the six researchers are now completed and the results hold great promise for the next phase of the project in which the tools will be combined. In the paragraphs below, we report on each of the six sub-studies and the most salient results achieved so far.
Temporal Changes in the Pupil Signal
Patrycja Książek is determining the best way to extract valuable information about the multiple processes involved in speech recognition from pupillometry time series data. Commonly used pupillometry parameters are the task-evoked maximum peak pupil dilation (PPD) and mean pupil dilation (MPD) relative to a baseline pupil size. Both parameters have been shown sensitive to changes in speech intelligibility levels and SNRs.7,8 However, conventional analyses like these may discard potentially valuable information on cognitive processes underlying listening. Hence, a more holistic analysis of temporal changes in the pupil response could provide information complementary to PPD or MPD.9
In her first study, Książek investigated two analysis approaches: Principal Component Analysis (PCA) and Index of Pupillary Activity (IPA). Both these approaches consider the whole signal. While PCA operates directly on the time series, IPA is a spectro-temporal measure. The two approaches were applied to the pupil size recorded while participants listened to sentences in noise. Parameters from both techniques were tested for their sensitivity to changes in task demand (ie, SNR) and luminance. The latter was done to investigate to what extent the measures derived are sensitive to light. The estimated parameters appeared to be differentially sensitive to task demand and luminance. Importantly, different components obtained with PCA appear to disentangle multiple, sequential cognitive processes involved in speech understanding in noise.
In a follow-up study, Książek will investigate the factors that influence these sequential components.
Myocardial Sympathetic Responses during Speech Perception Are Indicative of Changes in Listening Demand
Bethany Plain investigates listening effort using cardiovascular measures. Effort investment and engagement in physical tasks are associated with increased activity of the sympathetic branch of the autonomic nervous system. In her first study, Plain examined whether increased sympathetic activity would also characterize effort invested in understanding speech in difficult relative to easy listening conditions. Using a speech-in-noise (SIN) task, she demonstrated that variations in SNR reliably elicited changes in sympathetic impact on the heart: assessed using changes in pre-ejection period (PEP), the time interval between the excitation of the left heart ventricle and the beginning of the ejection of blood into the aorta.
Moreover, Plain’s findings also suggest that collecting physiological responses only during speech perception (rather than also during speech repetition) is sufficient to detect these effort-related changes in sympathetic activity. Her research provides a proof-of-principle of the application of pre-ejection period measurement in speech perception tests.
In her follow-up work, Plain will refine the study design in order to gain even more insight into the sympathetic response to effortful listening.
Our Social Context Affects Performance and Listening Effort
Hidde Pielage seeks to understand how our social context affects speech perception performance and listening effort. He hypothesizes that people invest more effort in listening when they perceive there is social pressure to do so, because of the perceived social consequences of understanding what is going on around them (or not).
From an experimental point of view, this led Pielage to devise an experiment to test whether or not people used more listening effort and/or whether as a consequence they performed better, when conducting a SIN test in the presence of another individual relative to when they did the task alone. He measured performance with a SIN test and simultaneously measured listening effort using pupillometry. Specifically, he hypothesized that effort would increase when the testing was done in the presence of another individual relative to when testing was done alone, leading to improved performance results when another individual was present. The results partly confirmed these hypotheses in that pupillometry showed increased effort when the task was completed in the presence of another person. However, performance did not change between conditions. Because the additional effort in the presence of another person did not lead to enhanced performance, he is now wondering whether the pupillometry findings were, in fact, related to the stress of having another person present rather than listening effort. In fact, early findings of a follow-up study indicate that, in some cases, performance becomes poorer when another person is present.
This leads Pielage to hypothesize that self-efficacy might be a mediator of his data. He is currently planning a new experiment to test this.
How Motivation and Fatigue Influence Listening Effort
The effort a person exerts to accomplish a listening task may not only depend on the difficulty of the listening situation, but also on the person’s current motivation and the amount of resources which have already been used up to accomplish earlier tasks (ie, fatigue). To shed light on these dependencies, Defne Alfandari investigates how motivation and fatigue affect pupil dilation while test participants listen to and repeat sentences in background noise.
She investigates pupil dilation in two essential time windows. First, the baseline pupil size is estimated shortly before a sentence is presented. This serves as a proxy for the listener’s level of arousal in anticipation of the upcoming task. Second, the mean pupil dilation (relative to the baseline) during the presentation of the sentence is measured. This serves as a proxy for how much effort the listener allocates to the processing of the current sentence. To manipulate the test participants’ level of motivation in her first study, Alfandari used variable amounts of monetary reward for reaching a certain level of performance (ie, correctly repeated sentences). To manipulate the level of fatigue, test participants did the same listening task both before and after a relatively long period of repeatedly doing another sentence recognition task. In addition, the task-demand was manipulated by varying the SNR of the sentences to be repeated. Alfandari’s hypothesis was that fatigue should limit the cognitive resources that a listener can deliberately allocate, such that after the fatiguing middle block of trials, changing the monetary reward for high performance should have less of an effect on the mean pupil dilation.
While the data of the first study are currently under analysis, in a follow-up study, Alfandari will address fatigue more particularly by manipulating the strength of fatigue, such that possible confounds such as training effects can be ruled out.
Tapping into the Listening Demands of an Active Mind
When researching listening effort and the allocation of cognitive resources, one clear place to look for evidence is the epicenter of cognition: the brain itself. Measures of neural signals, such as EEG, allow us to measure activity in regions of the brain while people do different tasks. But these methods also pose a challenge in disentangling all the things our brains are engaged in at any given moment.
Tirdad Seifi Ala is looking at how much can be gleaned from EEG about listening. His analyses of synchronous EEG and pupillometry recordings shows that they both vary as a function of demand (eg, SNR), hence both can provide a window into aspects of listening effort. Inspired by the motivation strand of the HEAR-ECO project (above), Tirdad’s work also shows that changes in the alpha band of the EEG signal vary according to the promised reward in a sentence recognition task. When adding realistic environments to a listening task involving different SNRs in collaboration with Sergio (see below), not only do changes in neural activity reflect the expected relationship between objective effort and demand (ie, little effort when the task is too hard or too easy), but also neural differences relating to the environment (eg, reverberation).
Seifi Ala’s work so far indicates the intriguing possibility that EEG may allow us to parse out different deficits in speech understanding due to different demands. His research therefore has importance not only for neuroscience analysis, but also raises questions on what aspects of speech-recognition testing are related to our cognitive abilities10 and how to improve these tests in the future.
Making Room for One More: Developing Realistic Environments for Real-world Communication
To reflect real-world communication, tests of hearing ability and compensation strategies should reflect the real-world. There have been many advances in recent years towards realistic presentations. One limitation in these realistic presentations, however, is that they require a lone listener—a hearing-aid wearer, for example, sitting in the center of a loudspeaker array.
Prompted by the questions of social context and motivation within HEAR-ECO, Sergio Luiz Aguirre went about testing a novel question: What happens to the environment if there’s someone else there? Results show that the simulation is not nearly as affected by the addition of another as the position of the listener. The position of the listener in these state-of-art simulations can affect not only the realism of the environment, but listener’s performance. Aguirre has been investigating how subtle movements could dramatically impact spatialized speech recognition. The leeway in where a listener can be is known to be related to the sophistication of the playback system; this can limit the use of these simulations in labs and clinics where two-dozen calibrated loudspeakers and an appropriately sized chamber are simply not possible.
Aguirre has also been developing novel hybrid forms of simulation that would allow realistic presentation with limited facilities. The goal is to provide realistic presentation techniques that would allow movement of the listener, as well as the presence of a communication partner, without requiring a state-of-art acoustics lab to test out the latest hearing technology with environmental validity.
Where to from Here?
In the first phase of the HEAR-ECO project, each student completed groundwork studies in one of the distinct methodologies relevant to the overall aims of the project. In the second phase, all students were gathered at Eriksholm Research Centre to combine their knowledge, experiences, and results of their groundwork studies. By combining their knowledge, they have been able to design, pilot, and set up three “combination studies” which are currently nearing completion. In the final phase of HEAR-ECO, follow-up studies (some by individual students, some by combined teams) will be carried out at the three main partner sites, maximizing the exploitation of the knowledge gained so far, and going deeper into some of the questions raised by the previous work.
Although hearing-aid technology has advanced spectacularly in recent years, methods used to develop and evaluate them are still not based on conditions in which they are actually used. Nor do they tap into essential aspects of real-world hearing performance. Thus, the potential of improved hearing aid technology to meet the needs of individual users in their daily lives is acutely under-exploited.
By combining factors involved in real-life communication, like motivation, the presence of others, acoustical characteristics and dynamics like (head) movement into speech perception test settings, this project contributes to the development of validated and innovative tools. We envision that these will enable the development and evaluation of hearing rehabilitation solutions that provide most listening effort benefit next to the best listening performances.
The science in HEAR-ECO will help to facilitate the paradigm shift needed to bridge the gap between real-world performance and lab-based testing. The hope is that the findings of HEAR-ECO will have both research and clinical applications.
You can learn more about this work at our website (www.hear-eco.eu) or by following the project on Facebook and Twitter (@ecohear).
Acknowledgements
This project has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie-Sklodowska-Curie grant agreement No 765329.
References
- Pichora-Fuller MK, Kramer SE, Eckert MA, et al. Hearing impairment and cognitive energy: The framework for understanding effortful listening (FUEL). Ear Hear. 2016;37(Jul-Aug)[Suppl 1]:5S-27S. doi:10.1097/AUD.0000000000000312
- Keidser G. Introduction to Special Issue: Towards ecologically valid protocols for the assessment of hearing and hearing devices. 2016. J Am Acad Audiol. 27(7):502-3. doi: 10.3766/jaaa.27.7.1.
- Smeds K, Wolters F, Rung M. Estimation of signal-to-noise ratios in realistic sound scenarios. 2015. J Am Acad Audiol. 2015;26(2):183-196. https://doi.org/10.3766/jaaa.26.2.7
- Bronkhorst A. The cocktail party phenomenon: A review of research on speech intelligibility in multiple-talker conditions. Acustica. 2000;86:117-128.
- Beechey T, Buchholz JM, Keidser, G. Eliciting naturalistic conversations: A method for assessing communication ability, subjective experience, and the impacts of noise and hearing impairment. 2019. J Sp Lang Hear Res. 2019;26;62(2):470-484. doi: 10.1044/2018_JSLHR-H-18-0107.
- Zekveld AA, Koelewijn T, Kramer SE. (2018). The pupil dilation response to auditory stimuli: Current state of knowledge. Trends Hear. 2018; doi:10.1177/2331216518777174
- Wendt D, Koelewijn T, Książek P, Kramer SE,Lunner T. Toward a more comprehensive understanding of the impact of masker type and signal-to-noise ratio on the pupillary response while performing a speech-in-noise test. Hear Res. 2018;369, 67–78. doi:10.1016/j.heares.2018.05.006
- Zekveld, A. A., Kramer, S. E., & Festen, J. M. Pupil response as an indication of effortful listening: The influence of sentence intelligibility. 2010. Ear Hear. doi:10.1097/AUD.0b013e3181d4f251
- Winn, M. B., Wendt, D., Koelewijn, T., & Kuchinsky, S. E. Best practices and advice for using pupillometry to measure listening effort: An introduction for those who want to get started. Trends Hear. 2018; 22[Jan-Dec]. doi:10.1177/2331216518800869
- Besser, J., Koelewijn, T., Zekveld, A. A., Kramer, S. E., & Festen, J. M. How linguistic closure and verbal working memory relate to speech recognition in noise—A review. Trends Amplif. 2016;17(2), 75–93. https://doi.org/10.1177/1084713813495459.
CORRESPONDENCE to Dr Kramer at: [email protected]
Citation for this article: Kramer SE, Bhuiyan T, Bramsløw L, Fiedler L, Graversen C, Hadley KVm Innes-Brown H, Naylor G, Richter M, Saunders GH, Versfeld NJ, Wendt D, Whitmer WM, Zekveld AA. Innovative HEaring Aid Research – Ecological Conditions and Outcome measures: the HEAR-ECO project. Hearing Review. 2020; Online first: May 8, 2020.
Image: © Raphael Daniaud | Dreamstime.com
