The eye-ear relationship: investigating auditory impacts on subtitle reading and comprehension with eye-tracking

A study by Hussein Abu-Rayyash, Shatha Alhawamdeh, and Yuri Ringomon, titled The Eye-Ear Relationship: Investigating Auditory Impacts on Subtitle Reading and Comprehension (2024) explored how subtitles affect language learning and comprehension. The study used RealEye eye-tracking system to gather detailed data on participants’ eye movements as they engaged with subtitled content in different languages and auditory conditions.

How RealEye Was Used

RealEye was instrumental in capturing the interactions between participants and subtitles. The analysis considered measures such as average fixation duration, total number of fixations, saccade length, and percentage of skipped subtitles, to measure how viewers processed visual information under different conditions. These metrics revealed patterns like increased fixation time when audio was absent, showing the cognitive demands of relying solely on subtitles. The data also highlighted differences in how Spanish and Arabic subtitles were read, providing insights into the cognitive challenges associated with processing different languages.

Key Findings

The study revealed several notable insights about how audio and subtitle combinations impact learning:

1. The Role of Audio: Participants spent more time fixating on subtitles when there was no audio, indicating a greater reliance on visual input. When the audio matched the subtitle language, participants demonstrated more efficient processing, as the two modalities reinforced each other.
2. Language-Specific Differences: Spanish subtitles prompted higher fixation counts and longer reading times compared to Arabic, suggesting that language transparency and familiarity influenced cognitive effort. Spanish, being more transparent, may have been easier for participants to process visually than Arabic.
3. Audio-Free Viewing: Interestingly, participants achieved the highest comprehension scores when subtitles were presented without accompanying audio. This suggests that removing auditory distractions may allow learners to focus more fully on the visual content.

‍

This study highlights the complex relationship between audio, subtitles, and language learning. The use of RealEye allowed researchers to analyze these dynamics in detail, providing valuable insights into how learners process subtitled content. These findings could help improve the use of audiovisual media as a language learning tool and inform the design of more effective subtitling strategies.

The authors of the study contacted RealEye to share their excitement about incorporating the platform into their research. They emphasized that RealEye was instrumental in making their work possible, allowing them to investigate how subtitles are read and understood under different auditory conditions.

The authors expressed their appreciation for RealEye’s accessibility and its ability to support innovative research outside traditional lab environments.

You can run a similar study!

Follow the steps below to start your own experiment with RealEye:

1. Go to RealEye Dashboard and create or log in to your account.
2. Purchase the License of your choice (https://www.realeye.io/pricing). If you need any custom adjustments, contact us at contact@realeye.io. We are happy to help!
3. Activate your license by following the instructions in the RealEye License Activation Guide

‍

Ready to set up your own study? Visit RealEye Support page to learn more and keep us posted on your results! 🚀