What can eye tracking reveal about cognitive processes?

Every day, people navigate complex visual environments where their retinas are bombarded with an immense amount of visual stimuli. Nevertheless, people can select what stimuli to attend to and which ones to ignore in this highly dynamic process.

Humans achieve this selective perception by directing their gaze toward a specific region of the visual scene. Eye movements do not merely reveal the visual information that is being selectively harvested on a moment-to-moment basis; they are tightly coupled to cognitive processes such as memory, decision-making, and associative learning. Understanding what humans look at and how that influences cognition and behavior is one of the most fundamental questions in psychology and neuroscience, making eye tracking a highly suitable technology to address it.

In this learn article, we will present how eye tracking technology has been used to study cognitive processes and the insights that these studies have generated. We will discuss memory, decision-making, and problem-solving.

Why use eye tracking technology to study cognitive processes?

Eye tracking grants continuous access to cognitive processes

Cognitive processes are interconnected and multilayered, as each undergoes different operational stages over time. For example, consider the sequence of cognitive processes that unfold during decision-making: scene exploration, target detection, consideration, and decision about the response. If, in such a task, we count on an overt behavioral response (e.g., key press, written text, or vocalization), the contributing processes which led to the decision are not captured. Eye tracking data can provide the behavioral end products of the cognitive processes (e.g., decisions) and isolate the different layers of related processes that occur during an experiment.

Eye movements provide insights into brain function  

The brain circuits that support the generation of eye movements have been studied for the past few decades and have resulted in a profound understanding of the relationships between cognition, eye movements, and brain physiology (Hannula et al., 2010; Knudsen, 2018). The rich knowledge of oculomotor circuitry makes eye tracking an excellent method to probe a diverse range of cognitive processes in healthy as well as pathological brain states, throughout a person’s life. 

Eye tracking is compatible with other biometric measurements 

Eye movement measurements can be combined with brain activity measures and biosensors (e.g., electroencephalography (EEG), electrocardiography (ECG), intracranial EEG (iEEG), galvanic skin response (GSR)), allowing for profound insights into how eye movements, cortical activity, and other physiological variables interplay to contribute to various behaviors.  

Eye tracking allows cross-species comparison and promotes translational studies  

Eye tracking and similar or even identical behavioral tests can be performed on different species, allowing for inter-species comparison and uncovering causal relationships between behavior and brain physiology. To date, eye movements measurements have been performed on nonhuman primates (A. M. Ryan et al., 2019), dogs (Karl et al., 2020),  mice (van Beest et al., 2021), rats (Wallace et al., 2013), pigeons (Kano et al., 2018), zebrafish (Dehmelt et al., 2018).  

Memory

A few key factors guide eye movements in the visual environment: physical characteristics of a stimulus (e.g., color, and luminance), our internal states, and previous knowledge related to visual stimuli (e.g., episodic, or semantic memories). We look at objects that are the targets of our voluntary search, but we also dwell on objects that are somewhat novel or contradictory to our knowledge and expectations. Findings from eye tracking studies show that multiple distinct memory representations can be retrieved upon visual scene scanning and compete for preferential oculomotor guidance (Wynn, Ryan, and Moscovitch, 2020). Anatomical and functional links between the oculomotor and memory systems (e.g., the hippocampus, frontal eye fields, dorsolateral prefrontal cortex) (Shen et al., 2016) further support the significance of eye movements in the memory processes. As we will illustrate further, eye movements are functionally relevant for memory encoding and retrieval.

During the encoding phase, free viewing behavior predicts the quality of a subsequent memory (Bylinskii et al., 2015; Damiano and Walther, 2019). For example, incidental memory is better for an object which is viewed longer and with multiple fixations, compared to an object in the same scene which is viewed shorted and with less fixations (Bylinskii et al., 2015; Olejarczyk et al., 2014). Eye movements supply memory with visual input and organize visual inputs in time and space, acting as a memory-binding mechanism (Nikolaev et al., 2022; J. D. Ryan & Shen, 2020). Upon visual information sampling, a scan path sequence is stored together with that visual input to facilitate memory retrieval by comparing new input with stored memory traces (Johansson et al., 2022; Wynn et al., 2019).

During visual information retrieval, people tend to look at previously associated but empty locations, the so-called “Looking at nothing” effect. The fixation of the gaze toward nothing (i.e., blank spaces) reflects the shift from externally to internally oriented attention to retrieve the stored memory representations (Scholz et al., 2018). During “looking at nothing,” previously encoded gaze patterns of a visual stimulus are recapitulated again when retrieving stored memories, referred to as gaze reinstatement. The quality of the recalled memory is predicted by the degree of similarity of the gaze scan path during the memory encoding and retrieval — the more the encoding and retrieval scan paths overlap, the better the quality of the recalled memory (Johansson et al., 2022).  

Decision-making 

Eye movements provide a broad spectrum of insights into decision-making processes with a fine temporal resolution on how these processes unfold. Eye movement measurements can help indicate how long it takes to reach a decision, the influence of the expected reward on a decision, or self-confidence about the decision outcome (Spering, 2022). Some specific spatiotemporal characteristics of gaze behavior can inform the decision-making process during all distinct stages — before, during, and immediately after the decision.   

Before making a decision, eye movements facilitate sensory information accrual from the visual environment — the process that will define what information will be accessible or even dominate the working memory when the decision-making process initiates. The gaze behavior will reflect the order in which sensory information is collected and how decision-related evidence is weighted and assimilated with prior knowledge (Gottlieb and Oudeyer, 2018; Spering, 2022).  

Some specific eye movement indices predict the decision’s timing and accuracy ​during the decision-making process. Saccade metrics (e.g., peak velocity, amplitude, vigor, and end-point scatter) can yield valuable information about the timing of perceptual decisions (Spering, 2022). Eye fixation on a moving target — smooth pursuit — can indicate the decision-formation process and even predict its outcome. In baseball or go/no-go sensorimotor decision-making paradigms (Fooken and Spering, 2019), high smooth pursuit velocity correlates with fast decision-making. The decision end point can be inferred from a response-related suppression of saccades and microsaccades, so-called oculomotor freezing, which indicates the response preparation and yields a marker of a temporal expectation of the decision (Abeles et al., 2020). ​ 

When making a decision, eye movements can indicate the subjective feeling of confidence. For instance, saccade peak velocity reflects the degree of certainty with which a decision is made, and it has been shown to increase with accrued evidence (i.e., gaining more certainty) (Seideman et al., 2018). ​Moreover, saccade, pursuit metrics, blinks, and pupil dilation are all intimately linked to dopamine activity, thus, are implicated in reward processing (Hikosaka et al., 2014)​. For instance, an increase in the speed of saccadic eye movements reflects the expectation of reward, whereas anticipation of effort decreases it (Shadmehr et al., 2019). ​  

Problem-solving and creativity 

Problem-solving involves constructing mental models that depict relevant information and finding the most appropriate solution to a problem. It is a multilayered cognitive process requiring several cognitive functions working simultaneously, such as attention, memory, and creativity (i.e., divergent thinking).  

Eye movements can reveal active cognitive representations and how they are manipulated in the mind when involved in problem-solving. When solving problems or thinking creatively, people tend to shift their gaze away from a relevant stimulus and focus on a blank space. The time spent looking away from a stimulus increases with the difficulty of a problem, as attention tends to shift internally, and the looking-at-nothing phenomenon is observed (Ferreira et al., 2008). ​ When viewing blank spaces, eye-movement patterns mirror the mental images used to solve problems internally (Spivey and Geng, 2001). ​Even during straightforward reasoning activity, such as distinguishing between left and right or above and below, people move their eyes in the respective directions (Demarais and Cohen, 1998).  Note how the same looking-at-nothing phenomena and related eye movements can infer different cognitive processes (memory retrieval, problem-solving, or creative thinking) depending on the task at hand.  

Successful problem-solving can be identified solely from eye movements. Knoblich and colleagues (Knoblich et al., 2001) demonstrated that fixation on the relevant object for solving the problem increases over time, especially immediately before reaching the solution. The study demonstrated the unique utility of eye tracking during a problem-solving task, which previously counted on traditional performance measures, like solution time and rate measured by a key press or mouse tracking (Knoblich et al., 2001).

Conclusion

Eye movements are tightly coupled to cognitive processes, such as memory, decision-making, problem-solving. As illustrated in this learn article, eye tracking technology can contribute to understanding what humans look at and how that influences their cognition and behavior.

If you are interested in discovering more on how eye movement measurements can yield insights to cognitive functions, and what are the commonly used paradigms to study those functions, check out our white paper
“Eye tracking – a window to cognitive processes”.

Cited publications

Abeles, D., Amit, R., Tal-Perry, N., Carrasco, M., & Yuval-Greenberg, S. (2020). Oculomotor inhibition precedes temporally expected auditory targets. Nature Communications, 11(1), Article 1.

Bekkering, H., Adam, J. J., Kingma, H., Huson, A., & Whiting, H. T. A. (1994). Reaction time latencies of eye and hand movements in single- and dual-task conditions. Experimental Brain Research, 97(3), 471–476.

Bylinskii, Z., Isola, P., Bainbridge, C., Torralba, A., & Oliva, A. (2015). Intrinsic and extrinsic effects on image memorability. Vision Research, 116, 165–178.

Damiano, C., & Walther, D. B. (2019). Distinct roles of eye movements during memory encoding and retrieval. Cognition, 184, 119–129.

Dehmelt, F. A., von Daranyi, A., Leyden, C., & Arrenberg, A. B. (2018). Evoking and tracking zebrafish eye movement in multiple larvae with ZebEyeTrack. Nature Protocols, 13(7), Article 7.

Demarais, A. M., & Cohen, B. H. (1998). Evidence for image-scanning eye movements during transitive inference. Biological Psychology, 49, 229–247.

Ferreira, F., Apel, J., & Henderson, J. M. (2008).
Taking a new look at looking at nothing. Trends in Cognitive Sciences, 12(11), 405–410.

Fooken, J., & Spering, M. (2019). Decoding go/no-go decisions from eye movements. Journal of Vision, 19(2), 5.

Gellman, R. S., Carl, J. R., & Miles, F. A. (1990). Short latency ocular-following responses in man. Visual Neuroscience, 5(2), 107–122.

Gottlieb, J., & Oudeyer, P.-Y. (2018). Towards a neuroscience of active sampling and curiosity. Nature Reviews Neuroscience, 19(12), Article 12.

Hannula, D., Althoff, R., Warren, D., Riggs, L., Cohen, N., & Ryan, J. (2010). Worth a Glance: Using Eye Movements to Investigate the Cognitive Neuroscience of Memory. Frontiers in Human Neuroscience, 4.

Hikosaka, O., Kim, H. F., Yasuda, M., & Yamamoto, S. (2014). Basal ganglia circuits for reward value-guided behavior. Annual Review of Neuroscience, 37, 289–306.

Johansson, R., Nyström, M., Dewhurst, R., & Johansson, M. (2022). Eye-movement replay supports episodic remembering. Proceedings of the Royal Society B: Biological Sciences, 289(1977), 20220964.

Kano, F., Walker, J., Sasaki, T., & Biro, D. (2018). Head-mounted sensors reveal visual attention of free-flying homing pigeons. Journal of Experimental Biology, 221(17), jeb183475.

Karl, S., Boch, M., Zamansky, A., van der Linden, D., Wagner, I. C., Völter, C. J., Lamm, C., & Huber, L. (2020). Exploring the dog–human relationship by combining fMRI, eye-tracking and behavioural measures. Scientific Reports, 10(1), Article 1.

Knoblich, G., Ohlsson, S., & Raney, G. E. (2001). An eye movement study of insight problem solving. Memory & Cognition, 29, 1000–1009.

Knudsen, E. I. (2018). Neural Circuits That Mediate Selective Attention: A Comparative Perspective. Trends in Neurosciences, 41(11), 789–805.

Nikolaev, A. R., Bramão, I., Johansson, R., & Johansson, M. (2022). Episodic memory formation in naturalistic viewing [Preprint]. Neuroscience.

Olejarczyk, J. H., Luke, S. G., & Henderson, J. M. (2014). Incidental memory for parts of scenes from eye movements. Visual Cognition, 22(7), 975–995.

Ryan, A. M., Freeman, S. M., Murai, T., Lau, A. R., Palumbo, M. C., Hogrefe, C. E., Bales, K. L., & Bauman, M. D. (2019). Non-invasive Eye Tracking Methods for New World and Old World Monkeys. Frontiers in Behavioral Neuroscience, 13.

Ryan, J. D., & Shen, K. (2020). The eyes are a window into memory. Current Opinion in Behavioral Sciences, 32, 1–6.

Scholz, A., Klichowicz, A., & Krems, J. F. (2018). Covert shifts of attention can account for the functional role of “eye movements to nothing.” Memory & Cognition, 46(2), 230–243.

Seideman, J. A., Stanford, T. R., & Salinas, E. (2018). Saccade metrics reflect decision-making dynamics during urgent choices. Nature Communications, 9, 2907.

Shadmehr, R., Reppert, T. R., Summerside, E. M., Yoon, T., & Ahmed, A. A. (2019). Movement Vigor as a Reflection of Subjective Economic Utility. Trends in Neurosciences, 42(5), 323–336.

Shen, K., Bezgin, G., Selvam, R., McIntosh, A. R., & Ryan, J. D. (2016). An Anatomical Interface between Memory and Oculomotor Systems. Journal of Cognitive Neuroscience, 28(11), 1772–1783.

Spering, M. (2022). Eye Movements as a Window into Decision-Making. Annual Review of Vision Science.

Spivey, M. J., & Geng, J. J. (2001). Oculomotor mechanisms activated by imagery and memory: Eye movements to absent objects. Psychological Research, 65(4), 235–241.

van Beest, E. H., Mukherjee, S., Kirchberger, L., Schnabel, U. H., van der Togt, C., Teeuwen, R. R. M., Barsegyan, A., Meyer, A. F., Poort, J., Roelfsema, P. R., & Self, M. W. (2021). Mouse visual cortex contains a region of enhanced spatial resolution. Nature Communications, 12(1), Article 1.

Wallace, D. J., Greenberg, D. S., Sawinski, J., Rulla, S., Notaro, G., & Kerr, J. N. D. (2013). Rats maintain an overhead binocular field at the expense of constant fusion. Nature, 498(7452), Article 7452.

Wynn, J. S., Ryan, J. D., & Moscovitch, M. (2020). Effects of prior knowledge on active vision and memory in younger and older adults. Journal of Experimental Psychology: General, 149, 518–529.

Wynn, J. S., Shen, K., & Ryan, J. D. (2019). Eye Movements Actively Reinstate Spatiotemporal Mnemonic Content. Vision, 3(2), Article 2.