Eye Tracking for Efficient Annotation and Enhanced Neural Network Training
The annotation of image data is often a time-consuming and expert-intensive task, particularly when working with new domains in the medical field. As a step toward making this process more efficient, we explore the use of eye tracking technology as an additional input for neural networks.
The core idea is to record expert gaze during routine observation, allowing us to passively collect valuable implicit annotations without requiring additional effort from the expert. This data can then be integrated into machine learning models to support segmentation and classification tasks. While still an early-stage approach, eye tracking-based annotations have the potential to help adapt networks to new domains more efficiently and reduce the need for extensive manual labeling.
By incorporating gaze data, we aim to develop a more intuitive and scalable way to generate training data, which could contribute to improving the performance of deep learning models.
Some of our related publications are listed below.
Related Publications:
-
Kockwelp, J., Beckmann, D., & Risse, B. (2025). Human Gaze Improves Vision Transformers by Token Masking. in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops
- Beckmann, D., Kockwelp, J., Gromoll, J., Kiefer, F., & Risse, B. (2024). SAM meets Gaze: Passive Eye Tracking for Prompt-based Instance Segmentation. Proceedings of Machine Learning Research, 2023.
- Kockwelp, J., Gromoll, J., Wistuba, J., & Risse, B. (2023). EyeGuide - From Gaze Data to Instance Segmentation. The British Machine Vision Conference (BMVC), Aberdeen.