Mobile recognition and tracking of objects in the environment using Augmented Reality and 3D audio signals for people with visual impairments
Oliver Beren Kaul, Kersten Behrens, Michael Rohs

CHI'21: ACM CHI Conference on Human Factors in Computing Systems
Session:
Late work day 3 – E
Late work day 3 – F

Abstract
People with visual impairment face challenges in scene and object recognition, especially in unfamiliar environments. We combined the mobile scene detection framework Apple ARKit with MobileNet-v2 and 3D spatial audio to provide auditory scene description to people with visual impairments. The combination of ARKit and MobileNet makes it possible to keep recognized objects in view, even if the user turns away from the object. For example, an object can serve as an auditory landmark. With a search function, the system can even direct the user to a specific item. The system also provides spatial audio alerts for nearby objects and walls to avoid collisions. We evaluated the implemented app in a preliminary user study. The results show that users can find items without visual feedback using the proposed application. The study also shows that the coverage of local object detection via MobileNet-v2 was insufficient, which we aim to overcome in future work using more accurate object detection frameworks (YOLOv5x).

DOI:: https://doi.org/10.1145/3411763.3451611
WEB:: https://chi2021.acm.org

Video previews of CHI 2021

Please take the opportunity to connect and share this video with your friends and family if you find it helpful.