Accessibility is an undeniably important subject, yet one which is all-too often overlooked in the development of new technologies. Despite the march of progress, the majority of legally blind individuals rely on a simple white cane for navigation — something researchers Manuel Zahn and Armaghan Ahmad Khan are looking to change.

The pair, working in the Center for Digital Technology and Management at the Technical University of Munich and in partnership with FRAMOS GmbH, have developed a system which combines a depth-sensing 3D camera with a wearable haptic feedback sleeve — providing, they say, navigation and obstacle avoidance capabilities entirely hands-free, and at a fidelity and range which far exceeds a cane.

Computer vision for the blind

The fully-wearable system proposed by Zahn and Khan is built around an Intel RealSense D415 depth-sensing camera, housed in a 3D-printed pair of glasses worn over the user’s eyes. This is linked to a single-board computer, an AAEON Up Board, which handles processing the data from the camera, and a battery pack for portability.

The RealSense Camera is designed around an active infrared stereo imaging system, projecting beams of invisible infrared light and measuring the depth of its surroundings. Compared with a system based on visible light cameras, it has one major advantage: It can operate in absolute darkness, allowing its user to navigate without difficulty.

An off-the-shelf depth-sensing camera combined with a 3D printed frame acts as the input to a hands-free navigation system for the blind.

Determining the depth of the wearer’s surroundings is only part of the system’s function, however: That information has to be communicated to the user in some way. Eschewing spoken or otherwise audible feedback, as likely to take away from the user’s ability to easily hear their surroundings, the researchers developed a wearable haptic feedback sleeve made from 25 vibration motors sewn onto fabric and linked to an Adafruit Feather 32U4 microcontroller development board.

With this, the system is complete: A depth-sensing camera which can map the wearer’s surroundings onto the 25-“pixel” haptic feedback system, providing a physical response when obstacles are near.

Novel navigation

Exactly what feedback is provided depends on where the user is navigating, the pair explain. For indoor navigation, the system provides vibration of nearby obstacles and walls — the location of the vibration mapped to the direction of the obstacle and the intensity mapped to its proximity. This way, a user can navigate with only the wearable system.

An outdoor mode switches from a standalone navigation device to an accessory, designed for use with the traditional cane: The obstacle detection range is extended to beyond two meters (around six and a half feet), providing information about obstacles and other objects outside the range of a traditional cane.

Designed to keep the user's hands free, this haptic feedback sleeve proved easy to use.

To prove the concept, the pair set up a navigation route which included walkways, doors, and various obstacles — and which had to be traversed in absolute darkness. Impressively, after a period of familiarization with the vibration feedback system, the five volunteers were able to complete the obstacle course perfectly on their very first try.

In subsequent attempts, the time taken to traverse the course reduced — showing increasing familiarity with and trust in the system: “Overall the averages for the runs decreased from 320 seconds in the first run to 148 seconds in the third run, two weeks later,” the researchers note — while admitting that one participant did become disoriented for a short period in their last run with the system.

The results encourage, with Zahn and Khan pointing to the system’s low cost and lightweight nature as making it of interest for future development — as too is the fact it does not interfere with the wearer’s hearing and leaves both hands free, either for using a traditional navigation aid like cane or for any other task.

Varying strength and positional haptic feedback turn obstacles into a feeling on the wearer's arm.

“Further developments could include the implementation of an object recognition, based on the RGB [visible light] image of the camera,” the pair add. “This would allow the system to artificially generate bordered paths, solving the problem of orientation in a large open space.”

The only sticking point: Intel, the manufacturer of the camera platform used for the prototype, recently confirmed it was to discontinue its RealSense platform — though, at the time of writing, it continued to provide the RealSense D415 for sale.

Zahn and Khan's paper is available on Cornell's arXiv.org preprint server.

References

Manuel Zahn, Armaghan Ahmad Khan: Obstacle avoidance for blind people using a 3D camera and a haptic feedback sleeve. DOI arXiv:2201.04453 [cs.HC].

Dylan Martin, CRN: Why Pat Gelsinger Decided to Wind Down Intel's RealSense Unit: Exclusive. https://www.crn.com/news/components-peripherals/why-pat-gelsinger-decided-to-wind-down-intel-s-realsense-unit-exclusive