Lookout - Assisted Vision

Features

1. Sort mail and read text

Problem

• Teach users who are blind how to scan a document. This involves framing the document and keeping the device steady long enough for the camera to find a steady frame.

Solutions

• Experimented with instructions that guided users to use different strategies to move the camera above a document using various patterns in order to capture all text on a page.
• Prototyped different ideas to accommodate for technology limitations.
• Explored solutions where humans and interfaces could meet half way by leveraging and balancing their strengths.

Experimented with instructions and strategies combining physical patterns like using a figure eight and providing auditory cues as edges were detected.

2. Identify packaged and canned foods

Experts that teach living skills at the Light House for the Blind in San Francisco explained to us the importance of labeling of packaged foods. This job is very time-consuming for people who are blind. The ability to confirm the flavor or contents of cans of soup or soda from the pantry or fridge, especially when not at home, could significantly provide confidence for individuals to do this task without asking others for help.

During the study, I learned more about strategies participants who are blind use to learn about and navigate the environment, how they manipulated their phone, and the role the screenreader and the app played to support the completion of those tasks.

Videos from the users point of view helped understand what went wrong when users tried to find out what snack they picked from their purse. Shared examples like these where the image the camera captured was blurred helped the team understand what was going on and allowed us to look for technical, interaction and physical solutions.

Problem

• People who are blind find it difficult to point the camera to frame an object without auditory support. Imagine taking a selfie without a live feed.
• The app needed users to manipulate the object until it could locate the barcode.

Jyotsna using Lookout to read the label of packaged food in the kitchen.

Screenshot of video feed of person aiming at a bag of snacks appears blurry and out of the camera making it hard for Lookout to read text.

Solutions

• Instructions and auditory feedback to help users aim.
• Recommendations to explore reading barcodes live.

3. Describe items of importance

Learn Places: Describe what the camera sees in a way that is useful and relevant for a person to understand a new place.

Problem

• The app also verbalizes things in the environment.
• People hear a continuous stream of objects recognized without hierarchy or learn of irrelevant objects. Knowing what is relevant or important to users in the moment.

Solutions

• Developed a hierarchy framework based on distance and placement to prioritize speaking objects in surfaces within reach first.
• Explored a conversational experience.
• Collaborated with soft goods designer to design lanyards to hold devices for a passive auditory interface featured at the booth at Google I/O.

4. Hands-free and gesture controls

Lookout’s Multimodal Controls Overview.

• Interface with Physical Auditory and Digital components.
• Designed hands-free interactions to support users in accomplishing daily tasks that involve detecting text, objects, US currency, and barcodes.
• I experimented with novel interaction design patterns leveraging the device hardware sensors to assist users in aiming the camera.
• Physical interaction pattern for wearable experience: knock the device to start, cover the camera to stop the app for a hands-free experience.
• Explored conversational experiences to help people understand places.
• Reconciled novel gestures with familiar patterns used in Talkback (Android’s screenreader).

C. Gestural interface

Designed gestures that would be consistent regardless of the input method used to control Lookout. These include visual gestures on the touchscreen, screenreader gestures, the fingerprint sensor in the back of Pixel devices, and external Bluetooth controllers.

Used gestural UI patterns that responded to hardware input. Swiping up on the screen will tell more about an object. The physical up gesture functionality in the back of the phone was consistent.

D. Interoperability of digital and physical controls

Interoperability exercise to understand the mapping of actions across inputs like the phone’s proximity sensor and other external hardware.

Validation results

Hypothesis: Hands-free operation will increase users confidence.

• Lookout’s form factor is also not ideal for the app’s most exciting use cases. Despite receiving a lanyard, few testers used it regularly; they felt more comfortable holding their phones. Users preferred to manipulate the phone with their hands, which provided them with a greater sense of control.
• Users needed to intervene a lot and could not rely on the app verbalizing what is important or relevant only. Lookout at the time was more useful when descriptions of objects happened on demand versus users placing objects in front of the camera worn.
• After competitive analysis, it is recommended to focus on identifying objects, reading text, and operating home appliances.
• Recommendation to create a way for users to have feedback easily repeated.