Designing and prototyping a talking AI that helps the elderly take action when at risk of a fall.
At the end of my UX semester in Amsterdam I had a final project to complete. I was given a client to work with for 6 weeks. There is a single client with one complex project, the assignments are individual. Everyone in the class is focusing on their own design challenge within the single project. This year the client we teamed up with was Digital Life Laboratory. They developed a device based on the Kinect technology that could analyse the body movements of an elderly person and carry out necessary action to prevent the fall. They named it Bravo.
In the Netherlands, every 5 minutes an elderly person is admitted to the hospital due to a fall accident. Fortunately, there are ways to reduce someone's risk of falling down. Fall prevention training or a medical examination are just two of them. However, the elderly living on their own often underestimate the risk of falling.Project DefinitionThe overall goal of the project is to prevent the elderly from falling. Divided into smaller parts my design challenge was:
"How to persuade elderly to take action before they fall. Convey the message when elderly are in risk of falling.”
Before this project started in May 2017, we’d done preliminary research in February of the same year. There were 28 people divided into teams working on different parts of the research. We had a team working on elderly empathy. Other teams worked on the actual target user research (elderly living alone, with family), and then also teams researching the other stakeholders involved (family, friends, neighbours). There was a team for professional stakeholders research (home caretakers, general practitioners) as well. The outputs were a variety of personas, scenarios, experience maps, cultural probes and even an empathy suit. Once worn, the person wearing it is able to actually feel how an elderly person feels in terms of physical abilities. Having a really deep understanding of the field through such extensive research was crucial for creating the best solution for the problem.
To keep this study brief I will point out the most relevant findings from the research:
Interest in new technologies. After interviewing a few users from our target group, it was clear that they are more open to use new technologies than one might initially think. The real problem is that the technology is often not designed inclusive enough for elderly making the onboarding and also the actual use really challenging for them.
Since I have zero experience in healthcare, I needed to do health research in terms of what is actually possible for me to implement in my design. To my surprise, not much. There is a bunch of exercises to train the body and keep it active, but everything has to be done in the presence of a professional. Most of the solutions might result in making the whole experience an even bigger risk if not done right. After reading studies of the World Health Organization, I came to the conclusion that the best way to prevent falls would be for the user to just sit down and relax. When necessary, a professional would be called in.
I initially started exploring some ideas, reading articles online, talking with the elderly and sketching out first ideas of what the experience could be like. After a few feedback meetings, I found out the best channel to connect with the target group would be - voice.
Knowing the routines of my target group, I came to realise that in terms of showing feedback and visual data, the best device is an actual TV. I combined the two ideas and sketched out the prototype of the product.
I decided to create a multimodal experience using a voice interface followed by a TV interface. This created a big challenge for prototyping and usability testing. Both of these interfaces required research into voice and TV user interfaces.
Sketch of a use case
I was focused on one specific scenario: when the Bravo device detects that the probability of falling is rising, it tries to persuade the user to sit down and relax. Initially, this is done by starting a conversation. The goal is not to tell the user about the danger directly, but rather keep them in a positive mood. Not scaring them off. If the device deems the health risk out of the ordinary, it also notifies the caretaker. To ensure that the user follows the relax routine, the TV interface encourages them by displaying feedback and various types of notifications.
To get an exact overview of the user’s interaction with the device and what is needed to be done, I created a brief scenario with exact experience touch points.
The goal is to have the device handle most of the cognitive processing, making the experience as straightforward as possible for the user.
During the voice interface design process, I designed for two different aspects of the product. Voice input and audio output. The challenge was to create an interface that doesn't merely have good copy, but also maintains a good conversation. It is crucial to make the conversation feel as natural as possible by creating a good flow between the inputs and outputs. When the device leads a good conversation, its persuasion capabilities increase thanks to being perceived as more human. During the process I also made the distinction between the parts of the experience that should be conveyed visually and the ones that ought to be handled by via audio.
Since the target group is very comfortable with interactions with a TV, it was the obvious choice when determining the feedback visualization channel. Since a TV is a device mainly for entertainment and relax, it perfectly matches the project’s objectives. Designing for a TV is different than designing for smaller devices. Usually, a user is watching it from distance, in a vastly different context. The most important part was the typography. In order to maintain good readability for the elderly user from distance, I decided to go with a wider font weight and large type size. Darkening the live TV content behind the message allowed me to create sufficient contrast and make the user shift their focus on a different element on the screen.
TV UI sketches
Creating different variations of the notification led me to conduct a small validation session on a TV screen. The goal was to maintain good readability from standard couch distance, while covering as little screen space as possible. For displaying feedback on how much time until the relax session is completed, I wanted to use a circular bar. However, since the TV UI it is all about covering as little screen space as possible, I decided to go with a straight progress bar that covers about half the space. Keeping the user motivated to finish the task is the primary goal of the UI. This is done by displaying the progress and varied motivational messages.
Edit after assessment: As TV programmes in the Netherlands often contain subtitles, the displayed message would not be clearly readable due to the collision with subtitles. This situation unfortunately occurs even if the background is dimmed.
Few initial variations of the TV UI.
Arriving at a clear idea about the whole experience and completing the first version of a few visual elements for the TV UI, I went straight into prototyping. I spent some time thinking about what would be the best way to showcase this solution and to test it with real users. The journey from an automatically triggered voice interface to displaying data during live TV in one user flow proved to be a rather challenging prototyping case. I decided that the best tool for this particular project would be Framer. Few hundreds lines of code into the process, I managed to create a basic prototype which could recognise human voice, speak (with a British accent :)), turn on simulation of a TV and display notifications combined with different dynamic visual elements. This enabled me to test the solution with real users without having to sacrifice some parts of the experience. I also prepared the prototype in a way that allowed me to change what the device is saying with a single command. This feature enabled iteration over different conversation types during the testing.
This gif shows how fast it is to add a reply to the system during testing.
The first session took place with a moderator and 2 potential users from the target group. At the beginning, I wanted to have as little influence on the users as possible. I briefly told them about the device and what it is actually doing, leaving out the details. The first testing was a profound learning experience for me, as it allowed me to identify multiple previously unseen pain points.
I made a short video illustrating one use case.
I am intentionally writing about the details of the interface and the microinteractions only after describing the testing session. The reason being that the usability testing had a great influence on these factors. The interactions are designed around a single objective - supporting the user throughout the experience. In order to make the interactions as smooth as possible, the system has to display whether the device or the user is talking at any point in time. This was achieved by creating a pulsing dot animation and changing its colour to blue (device speaking) and green (user speaking). To incentivize the user to go through with the entire experience, feedback is shown at the edge of the screen, displaying the period of time left until the end of the recommended relaxation.
Since the very beginning, this project quickly shifted towards a new territory of voice-driven interfaces, TV UI and a new approach to human-computer interaction. Together with an interesting target audience, solid research and a unique challenge in prototyping/testing solutions, this has become a very exciting project to work on. I would not consider this by any means a final product. Rather, this is a start of an exploratory journey into the world of new possibilities.
By helping prevent older people from falling, this problem is definitely solved for the long term. What’s more, I often felt like I am making their health even worse by making them sit and watch the TV, the original intent being making them relax and wait for the caretaker. After digging more into health research I found out that there are a few easy exercises that the users could do while watching the TV. As a result, their health could improve in the long term. I definitely see potential in making the interaction flow longer and implementing an exercise session. In a broader sense, there are many other areas and scenarios in which a human-computer voice-based interaction could help the elderly around the house.
Design for Voice Interfaces by Laura Klein Copyright © 2016 O’Reilly Media, Inc.