Touchscreen menu innovation faster, simpler

In a world where people learn how to use iPads before they learn to walk, finding simpler ways to interact with touch devices is a must.   

University of Saskatchewan PhD computer science student Mohammad Sami Uddin has tackled the problem by developing a new multi-touch menu called HandMark-Finger. The interface, which is designed for large touchscreens such as tablets, makes it easier and about 30 per cent faster to input commands.

“This research has the potential to make everyday interactions with touch devices simpler,” said Carl Gutwin, computer science professor and Uddin’s supervisor.

“People are beginners for the first few hours they use their device, but are experts for the rest of the time,” he said. “We are looking at ways to make touchscreen menus fast for experts.”

The researchers have found that by speeding up human interaction with devices and by fitting in more commands, HandMark-Finger may have real-life applications in devices such as in tablets, computer desktops, and smart TVs. Uddin has both published his results and presented them at national and international conferences.

To use HandMark-Finger, users have to touch the screen palms down. Different sets of commands then pop up around the fingers. Like “digital pianists,” they use all the fingers to interact with the screen, instead of the one- or two-finger tapping and swiping done on devices now.

Unlike with familiar interfaces, the HandMark-Finger comes with around 40 commands. Since the menu is built around users’ hands, it helps them remember the location of commands, with fingers serving as memory cues.

“I got the idea from the way people use their fingers to play the piano,” said Uddin. “People have intimate knowledge of their hands, so we used them as a memory device by associating commands to fingers.”

Uddin has installed the HandMark Menu on a large touchscreen device and tested how quickly and accurately people could find and select commands on the screen. His results from 13 users show that the time needed for learning how to use the menu is similar to current devices, as are the error rates.

Uddin has also developed a second 160-command interface he calls HandMark-Multi. In this version, sets of commands appear between the thumb and index finger, depending on how many fingers touch the screen. Unlike with HandMark-Finger, users make more mistakes and are slower to learn the menu. 

“We played with different HandMark versions to see their performance and whether they could fit a large number of commands,” said Uddin. “People in our study preferred HandMark-Finger over regular menus and over HandMark-Multi because it provided clear reference points for the location of commands. 

Uddin and Gutwin agree that more research and re-design is needed before HandMark Menus reach commercial devices. The researchers will now focus on how best to group commands on the screen, and perhaps conduct a large-scale study.

The project is funded by the federal agency NSERC and is part of the U of S Interaction Lab, which has been achieving success in advancing computer interfaces, computer games, and systems used to visualize information.

This article first ran as part of the 2018 Young Innovators series, an initiative of the U of S Research Profile and Impact office in partnership with the Saskatoon StarPhoenix.

Federica Giannelli is a graduate student intern in the U of S research profile and impact unit.