Assessing the Utility of the System Usability Scale for Evaluating Voice-based User Interfaces

2018.04 | ChineseCHI 2018


Voice-based User Interfaces (VUIs) challenge our existing conceptions of usability since the standardized evaluation tools we use were typically developed for interfaces with visual feedback, whereas VUI’s have predominantly eyes-free interactions. We experimented with the use of a well-validated tool, the System Usability Scale (SUS), to evaluate two existing, commercially available VUIs. We administered the SUS to 12 participants after they completed a set of scenario tasks on Amazon’s Alexa and Apple’s Siri. The results were consistent with previous studies comparing subjective rating scores and SUS, suggesting that the SUS is a valid evaluation tool for VUIs. Additionally, despite large, significant differences in adjective scale ratings and SUS scores, both systems performed similarly on the learnability items. We conclude with recommendations on the use of the SUS for evaluating VUIs.


voice-based user interface, evaluation, conversational user interface, voice user interface


Proceedings of the Sixth International Symposium of Chinese CHI

Project Info



Debjyoti Ghosh, Pin Sym Foong, Shan Zhang, and Shengdong Zhao