There are no secrets to success. It is the result of preparation, hard work, and learning from failure.

- Colin Powell

The paper also received the Overall evaluation: 4 from each of the reviewers which is not an easy feat.

REVIEW 1

Overall evaluation: 4 (accept)

———- Overall evaluation ———-
This paper presents a study assessing the utility of the System Usability Scale for evaluating VUI. By asking participants to rate two VUIs via SUS, the results were consistent with previous studies comparing subjective rating scores and SUS, which implied that SUS is suitable for such evaluation.

While the SUS was developed 20 years ago, it is unsure whether it still works on evaluating the most novel interaction paradigms such as VUIs. To my knowledge, even though prior researchers have used SUS in evaluating VUI, there is no research efforts in investigating whether it is suitable and applicable to do so. Therefore, I think the conclusion of this paper could be inspirational for future work.

REVIEW 2

Overall evaluation: 4 (accept)

———- Overall evaluation ———-
I would argue for accepting this paper. This work examines the validity of the SUS through a 12-participant user study. Cross-validation results show that the SUS is appropriate for evaluating VUIs, offering both convergent and concurrent validity. The authors also indicate that researchers should create tasks that cause users to examine or explore the full extent of the VUI’s functionalities to warrant the sensitivity items 4 and 10 of the SUS to be sufficiently sensitive, which is an insightful sidenote for the practitioners to keep in mind. The results may be not surprising but did reassure the usability of this common tool can be transferred to the VUIs.

REVIEW 3

Overall evaluation: 4 (accept)

———- Overall evaluation ———-
This paper presented a small-scale within-subject lab study showcased that system usability scale is suitable for Voice-based user interface. The paper has a clear contribution with a simple but well-designed study. The reference is sufficient and the study is well articulated. I would argue for acceptance.

The authors believe that the paper is a good reference for anyone who wants to use SUS for evaluating VUI.