Jasay: Towards voice commands in projectional editors

Abstract

Permanent disabilities or temporary injuries (e.g., RSI) hinder the activity of writing code. The interaction modality of voice is a viable substitute or complement for typing on a keyboard. This paper describes the design of Jasay, a prototype tool that enables developers to write Java code using voice commands. Our implementation relies on a third-party speech-recognition system to convert the voice into text. In turn, such a text is translated into commands that transform the abstract syntax tree (AST) of the code being edited. Jasay works as an extension to a projectional editor, taking advantage of having the abstract syntax tree always available without parsing, a permanent well-formed structure of the code, and unambiguous editing locations (e.g., class member, statement, expression, etc). An early experiment with Jasay involving 5 programmers has shown encouraging results, as they were able to perform small program modifications within reasonable time.