For a recent project, I needed the ability to accept input from users with movement disorders. These users wore a HoloLens device that instructed them to perform a simple walking task. When they finished the task, they needed to signal to the device to continue. Because many movement disorders affect manual dexterity, gesture input was not an option.
To accomplish this, I implemented phrase recognition with the HoloLens. The app will listen for audio cues like "start," "stop," and "help" to progress through the task. This feature allows users to interact with the application without using hand gestures - an action which a Parkinson's patient would find very difficult.
Below is an example implementation of phrase recognition using C#, Unity, and HoloLens.
Preparation
You will need to enable the Microphone capability for your app to use the microphone as an input device:
- In Unity, go to Edit -> Player Settings -> Player
- Click on the Windows Store tab
- Check Microphone in Publishing Settings -> Capabilities
Sample Code
Below is sample code I used as a base for the official implementation.
To give a quick explanation: this script is attached to the Main camera in Unity. Before the first frame update is called, I setup three words the user can say to interact with the app: start, stop, and help.
These words are passed into a KeywordRecognizer, which has a callback block named KeywordRecognizer_OnPhraseRecognized. I start this recognizer in the Start() function before the first frame is drawn.
When the user speaks one of these words, the KeywordRecognizer_OnPhraseRecognized function is called. It checks which keyword was spoken and invokes the action associated with it.
Potential Improvements
The official Microsoft help article on voice input suggests using a Cognitive Speech Services SDK Unity plugin that may have improved accuracy than the method above. In my testing, I found no reason to switch to this, as the method above seemed incredibly accurate.
We used a HoloLens 1 device but the HoloLens 2 is supposed to bring great improvements to the microphone and ambient noise cancelation. Alex Kipman, the inventor of HoloLens, talks about these improvements in great detail here, starting at 36:50.