As with any speech recognition software, the Microsoft speech recognition system is designed to recognize speech, so that's what it'll do when you provide it with input.
This is comparable to pressing random keys on a keyboard, or clicking in random locations with a mouse, which would also lead to unintentional effects.
The way around that would be to suspend listening while you're not issuing commands, either using a physical control, if that is feasible for you (multiple options for that are available on the "Hotkeys" tab of the VoiceAttack options window), or spoken commands containing the "Make VoiceAttack Start Listening", "Make VoiceAttack Stop Listening", and/or "Make VoiceAttack Toggle Listening" actions.