The Windows speech engine can be set to a specific recording device, or it can be set to whatever the user has set up as the default recording device. Either way, it's a single connection point - one is constant, the other can float.
In VA's settings, there's an option that lets you set Windows' default recording device on startup (audio tab) - this is the older option. This relies on the user leaving the speech engine set to 'default audio device' within its control panel settings for speech recognition to work. It's an awkward option for sure - it serves a dual purpose (it rode in with some other features - it's why it is on the audio tab). I can't remove it, as there are folks that use it for one reason or another.
Later on, the 'Windows Speech Recording Device' option was added to the recognition tab. This is a shortcut to changing the specific recording device to use for the speech engine across all applications. This relies on the selected device to always be present and enabled - if it's not, no speech recognition will occur. It changes a system setting that is not normally used (or even known about) - I hesitated to add it for the longest time.
If your setup has the Windows speech engine pointing to a specific device (either set up through VA or through control panel), and there is audio coming from other sources that is getting into that channel, you've got some type of bleed going on. This is going to need to be mitigated locally, unfortunately, as this indicates either a hardware/driver issue or some type of software routing configuration. As Pfeil had mentioned, turning off speech facilities will definitely eliminate the problem of speech recognition occurring during your VA session no matter what channels are used
Hope some of that helps clear up some confusion!