Hi, kermitt
You're going to want to look at the option, 'Reject Pending Speech', located on the Recognition tab of the Options screen (try checking the box to see if it helps).
When that option is not checked, VA will continue to listen until you finish saying what you started saying when listening was turned on. This is generally a good thing for most as users tend to let the hotkey go shortly before finishing what they are saying (this would result in nothing being executed, resulting in a flipped desk/table). If you've got a noisy environment, VA may still think something is going on, so, the option to turn this off is there.
If doing that turns out to not be a great fit, the other option for you is to pause a second after letting the hotkey go to let recognition finish up.
Hope that helps!