Yup, I use a desktop mic as well, and made a command for this issue.
What it does is, it saves you speaker volume settings, then drops the audio of your speaker to 1, as long as you hold the button, and when you let go, it restores the volume to where you had it. It's works great!
I use the F22 as my key, but you can change it.
I've attached the profile below.