dictation while interacting with game console?

[Tazling]

I am really appreciating VA as a hands-free way of using console commands in VR games.  When you're wearing a box on your head, there are only a paltry few buttons on your controllers all of which are pre-assigned, and you're yards away from any keyboard, voice input is just what the doctor ordered.

In Skyrim VR, I get access to console cheats and shortcuts (and compensate for broken menus) using single words or 2-word phrases which trigger VA to invoke the console and issue commands.

So for example, "right spear" might trigger the console command "player.equipitem 0004A67F right" (I'm somewhat making this up, not sitting at my gaming platform right now).  On hearing "right spear", VA would issue the tilde to bring up the console, enter the command, then <Enter>, then tilde to dismiss the console.  Neat and tidy, I get my trusty magic ice spear in right hand without having to navigate any menus.

However, there are times when I want to poke around in the game internals using the console, so I'd like to be able to dictate free-form input to it.  I might want to issue a command like "help 'Night Eye' <enter>" today, and tomorrow it might be "help 'Vampire Amulet'" or "prid 000457AB" or whatever.  It gets tedious setting up a VA command every time I want to do some once-off console inquiry or try something out.  OTOH it can also be very tedious and frustrating trying to use the WinDoze virtual popup keyboard with Quest 2 controllers :-)

So I'd love to be able (somehow) to invoke the console and speak or spell words and numbers into it ad lib.  So I found this discussion...
https://forum.voiceattack.com/smf/index.php?topic=1825.0
which seems to be a good starting place for turning dictation mode on and off.  Is this the direction I should take?

Is there a mode in which I can spell words into VA (without having to set up 26 letters and 10 digits as voice commands!)?  I did read https://forum.voiceattack.com/smf/index.php?topic=142.0 but I don't know the alpha-bravo code well enough to use it really fluently (and ohboy is it slow for spelling longer words).  That post is from 2016, has any advance been made since then on spelling (like putting VA into a special mode where it knows that e.g. "c" is "c" and not "see" or "sea")?

Or would I be better off to create VA commands for the fairly short list of console lexemes I plan to use, like "player", "help", "equipitem", "prid" and so on?  Also "dot", "right paren", "left paren", "single quote", "double quote", and whatever other syntax elements are required?  And then only ad lib strings of numbers as needed?

I'd like to know in which direction to focus my effort for the best result, rather than bang my head on an insoluble problem, trying to make VA do something that it was never meant to do :-)

[Pfeil]

The speech recognition engine does best with predefined commands. While you can in theory use the dictation mode to spell words, the chances of letters getting recognized as short words, as you point out, are not insignificant.
You may also find that "fantasy" terms like "night eye" or "vampire amulet" are not easily recognized without explicitly being added to the dictionary.


For spelling words, and recognizing console-specific terms, rather than creating separate commands which could get recognized when you don't intend them to be, as long as you require fewer than 500 phrase variations to be recognized at any one time, the "Wait For Spoken Response" action can be used, combined with a loop, some conditions, and the "Quick Input" action.

This could still be combined with regular commands for more commonly-used input, E.G. speaking "item search" to open the console, insert "help '", wait for you to speak the characters forming the term you want to search for, and then insert "'" and press enter when you speak "search", or something similar.

Do note that any single-character approach will require a short pause between each character, due to the way the speech recognition engine attempts to wait for you to finish speaking.

[SemlerPDX]

...

I'd like to know in which direction to focus my effort for the best result, rather than bang my head on an insoluble problem, trying to make VA do something that it was never meant to do :-)

I'm in the same boat, friend - at a loss for simple input while in VR and not wanting to fumble for the keyboard (or take hand controllers off to type).  As a side project to my larger voice control profiles, I've been developing a character level input system using the NATO phonetic alphabet, VoiceAttack wildcard commands, and a phrase parsing system similar to the one I use in my AVCS Voice Calculator & Conversions profile.

This is still a Work in Progress Alpha, but works well enough that I've been using it now and then since I wrote it, and shared it with a few friends for example/review.  Since it uses wildcards and can fire off at undesired times, an 'on/off' command is included to stop these commands from executing even if recognized when NATO Typing is "off"

Since single word commands can be rather hit and miss, for direct input of a single number or to begin a string of numbers, phrases are preceded by the word "Numbers", or for special letters input, "Letters".  A letter can be further preceded by the word "Capital" to capitalize the first letter in the string of letters spoken, and therefore shift symbols over numbers can be accessed by saying "Capital" first, such as "Capital 4" for "$ ".  Most common symbols can be pressed by direct speech, "Press <X>", including Enter and Backspace, and of course most of this will be expanded and refined once I take this profile into a proper public beta test and beyond.

Feel free to modify it for your own personal use as needed, or to learn how to make such things yourself.  The heavy lifting is performed by an Inline Function in VB.net, any questions, just ask.


AVCS NATO Phonetic Typing (0.8a):  ( Click Here )

Code: [Select]

===Example Usage===
say, "Enable Typing", to allow recognized commands to process
say, "Disable Typing", to deny recognized commands from processing

say, "Alpha Papa Papa Lima Echo Apostrophe Sierra"
VA types out, "apple's"

say, "Press Enter"
VA presses 'enter' key

say, "Press Backspace"
VA presses 'backspace' key

say, "Go Back 4"
VA presses 'backspace' key 4 times

say, "Space"
VA presses 'spacebar'

say, "Numbers 42489"
VA types '42489'

say, "Letters Z H"
VA types, 'zh'

say, "Press CAPSLOCK", then say, "Letters H I P Dash Ten Seven Sixteen"
VA types, 'HIP-10716'

say, "Letters jerry goes to the store"
*VA types, 'jerrygoestothestore'
*(it's an alpha WIP)
((this is a VR side project and I've not been playing VR lately; likely ETA to beta before end of Summer, full version by end of year))

A simple version check occurs once per session on use of the command "Enable Typing", looks at a decimal on a blank .htm page on my website, currently v0.8 - will show note in the VoiceAttack event log if/when I release an updated profile download.  This inline is noted in this command, and can be disabled if undesired without affecting the rest of the profile/commands.

NATO Phonetic Alphabet reference:
https://upload.wikimedia.org/wikipedia/commons/e/e0/FAA_Phonetic_and_Morse_Chart2.svg

[Tazling]

Thanks for the useful responses and your patience with the relative n00b.

I grabbed the original NATO vap, installed it into a profile, and have been trying to wrap my head around it.  The code seems awfully redundant, i.e. the whole giant if-else block is replicated for every possible input.  I was thinking of adding some more tokens like "dot" and "quote" but it seemed like a hideous job making those changes to umpteen copies of the if-else block. 

But possibly I am not understanding something important, like the individual instances of that code block under command "alfa" and command "bravo" are actually just ghosts of a single master copy?  I have not previously used any of VA's internal coding options, so this is new territory.  I looked at the XML briefly but it was brain-numbing so I gave up for the night :-)

Should I assume that the new AVCS NATO vap is a modification of the original one, or is it a whole independent effort?  I will DL and try it out later today.

I get the "dictation on, dictation off" logic, that is intuitive.  But...

Is there any reason why one could not define a set of command words "eh bee sea dee ee eff gee aitch I" and so on, rather than the NATO radio codes?  So that when dictation is On, "gee" results in "g" being output?  Isn't that easier than remembering "golf"?

Still trying to figure out the shape of the problem and very grateful that others have trodden the path before me!

[Pfeil]

It's worth pointing out that when you're looking at or downloading examples made with old versions of VoiceAttack, certain features that allow more concise commands may not have been available at the time.

That said, unless you're looking at a different profile (I went with the one attached to this post), there is exactly one command containing a long set of conditions, to which you could easily add by simply adding/duplicating an "Add Else If to Conditional Block" action, and a keypress/"Quick Input" action, in addition to adding your word/phrase to the second dynamic section in the "When I say" field.

Modernizing that profile could include integrating everything into a single command by using a loop and the "Wait For Spoken Response" action, as well as numeric ranges for the numeric characters, and the "{CMDSEGMENT:}" token combined with the "{TXTSUBSTR:}" token to get the first character of a spoken word, to handle all single characters in a single conditional branch.


Have you made any attempt to use the "Wait For Spoken Response" action? As with regular commands, it allows you to define words or phrases, or even single characters.

[SemlerPDX]

Thanks for the useful responses and your patience with the relative n00b.

I grabbed the original NATO vap ...

Should I assume that the new AVCS NATO vap is a modification of the original one, or is it a whole independent effort?

That NATO-Profile.vap is not related to my own project there, had not been trying to improve on it, rather to make my own entry in my familiar style (didn't know about it, or if I did, had forgotten for years).  I focus heavily on handling homophones and working around the limitations of voice recognition such as single letter or single syllable words as commands ("gee" instead of "g", for example).  I like to make things work in any way I can think to say it when able, and that requires some extra checks and sometimes redundancy to help make sure it works 100% of the time.

Like Pfeil wrote, there are a number of ways to overcome these limitations, including 'Wait for spoken response' actions and more, my profile is just another of many ways of doing this sort of stuff with some currently available features, which is why I included it for reference or use.  Together with the other, the two methods show the range of possibilities with VoiceAttack.

[Tazling]

BTW @SemlerPDX I forgot to mention that your examples are quite impressive.

I will have to try it out! 

Being a more recent effort than the NATO.vap I found (yes, in the thread that Pfeil linked), it probably contains useful examples of recent programming features.

More later.  I am enduring a WinDoze automated upgrade (ptui!) and my Win10 box is temporarily unusable.  When it has recovered I will be playing with VoiceAttack again.

[UPDATE] Having survived the upgrade (though for some reason it nuked my Virtual Desktop Streamer app), I have added SemlerPDX's project to my VoiceAttack profile and tried it out.  It's pretty exciting!  I was able to interact almost ad lib with the console. 

One caveat:  personally I would prefer it if letters and numbers were not separate modes, because I'm entering hex codes!  So I have to say things like

"number zero zero zero letter A number six letter B number three letter C"

which is a bit of a mouthful.  looking at the code (which admittedly I still don't understand) it seems on the surface to my ignorant eye as if numbers/letters could be one mode, "spell it", because no digit is phonetically ambiguous with any letter.

so if I had my druthers, I would say,

"spell:  zero zero zero A six B three C"

other than that, it's dandy.  I can use backspace to correct errors of interpretation, for example delete a few chars and switch to NATO codes if the VA engine has difficulty with my alphabet letters.

I'm a little puzzled still about how it turns on and off.  it seems the user can say "stop", or maybe it times out?  Because a couple of times I said "letters blah blah blah [a short string of letters]" then paused to think, and when I started again it wasn't listening any more.  I had to say "letters" again to keep dictating.  that actually seems fine to me.

I find I can easily speak too fast for it, in which case it can skip a letter in the sequence.

but on the whole it is just what the doctor ordered and really solves my problem!  Many thanks SemlerPDX!

[SemlerPDX]

One caveat:  personally I would prefer it if letters and numbers were not separate modes, because I'm entering hex codes!  So I have to say things like

"number zero zero zero letter A number six letter B number three letter C"

You could just say,
"number zero zero zero A six B three C"

...The numbers/letters thing in this beta test (WIP) profile is just a means to initiate command recognition on single letter/syllable words.

which is a bit of a mouthful.  looking at the code (which admittedly I still don't understand) it seems on the surface to my ignorant eye as if numbers/letters could be one mode, "spell it", because no digit is phonetically ambiguous with any letter.

so if I had my druthers, I would say,

"spell:  zero zero zero A six B three C"

definitely would be just fine.  In fact, this WIP typing profile has the word "type" holding that exact function, since this is what we are doing (since if I asked my computer to 'spell' something, I might expect it to speak back the letters of a word)

I'm a little puzzled still about how it turns on and off.  it seems the user can say "stop", or maybe it times out?  Because a couple of times I said "letters blah blah blah [a short string of letters]" then paused to think, and when I started again it wasn't listening any more.  I had to say "letters" again to keep dictating.  that actually seems fine to me.

I find I can easily speak too fast for it, in which case it can skip a letter in the sequence.

It thinks you stopped talking when you paused so it began to process the speech so far - this can be improved later, but also requires users to be aware of that factor when speaking.

These are common known issues in this test profile and it certainly needs refinement, so don't feel like it is anything you are doing wrong - any voice command must start with a word that is on the 'When I say' list as a starting word. Long strings of single letter or number commands tend to be less accurate as well.  Testing your command above, I found it catching "A" as "8" far too often, or not at all - but I had continued success (barking like Lt. Data from Star Trek) with,
"type zero zero zero Alpha 6 Bravo 3 Charlie"

But it was even easier to say it in two parts -- "type zero zero zero" and then "Alpha 6 Bravo 3 Charlie"

I think no matter what, even when this profile is as developed and refined as it can be, there will still be a degree of choice on the part of the user on how to initiate an input phrase, and I can't see how that will work just yet.  It is a known limitation of VoiceAttack and voice recognition in general that single syllable works or characters can be rather hit or miss, which is why use of the NATO phonetic alphabet can help - but using numbers and adding optional letters (proper) as well, things get complex fast.

but on the whole it is just what the doctor ordered and really solves my problem!  Many thanks SemlerPDX!

Glad it's working well for what it is and can do for now!  I use it myself now and then, but also find it's quirks a bit bothersome.  Here's looking forward to a better version one day - got my focus tied up in a robotics project at the moment, and it's far too much fun to pull away right now.