Prepar3d with FlyInside: VA stops working when FI mirror window is open

[aurel42]

Hi, I'm using VA 1.5.12 on Win10 to control Prepar3d (one of the current incarnations of Microsoft Flight Simulator) while I'm flying using an Oculus Rift and FlyInside, a tool that adds VR support to FSX/P3D. FlyInside lets me toggle a "mirror window" on the normal desktop, I use that to stream what I'm seeing inside the Rift. And this mirror window is surprisingly causing me some grief with VA.

As long as it is closed, everything works perfectly fine. Whenever the mirror window is open, VA can't seem to see the main Prepar3D window any longer, even though it's still on the desktop where it was before, and still has the same window title.

I posted this on the FlyInside forum:

Whenever the mirror window is active, VoiceAttack can't send commands to P3D.

When I set VoiceAttack to send input to the "active window", it probably sends it to the mirror window where it gets lost.

When I set VoiceAttack to send input to the window with the title "Lockheed Martin® Prepar3D® v3" (which is only possible while the mirror window isn't open, or the title won't even be in the menu to pick from) I get this VoiceAttack error message whenever I try to execute a voice command and the mirror window is open: "Error acquiring process handler for command 'such and such' (Global Level)".

So I guess the main P3D window is somehow hidden from Windows or other applications (or whatever specific method VoiceAttack uses to identify windows) while the mirror window is open?

Dan, the developer of FlyInside, replied:

That's very strange... the mirror window doesn't try to hide Prepar3D itself at all. It may be worth writing to the Voice Attack authors in this case and seeing if they have any idea

Does anybody have an idea what could be going on here, or how to find out what's going on, and possibly how to work around the problem? Is there perhaps something special I need to do for applications that open multiple windows?

[Gary]

Do you have VoiceAttack set to send commands to the, 'active window'?

If so, you may want to set VoiceAttack to send commands to the Prepar3D (at the profile level... that would be in the profile options screen).

[aurel42]

Thank you for your advice. As I mentioned, either setting doesn't work and the window title of Prepar3D isn't even in the "Send commands to:" dropdown when the list is refreshed while the problematic "mirror window" is open. This affects both the dropdown in the main window as well as the option in "Edit Profile / Options and Overrides for this profile".

[Gary]

If you look in Task Manager, do you see the Prepar3D process?  You can also use the process name in the dropdown box (just type it in).  Also, you can use wildcards (*prepar3d*).  I'm not sure what that other app is doing, but it's certainly doing something.

[aurel42]

Thanks again for your advice, it's very much appreciated.

If you look in Task Manager, do you see the Prepar3D process?

Yes. I also see the main P3D window on the desktop.

When I open the mirror window, the line in the task manager changes from "Prepar3D exe (32 bit)" to "Prepar3D exe (32 bit) (2)", see attached screenshot. (Or not. Trying to attach a screenshot resulted in the forum software complaining about missing write permissions on the attachment folder. The picture would have shown the task manager listing the two window titles as "Lockheed Martin® Prepar3D® v3" and "FlyInside Mirror View".)

You can also use the process name in the dropdown box (just type it in).  Also, you can use wildcards (*prepar3d*).  I'm not sure what that other app is doing, but it's certainly doing something.

Using "Prepar3D*" works as long as the Mirror windows doesn't have the focus (because that window doesn't accept inputs).

Using "Lockheed*" seems to work better (I didn't try that before because I assumed it would behave exactly the same way as the full window title, which isn't the case). Sometimes inputs seem to get lost because of focus issues: to test this, I opened a Notepad window, clicked it into focus, used a voice command, in most cases the command was sent to the correct window, in some cases, I got the keypress in Notepad instead and after that the P3D window popped into the foreground. Not sure how this can happen when "Send commands to:" and the profile options are both set to "Lockheed*".

But, most importantly, I'm not stuck any more with VA not working at all, this workaround might work well enough when the P3D windows stays in focus.

Still, I agree, something weird is going on.

Thank you for holding my hand!

[aurel42]

Well, this seemed to work when I was testing it, but now when I tried to use it for a stream, I'm back to this:

https://drive.google.com/open?id=0B7Z-8ZNJtXTPRFhmTTN1allVUmM

"Error acquiring process handle..."

I'm out of ideas.

[Gary]

Error acquiring process handle...

That error is presented when you specify a target window/process and then that window/process can no longer be located.  What that means is that whatever app you are pointed at is closed or (I think) if the window title changes.  VA no longer has a target to send input to and then alerts the user.

[aurel42]

…and that error message would be totally appreciated, if there was any truth to it.

That error is presented when you specify a target window/process and then that window/process can no longer be located.  What that means is that whatever app you are pointed at is closed or (I think) if the window title changes.  VA no longer has a target to send input to and then alerts the user.

P3D is still running as before, it just opened an additional window.

The title of the window I want to send input to is unchanged.

I can still send input to the window in other ways, only VoiceAttack complains.

When I close the additional window, VoiceAttack immediately locates the "un-locatable" window again.

The developer responsible for opening that additional window claims there is nothing they are doing to hide the original P3D window in any way, shape or form.

Since nothing else is affected, I'm left with the conclusion that this is a VoiceAttack bug.

[Gary]

Try running VoiceAttack with the -ignorechildwindows command line option.  Let me know if that helps at all.

[iceblast]

Sorry, I have no idea how you have this setup. I'll just throw something out there.

You could rename the primary window when it opens, and have it do a check if the window exists when opening the second, and rename it to something else as well. So, both windows have very unique names.

If VA is confused, it might just be ignoring the windows. No idea, just guessing at a cause.

But if you change the window titles, then VA will at least have a chance to target unique named windows.

Wildcards that use the same names, will cause issues with VA.

Begin Text Compare : [{ACTIVEWINDOWTITLE}] Equals 'Lockheed Martin® Prepar3D® v3'
    Change window 'Lockheed Martin® Prepar3D® v3' title to, 'Prepar3D'
Else If Text Compare : [{ACTIVEWINDOWTITLE}] Equals 'Lockheed Martin® Prepar3D® v3'
    Change window 'Lockheed Martin® Prepar3D® v3' title to, 'Mirror3D'
End Condition

You can either have this run each time you run the command, or loop it.

Another way, is to target (2)

Begin Text Compare : [{ACTIVEWINDOWPROCESSNAME}] Contains '(2)'
End Condition

Of course, that window would have to be the active window.

The problem mainly is with Wildcards, if the match is using almost the same name. You have to have VA match 2 things, so it sees them clearly as different, otherwise, you'll have issues.

I had the same problem, targeting a window, within same program.

Let me know if this helps, or at least, gives you ideas to workaround your problem.

[Gary]

I would definitely try the -ignorechildwindows option...  I had to make this post since my first one was so tiny it may get missed   What I'm thinking may be happening is that the application is throwing up some type of tool window that may be stealing focus.  Just a theory.

[aurel42]

You could rename the primary window when it opens, and have it do a check if the window exists when opening the second, and rename it to something else as well. So, both windows have very unique names.

I appreciate your advice, but I don't think it applies to my case.

Both windows already have very unique names. There's zero chance VA is just confusing window titles. They are called:

"Lockheed Martin® Prepar3D® v3" and
"FlyInside Mirror View"

Begin Text Compare : [{ACTIVEWINDOWTITLE}] Equals 'Lockheed Martin® Prepar3D® v3'
    Change window 'Lockheed Martin® Prepar3D® v3' title to, 'Prepar3D'
Else If Text Compare : [{ACTIVEWINDOWTITLE}] Equals 'Lockheed Martin® Prepar3D® v3'
    Change window 'Lockheed Martin® Prepar3D® v3' title to, 'Mirror3D'
End Condition

...the problem being that VA can't actually "see" the window called "Lockheed Martin® Prepar3D® v3" -- would it still be able to rename it?

Let me emphasize this again:

- I close the "mirror" window, I open the "Send commands to" dropdown in the main VA window, I select "Refresh this list", the title of the "main" window ("Lockheed Martin® Prepar3D® v3") is in the list.
- I open the "mirror" window, I open the "Send commands to" dropdown in the main VA window, I select "Refresh this list", the title of the "main" window ("Lockheed Martin® Prepar3D® v3") is not in the list.

I can switch between these two states -- main window title in the list and main window title missing from the list -- at will by opening and closing the "mirror" and using "Refresh this list" window.

[aurel42]

I would definitely try the -ignorechildwindows option...

Thanks for making sure. This is definitely what I'm trying next, as soon as I can.

[aurel42]

Gary, I've changed the "Target" in my VA shortcut to

Code: [Select]

"C:\Program Files (x86)\VoiceAttack\VoiceAttack.exe" -ignorechildwindows
That's a valid way to do this, right? Sadly, that didn't help.

I didn't mention this before (because I didn't notice, since I always put the "mirror" window on a separate display): the "mirror" window is "always on top". It doesn't grab window focus from the "main" window, though.

[iceblast]

Sorry, I could have swore I read you said that the title didn't change, my mistake.

Gary's last suggestion might do the trick for you, but I'll throw a couple of thoughts out just in case.

Does the other option I gave you help any.

Or

Unless you don't want the window to be Active, while using it. Does this work?

Instead of sending commands directly to the vanishing window, could you send commands to it, if it's actually the Active window. So, VA isn't actually looking for the window, it's just going to enter commands blindly into whatever the Active window is?

If the Mirror program has a Windows tab, and you make it appear in the same tab every time, you could just use the Windows key, and the number of the tab. Like Win+6 or whatever. That makes it Active, and if VA will work without the direction to the Active window, then that could be your workaround.

Of course, if VA still doesn't send commands, even blindly, then I don't know.

[aurel42]

Sorry, I could have swore I read you said that the title didn't change, my mistake.

No, you're right, I said that the title doesn't change.

The title of the main window that is always present while Prepar3D is running, is always the same and never changes from "Lockheed Martin® Prepar3D® v3".

The title of the "mirror" window that I open and close depending on whether I need it, is different from the title of the main window, and it's always the same and never changes from "FlyInside Mirror View".

Of course, if VA still doesn't send commands, even blindly, then I don't know.

Sending commands to the "Active window" sometimes works and sometimes doesn't and I couldn't figure out yet why or why not. When it doesn't work, there is no error message, it just silently fails. But I would really prefer to send input to a specific window, because when I'm wearing the VR headset and recording the "mirror" window, it's hard to make sure that the P3D main window keeps the focus.

I understand that you're trying to help me find a way to identify and address the window I want. I don't think I have a problem identifying or addressing the window I want. I know where it is, I can see it right there on my desktop. I know what it's called. AutoHotKey's "Window Spy" tool can see it just fine, with the "mirror" window open or closed.

I think I have the problem that VA ignores the window I want when the application opens a certain other window under certain circumstances, and only then. And, by now, I'm not even sure what those conditions are, because during my tests today it suddenly seemed to work flawlessly, and when I tried to reproduce the same situation moments later, it didn't work any longer. I hate Heisenbugs.

I'll keep experimenting.

[iceblast]

Sorry to hear that. Because I don't have the problem in front of me, it's like throwing darts in the dark.

Since you mentioned AutoHotKey, Can you use a script, that will send the commands directly to the Mirror, using VA. Just have VA trigger the script. AHK has helped me with workarounds till I was able to figure out how to get VA to do it by it's self, but till then, you have it working.

Anyway, I hope you can find a answer to your problem, if you do, please let us know.

Have a good one!

[Gary]

I'm not sure how to direct you at this point, since I really don't know what's going on with that app.  You said that it works fine until you launch a secondary utility that the developer says shouldn't make a difference.  I can't test it, as I don't own those products.  I was thinking the secondary app was opening a tool window that was stealing focus (which happens more so in desktop apps than in games), so -ignorechildwindows is out.

For sanity and to make sure I hadn't broken something along the way, I threw together a small script that launches five separate apps and then rotates between them ten times, sending input to each, one character at a time (this one relies on getting each window via display to get the process target, versus creating subcommands that set the targeting at the command level):

>>>>>>>>>>>>>>>>>>>>>>>>>
Set small int (condition) [abc] value to 1
Run application 'C:\Users\Gary\Desktop\Book1.xlsx' - wait until it starts
Run application 'wordpad.exe' - wait until it starts
Run application 'notepad.exe' - wait until it starts
Run application 'C:\Users\Gary\Desktop\Doc1.docx' - wait until it starts
Run application 'onenote.exe' - wait until it starts
Start Loop While : [abc] Is Less Than Or Equals 10
    Write '[Orange] {SMALL:abc}' to log
    Display window '*excel*' as [Normal] (pause up to 3 seconds for availability)
    Press X key and hold for 0.1 seconds and release
    Display window '*wordpad*' as [Normal] (pause up to 3 seconds for availability)
    Press P key and hold for 0.1 seconds and release
    Display window '*notepad*' as [Normal] (pause up to 3 seconds for availability)
    Press N key and hold for 0.1 seconds and release
    Display window '*winword*' as [Normal] (pause up to 3 seconds for availability)
    Press W key and hold for 0.1 seconds and release
    Display window '*onenote*' as [Normal] (pause up to 3 seconds for availability)
    Press O key and hold for 0.1 seconds and release
    Set small int (condition) [abc] value as incremented by 1
End Loop
<<<<<<<<<<<<<<<<<<<<<<<<<

This worked without issue on this machine, and even waited the extended amount of time it takes Excel to load.   So, I'm going to say that normal targeting is working, even though I would consider the script above a little on the excessive side.   I'm going to have to call this an edge case rather than a bug unless there are more examples in various applications that I can get information about.  I encourage you to keep trying different combinations of actions to try to get the desired effect.  Also, there is the, 'Application Focus (Process Target) Guide' near the end of the VA help document.  Not sure if that will help in this situation but it's good to know when using VA.

[aurel42]

I'm going to have to call this an edge case rather than a bug unless there are more examples in various applications that I can get information about.  I encourage you to keep trying different combinations of actions to try to get the desired effect.

Oh my, I didn't want you to start doubting your whole codebase. Please forgive if I didn't give enough context, leading you on a wild-goose chase.

Prepar3D is just a version of good ole (32-bit) Microsoft Flight Simulator. FlyInside is a 3rd party tool that uses a modified DLL to grab P3D's DirectX rendering and turn it into an immersive VR experience for Oculus Rift and HTC Vive and it provides some nifty quality-of-life features like grabbing windows from the desktop and displaying them inside the cockpit as floating windows (so pilots can see P3D dialogs, but also something like a browser window with airport information while in VR).

FlyInside lets P3D render its "normal" main window on the desktop (P3D can't run fullscreen in this setup), and that's the window that usually has the focus and receives all the input. There is no separate FlyInside application or "control" window, all interaction with FlyInside itself happens in VR.

The good news:

I've used P3D with FlyInside in combination with VoiceAttack for a few months now without any issues whatsoever.

I often have a handful "undocked" P3D dialog windows open (each of them is a normal desktop window belonging to the P3D process), in addition to other windows I use inside the cockpit, and I switch between them all the time, either on the desktop or inside the cockpit, and I've never had any issue with VA commands not working in P3D because of other windows, or focus issues, or task switching.

And here end the good news.

I recently started recording and streaming sessions, and that's when I began to use the "mirror window" feature of FlyInside. FlyInside projects what I see inside the Rift to an extra window (I put that window on that part of my "extended" desktop that I grab with a capture card). Since this window is created by code in the "injected" DLL, it's still inside P3D's "context" (or whatever the correct technical term might be). This window does not accept input, when I hover the mouse over it, the mouse pointer turns into an "entry forbidden" sign. And only when this window appears, the main P3D window (often) appears to become "invisible" to VA.

I don't know enough about Windows internals to even begin to speculate how the Mirror window is different from "normal" windows that VA encounters (and has no problems with), but I noticed that it's the only extra window that appears in the Task Manager in addition to the main P3D window -- unlike the undocked dialog windows I mentioned before (like the ATC in this example):

https://drive.google.com/file/d/0B7Z-8ZNJtXTPWkUtTTBFeTVJWFk/view

When I set VA to send commands only to the "Lockheed Martin..." window, it's as if it tosses a coin when it first encounters the second window and decides whether it will still remember the first window whenever there are two to choose from.

When I start VA before I start P3D/FlyInside (which is my usual procedure), chances are very high that I will see the "Error acquiring process handle" message. When I start VA after I launched P3D/FlyInside and opened the mirror window, chances are higher that VA will send commands to the correct window without complaining, and I think this persists for the whole "session" (ie. until I restart VA and/or P3D), no matter how often I toggle the mirror window off and on subsequently.

So, I guess what I'm saying is, could there be an issue that only affects applications that open multiple application windows?

Possible test case(?): If the window list in the "Send commands to:" dropdown is "authoritative" for what VA sees or accepts as targets, and I use the "New Window" or "New Incognito Window" function of Chrome, should I be able to see all the different Chrome windows in that list after using the "refresh" option? Because I don't. What I see seems a bit inconsistent:

- when one Chome window is open, its title will always be in the dropdown, obviously
- when two Chrome windows are open, (almost?) always only the window that last had the focus will be in the list
- when three Chrome windows are open, the list will very often contain one, sometimes two (!) of the window titles, never all three of them

All the Chrome windows had different tabs open, so they had different names. Is this working as intended? If not, could it be related to my issue? If yes, could it still be related to my issue?

[Gary]

I may have a test build for you guys to try in the morning.  VA as it stands queries for just the active window for each application, as that would be the window to receive input for that application.  This is actually fine for most things.  What VA is going to do now is still search first for the active window (the most-likely scenario), but if that window is not found, it will search the windows that are not the top-most, active window.  If the window is located with this search an attempt is made to make the window the topmost window before sending input to it.  I may consider showing all windows in the process target (send commands to) dropdowns, but those lists may get kind of long. 

I have a few more things to test, but I need to step out for a bit o_O  I'll be back in later and hopefully get either a beta or at least an, 'unofficial' build out.

[Gary]

There's a new beta out there.  Hoping this may help out (for you as well, iceblast).

http://www.voiceattack.com/beta

[iceblast]

VA can hit the keys Alt+Tab again, and it works properly.

Still have a problem with Left Clicking, not every program though. What happens is, when you Left Click, the menu goes away, before you get to click on it. Doesn't seem to matter if it's set for active window or not.  It's not really a big deal, because a simple AHK script will press Left click for me.

I like the smooth mouse movement option though. Before this Beta, I couldn't use the positional mouse movement option, it would cause the menu to close like a Left Click, now, it doesn't. I could move the mouse by direction, just not by positional. So, that's nice. Positional movement is easier to setup. Glad I can use it again.

VA keeps getting better and better, keep up the good work.

[aurel42]

There's a new beta out there.

That's utterly brilliant, thank you so much! I'll check it out tonight and will report back.

[Gary]

What happens is, when you Left Click, the menu goes away, before you get to click on it.

What menu is this and do you have steps to reproduce the issue?  I could see if there's something amiss and fix it.

[iceblast]

ok, I made a video, to show you what's happening.

This menu I want VA to Left Click isn't a menu that goes away easily. I show you what I mean, by opening the menu, then showing you the command in VA, you can clearly see, the menu remains open while I'm Navigating VA, I even cycle through some other windows, and the menu still remains open the whole time.

I use the mouse to show you what should happen when you Left Click to turn off a style. I then turn it back on. Then I have VA Left Click the menu the same way, and you'll just see it close, it doesn't click off the style, you see the webpage didn't change. I try a few times, same thing, then use the mouse, and that works fine again.

I hope this gives you a better idea what's happening. The browser extension that is this menu is Stylish. Before the current beta, just doing Positional Mouse Movement on the menu, would also, cause it to close. Now I can use it, and it remains open. At the moment, a AHK script is just hitting the Left Mouse button when needed.

Let me know, if you need more info.

Windows 10 64 bit, 32gigs of ram. 3770k CPU

https://www.youtube.com/watch?v=HVghns25Ofw

[aurel42]

The beta version is a huge improvement for me.

I set the profile options to make VA send commands to the main ("Lockheed Martin...") window of P3D.

When the application (P3D) has two application windows open ("Lockheed Martin..." and "FlyInside Mirror Window"), 1.5.12 would (most of the time) not send commands to the "Lockheed Martin..." window, even when it had the focus.

The beta version 1.5.12.15 fixed this specific problem: the error message is gone, and as long as the "Lockheed Martin..." window has the focus, VA commands work reliably, no matter whether the secondary window is open or not.

When the application window doesn't have the focus, my VA commands are sent to the Active Window, instead, though (even though the target window is set in the profile options). I assume I wasn't supposed to test this codepath yet?

I then used the process target dropdown in the main VA window to select the main ("Lockheed Martin...") window of P3D.

Now, everything seems to work perfectly in every situation I tested. The main window always gains focus and receives my VA commands reliably, no matter whether the secondary window had the focus or another application. This codepath solves all my problems.

Thank you! I really appreciate that this didn't end up on the todo list for some future version. Best Customer Service Experience Evah!

[Gary]

Iceblast, can you do me a favor and try your mouse click scenario again?  But, before you do, go into the options screen on the last tab and check the box labeled, 'bypass mouse targeting'.   Let me know if that makes a difference in what you are experiencing with that menu.

[iceblast]

Yup, that did the trick. It use to be, that bypass mouse targeting check box, wouldn't stay checked. I guess that has changed. I wonder what difference that command does.

Well with that working for me now, I have no ahk scripts running, pure VA now.

Thanks for looking into this for me. Keep up the great work!!!

[Gary]

Alright... now one more task for you.  If you would, please get the latest beta:  http://www.voiceattack.com/beta

Next, go BACK into your options and uncheck the box, 'Bypass Mouse Targetting', putting things back where they were.

Then, try your Stylish plugin test to see if your menus continue to work.  What was happening is VA was being too aggressive at bringing things to the front and it was causing that plugin to think that Chrome was receiving the focus (so it would disappear).  You should be good to go.

What that option does is it basically puts VA back to the way the mouse used to target windows (which means you would have to manually focus the window you would want to click).

Thanks for the heads-up!

[iceblast]

Yup, the new beta did the trick again. 'Bypass Mouse Targetting' is now off, and VA is clicking properly again.

Thanks for getting that fixed, and again, Thanks for the great work!!!