SYSTEM_THREAD_EXCEPTION_NOT_HANDLED with 4K resolution in Win10 guest

[kFYatek]

I have Win10 1703 (Creators Update) as both the host and the guest. My host computer is connected to a 4K (3840x2160) display which is set at 100% scaling (no HiDPI handling). The guest machine has both 2D and 3D acceleration enabled and virtual GPU RAM size is set all the way up to 256 MiB.

When I install guest additions, it works great at first. But when I maximize the guest window (which sets the guest's screen size at 3840x2055) and attempt to reboot the VM in that state, the guest OS tries to switch to that screen mode (borders of the black screen in guest window expand to the desired size) for a brief moment, but then BSODs with SYSTEM_THREAD_EXCEPTION_NOT_HANDLED.

The only way to fix this is to select low-res mode from recovery boot options and set some lower resolution so that it gets saved in the registry - rebooting in normal mode works fine afterwards.

So all in all, it seems like the virtual GPU driver crashes if screen size is too big at boot time, even though it can switch to such mode just fine on a fully booted system.