grsecurity host hangs after trying to run guest

[martin.]

When host uses hardened kernel 2.6.24 or higher with pax patch, then host hangs after trying to run guest.

steps to reproduce:

1. patch vanilla kernel with pax patch (​http://www.grsecurity.net/test/pax-linux-2.6.28.2-test11.patch)
2. leave all pax options disabled
3. try to run guest under pax kernel

result: host will hang after opening guest window and before guest begins to boot without any log. I tried to create a core dump after opening guest window, but before host hangs. Dump is in attachment.

I reported this to pax paxteam too. Here are our current mails:

I tried use virtualbox-ose on gentoo hardened-sources-2.6.27-r3. But running of virtual machine causes the host computer to hang.

I tried to run virtualbox-ose-2.0.4 and vmware-workstation-6.5.0.118166 on vanilla 2.6.27.8 kernel and both works fine.

I tried to patch vanilla 2.6.27.8 with grsecurity-2.1.12-2.6.27.8-200812102238 patch. I leaved all new freatures disabled, so I expect that this kernel is same as vanilla. But on this kernel, trying to run virtual machine under virtualbox causes hang of host machine after few seconds. Atempt to run virtual machine under vmware caused host reboot.

I tried to patch with pax-linux-2.6.27.8-test23.patch too, but I have the same results as with grsecurity patch.

It looks like patching vanilla kernel with grsec patch breaks something in kernel.

there're some things that PaX changes even if all configurable options are disabled, it's possible that some of these interfere with existing code (and they'd need to be patched to work under PaX). now to tell which exact PaX change causes such failures is a hard question to answer, CPU crashes are not exactly trivial or easy to debug ;). if you feel like, you can try to patch some logging code into the vmware/vbox kernel modules and do some binary search to find the code that triggers the crash, then i can probably figure out what goes wrong, but i won't have time or motivation for this myself in the foreseeable future, i'm afraid...

I'm not be able to create patch into kernel or modules, but when you will write patches then I can test them and send result back to you.

I tested older versions of grsecurity. In current stable grsecurity-2.1.11-2.6.24.5-200804211829 vmware still reboots, but vbox didn't crash and write this message:

vboxdrv: Trying to deactivate the NMI watchdog permanently... vboxdrv: Successfully done. vboxdrv: Found 2 processor cores. vboxdrv: fAsync=1 offMin=0x535c2 offMax=0x535c2 vboxdrv: TSC mode is 'asynchronous', kernel timer mode is 'normal'. vboxdrv: Successfully loaded version 2.0.4 (interface 0x00090000). PAX: VirtualBox:12283, uid/euid: 1000/1000, invalid execution attempt at ffffc20004969000 RIP:

[<ffffc20004969000>]

PGD 8000000000530063 PUD 9f015067 PMD 9e384063 PTE 80c8e003 Oops: 0011 [1] PREEMPT SMP CPU 0 Modules linked in: vboxdrv snd_pcm_oss snd_mixer_oss snd_seq_oss snd_seq_midi_event snd_seq snd_seq_devic e bridge llc fuse ide_cd pcmcia cdrom ata_piix snd_hda_intel yenta_socket ohci1394 snd_pcm rsrc_nonstatic

sdhci snd_timer ieee1394 pcmcia_core mmc_core piix iwl3945 ide_core i2c_i801 snd_page_alloc mac80211 joy

dev pcspkr snd_hwdep video cfg80211 snd output button Pid: 12283, comm: VirtualBox Not tainted 2.6.24.5-grsec #2 RIP: 0010:[<ffffc20004969000>] [<ffffc20004969000>] RSP: 0018:ffff810094bc7d70 EFLAGS: 00010046 RAX: 000000000000000f RBX: 0000000000000246 RCX: ffff81008b5b0810 RDX: 0000000000000000 RSI: 000000000000000f RDI: ffffc200050a7000 RBP: ffff810094bc7d98 R08: ffff81008b5b0810 R09: 0000000000000000 R10: 0000000000000000 R11: ffffffff803463fa R12: ffffc200050a7000 R13: 00000000ffffffdb R14: ffff81008b5b0810 R15: 0000000040234e50 FS: 0000000040235950(0063) GS:ffffffff8066e000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffffc20004969000 CR3: 000000008b5a6000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process VirtualBox (pid: 12283, threadinfo ffff810094bc6000, task ffff81008b47cf60) Stack: ffffffff8820f734 0000000000000000 ffff81008b596240 ffff8100949bff10

ffffffff88384290 ffff810094bc7dd8 ffffffff8820f9b8 ffffffff8825dc35 0000000000000046 ffffffff80366f81 5e51fc3c0cf1dbd7 ffffffff806789e0

Call Trace:

[<ffffffff80366f81>] memcmp+0x0/0x4f [<ffffffff8838a359>] :vboxdrv:supdrvIOCtl+0xd23/0x1162 [<ffffffff8838c922>] :vboxdrv:rtMemAlloc+0xac/0xf4 [<ffffffff88386281>] :vboxdrv:AssertMsg1+0x16a/0x1fe [<ffffffff802aef7d>] do_ioctl+0x35/0x96 [<ffffffff802af21e>] vfs_ioctl+0x240/0x26e [<ffffffff802af2ad>] sys_ioctl+0x61/0x98 [<ffffffff8020301e>] system_call+0x7e/0x83

Code: 49 89 f9 49 8d 91 20 03 00 00 8c c8 50 e8 0e 00 00 00 c3 cc RIP [<ffffc20004969000>]

RSP <ffff810094bc7d70>

CR2: ffffc20004969000 ---[ end trace e68c0759744cb53c ]---

I didn't find older version of grsecurity or pax patches, but I tried gentoo hardened-sources-2.6.23-r13, on this version everything works fine.

I hope that it's helpful.

Martin

[martin.]

I tested this with virtualbox-ose-2.1.4 and pax-linux-2.6.28.8-test23. Problem still persist. There are new message after I loaded modules: vboxdrv: Trying to deactivate the NMI watchdog permanently... vboxdrv: Successfully done. vboxdrv: Found 2 processor cores. VBoxDrv: dbg - g_abExecMemory=ffffffffa02dd000 vboxdrv: fAsync=1 offMin=0x54177 offMax=0x54177 vboxdrv: TSC mode is 'asynchronous', kernel timer mode is 'normal'. vboxdrv: Successfully loaded version 2.1.4 (interface 0x000a0009). VBoxNetFlt: dbg - g_abExecMemory=ffffffffa047f000 warning: `VirtualBox' uses 32-bit capabilities (legacy support in use)

[martin.]

discussion about this bug is also there ​http://forums.grsecurity.net/viewtopic.php?f=3&t=2104

[Frank Mehnert]

Duplicate of #2652. grsecurity-enabled hosts are not supported by VirtualBox.