Memory leak in VirtualBox

[indifferent]

• Host: openSUSE 13.1 x86_64
• Guest: RHEL 6.5 x86_64 (can be reproduced also with RHEL 6.5 i586, RHEL 7.0 x86_64)

Steps to reproduce:

• Launch VM.
• Launch a command to extract and execute TCK harness for Java, and create a report into shared folder.
• TCK execution may take more than 8 hours, so leave the machine overnight.
• In the morning 100% RAM and 100% swap are eaten.
• Looking in KSysGuard, it appears that VirtualBox guest RAM goes to "shared memory", while memory leak happens in "memory" (other vms have appropriate "Shared memory" size and only couple of megabytes for "memory").
• TCK mostly launches child processes, every child process perofrms some kind of test, including network connections, both to local machine and to other machines.

Attaching VBox.log, config for VM and /proc/id/maps file.

In addition, I saved full memory dump, if that can be helpful. It occupies 24GB, so no way to attach it anywhere.

[indifferent]

Both VBox.log and /proc/id/maps files, packed because of 512KB restriction.

[indifferent]

Screenshot of KSysGuard and VirtualBox window that is trying to shutdown, but stucks on guest additions.

[indifferent]

Additional info: VBox.log shows multiple error messages after 20:40, which is probably the time host OS run out of RAM and swap. At that time host os was 100% loaded on all cores, all RAM and all swap.

Then, shutdown command was issued through guest gui at 23:10, so all the following messages is the attempt to shut down properly, and at 23:40 I gave up and killed the process.

[Frank Mehnert]

Thank you for this report. I would indeed be interested in the core dump. However, unfortunately I cannot provide you a server where you could upload it. Can you provide the core dump in any way? If so, could you contact me at frank _dot_ mehnert _at_ oracle _dot_ com? My current guess is that this has something to do with the shared folders code.

[Frank Mehnert]

Finally found and fixed the leak. Thank you for providing the detailed information for reproducing the problem. I have created a 4.3.17 test build, could you install this build and confirm the leak is gone? Btw, we are aware that guest execute needs more CPU load than necessary (look at the CPU consumption of VBoxSVC, VBoxManage and VBoxXPCOMICPD). This is a different issue and mentioned elsewhere. We will fix this problem later.

[Frank Mehnert]

Fix is part of VBox 4.3.18. Thanks again for your help!