Maybe I'm the last one on the planet to discover this horrible default setting in VirtualBox but the difference is so awesome I have to tell the world.
MAKE SURE THIS IS CHECKED! If you find it default unchecked, you'll be amazed.
EDIT: As pointed out, you do need memory to get this advantage. In my case, I had the memory (16GB) but still bad performance. Your mileage may vary.