I had the same problem and solved it by switching to Intel Pro/1000 T-Server network adapters.
Apparently it's a bug in FreeBSD, see http://eligere.wordpress.com/2008/03/05/virtualbox-156-freebsd-70-and-pcnet-fast-iii-problem/
Unfortunately, I figured it out after messing around for quite some time.
Perhaps a notice should be added to the download page, since it's pretty common to try it under a VM.
Edit: I'm using VirtualBox 2.0.6