Bridged networking on VirtualBox installed on pfSense host

anand_phulwani

Bridged networking on VirtualBox installed on pfSense Host

Hi,

We are using pfSense in our small office and i have been asked to run a windows XP virtual machine on pfSense host.

I have created two separate machine for this

FreeBSD 10.1-RELEASE (GENERIC) r274401
pfSense RELENG 2.2.i386(build on FreeBSD 10.1-RELEASE-p25 c39b63e)

i have tried installing virtualbox-ose and virtualbox-ose-additions in both of these via command.

pkg install virtualbox-ose virtualbox-ose-additions

which worked perfectly on FreeBSD but on pfSense it installed/updated pkg manager and gave a warning "makewhatis not found", i copied the makewhatis file from the FreeBSD to pfSense, and re-run the installation and it installed perfectly.

I used the commands

kldload vboxdrv
kldload vboxnetadp
kldload vboxnetflt

to load the modules in kernel.

Further i tried creating a VM and importing a VM in the same way for both the machines, on the FreeBSD it works find with bridged networking or NAT networking, in pfSense it is unable to work with bridged networking but NAT networking works fine.

i tried disabling the firewall by the command

pfctl -d

and tried again but in vain.

Further i added a tracefile in both the machine adapters using command

VBoxManage modifyvm "WinXp" –nictrace1 on --nictracefile1 file.pcap

and pinged the router using the guest, both the files show that the ICMP is requested, but only FreeBSD machine trace file shows that it is recieving the reply.

Further more i watched both the host bridged adapters using the commands

FreeBSD: tcpdump -i em0 -n host 192.168.21.60
pfSense: tcpdump -i em0 -n host 192.168.21.70

With both tracefiles on the vms and tcpdumps on bridged adapters, i found out that

On freebsd:

The ARP request originates from guest (Source: VM's tracefile)
Gets forwarded to the adapter(em0) (Source: Host's em0 tcpdump)
The adapter(em0) recieves an ARP reply (Source: Host's em0 tcpdump)
The ARP reply is reflected in the VM's tracefile (Source: VM's tracefile)

On pfSense:

The ARP request originates from guest (Source: VM's tracefile)
Gets forwarded to the adapter(em0) (Source: Host's em0 tcpdump)
The adapter(em0) recieves an ARP reply (Source: Host's em0 tcpdump)
The ARP reply is not reflected in the VM's tracefile (Source: VM's tracefile)

i have asked them to run a virtual enviroment like ESXI, but we have had bad experiences with it which is why they are reluctant in doing so, i have no other option than working the current situation out.
freebsd_vm_pcap_tracfile_tcpdump.txt
freebsd_em0_tcpdump.txt
pfSense_vm_pcap_tracfile_tcpdump.txt
pfSense_em0_tcpdump.txt

heper

brilliant plan to run a crappy desktop OS inside crappy virtual machine software on a firewall-host …. what could go wrong :p

good luck finding someone who wants to support that

anand_phulwani

The boss wants what it wants. :-\

kpa

You're compromising the integrity of your firewall by running software that might have unknown vulnerabilities, VirtualBox is not known for its stellar security record. Why does your boss insist on using the pfSense system for that, don't you have any other options for running virtualbox, a separate system running FreeBSD or Linux for example?

anand_phulwani

I know and I have told him a 100 times, but he isn't ready to listen, he thinks it as an excuse for not doing the job, he will surely listen to my argument and the huge security risk involved once I complete the job.

johnpoz

If you want to run VMs on this host, why don't you run Type 1 hypervisor on it, then run whatever vms you want.. Pfsense would just be one of the VMs

This would be clearly a better option both from security and performance and ease of setup.

On a side note XP is a dead OS, it is no longer supported at all.. No security fixes, etc.. That is a bad choice for an OS no matter running physical or virtual.

CSylvain

I also see this same problem with a version 2.3.1-RELEASE-p1 to be able to launch compatible tools only under Windows, it will be run for only interventions on a wireless infrastructure.
I also did the test on a "FreeBSD 10", and it works, so I feel that lack of things in the Kernel.
I continued my research by making this comparison with PfSense and installing a FreeBSD, then coming back to you if I find the solution. ;)

CSylvain

After several tests, it is the Kernel problematic, replacing the /boot/kernel by FreeBSD 10.3, it works !
The question is who is involved ?

Because the Kernel from pfSense includes mostly the modules, which is not the case of FreeBSD which is compiled individually and place in /boot/kernel.
I looked if modules were missing, and everything is present :

.............................................
2    3 0xffffffff819bd000 6d370    vboxdrv.ko (/boot/modules/vboxdrv.ko)
	Contains modules:
		Id Name
		 1 vboxdrv
 3    1 0xffffffff81c11000 3831     ng_socket.ko (/boot/kernel/ng_socket.ko)
	Contains modules:
		Id Name
		484 ng_socket
 4    3 0xffffffff81c15000 ba02     netgraph.ko (/boot/kernel/netgraph.ko)
	Contains modules:
		Id Name
		483 netgraph
 5    2 0xffffffff81c21000 29b2     vboxnetflt.ko (/boot/modules/vboxnetflt.ko)
	Contains modules:
		Id Name
		485 ng_vboxnetflt
 6    1 0xffffffff81c24000 4123     ng_ether.ko (/boot/kernel/ng_ether.ko)
	Contains modules:
		Id Name
		486 ng_ether
 7    1 0xffffffff81c29000 3f64     vboxnetadp.ko (/boot/modules/vboxnetadp.ko)
	Contains modules:
		Id Name
		487 vboxnetadp

Is it because everything is integrated, for this to be a problem ?

I discover every day FreeBSD I'll see if I can make for a pfSense Kernel with non-integrated modules.

CSylvain

[Fixed] Bridged networking on VirtualBox installed on pfSense : https://forum.pfsense.org/index.php?topic=113516.0

anand_phulwani

@CSylvain:

After several tests, it is the Kernel problematic, replacing the /boot/kernel by FreeBSD 10.3, it works !
The question is who is involved ?

Because the Kernel from pfSense includes mostly the modules, which is not the case of FreeBSD which is compiled individually and place in /boot/kernel.
I looked if modules were missing, and everything is present :
.............................................
2    3 0xffffffff819bd000 6d370    vboxdrv.ko (/boot/modules/vboxdrv.ko)
	Contains modules:
		Id Name
		 1 vboxdrv
 3    1 0xffffffff81c11000 3831     ng_socket.ko (/boot/kernel/ng_socket.ko)
	Contains modules:
		Id Name
		484 ng_socket
 4    3 0xffffffff81c15000 ba02     netgraph.ko (/boot/kernel/netgraph.ko)
	Contains modules:
		Id Name
		483 netgraph
 5    2 0xffffffff81c21000 29b2     vboxnetflt.ko (/boot/modules/vboxnetflt.ko)
	Contains modules:
		Id Name
		485 ng_vboxnetflt
 6    1 0xffffffff81c24000 4123     ng_ether.ko (/boot/kernel/ng_ether.ko)
	Contains modules:
		Id Name
		486 ng_ether
 7    1 0xffffffff81c29000 3f64     vboxnetadp.ko (/boot/modules/vboxnetadp.ko)
	Contains modules:
		Id Name
		487 vboxnetadp
Is it because everything is integrated, for this to be a problem ?

I discover every day FreeBSD I'll see if I can make for a pfSense Kernel with non-integrated modules.

Dear CSylvain,

Unfortunately it is very hard to access forums from my country India, as pfsense forums are blocked, i don't know why, but you are bang on, i was following the forum before your first comment very aggressively but once the forum didn't respond well, there was no choice to dig in deep myself, a lot of research led me to kldstat and yes since everything is integrated into kernel itself, i started playing with kernel options, and stripped all the kernel options to find out that it was working then, then i used Binary search algorithm to find out the culprit and it worked and removing NETGRAPH_SOCKET made things working from the kernel configuration, and building the ISO worked.

But still lot lot lot of thanks, for taking the pain for working this out, also i never knew that just re-building the kernel can make things work out, loads loads and loads of thanks mate, for doing so much for me, i know somebody hardly would do so much without any incentive, i just cant thank you much for this.

Thanks,
Anand