Bhyve networking blocked by pfsense?

azzid

I'm running pfsense on one of these: http://www.pcengines.ch/apu2c4.htm

I'm trying to get a linux vm running under bhyve on it. The machine has three physical network interfaces (igb{0,1,2}), and I use one for WAN and one for LAN for the standard firewall stuff, so I have the third interface (igb1) which I can dedicate to bhyve.

I've been using a "vanilla" FreeBSD VM to create a linux disk image, so I already have a system to boot, but I am unable to get working networking for the VM under pfsense.

I use the following command to start the VM:

grub-bhyve -r hd0,msdos1 -m device.map -d /grub -M 1024 ubuntu && \
bhyve -c 1 -m 1024M -H -P -A -l com1,stdio -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,ubuntu.img ubuntu

tap0 and igb1 are both members of bridge0, which I create like:

ifconfig tap0 create up
ifconfig bridge0 create up
ifconfig bridge0 addm igb1
ifconfig bridge0 addm tap0

Interfaces, as described by ifconfig:


igb1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=400b8 <vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso>ether 00:0d:b9:41:60:95
	inet6 fe80::20d:b9ff:fe41:6095%igb1 prefixlen 64 scopeid 0x2 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	ether 02:a4:4a:32:c1:00
	nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: tap0 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 9 priority 128 path cost 2000000
	member: igb1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 2 priority 128 path cost 20000
tap0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
	options=80000 <linkstate>ether 00:bd:ad:21:e8:00
	inet6 fe80::2bd:adff:fe21:e800%tap0 prefixlen 64 scopeid 0x9 
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect
	status: active
	Opened by PID 66171</performnud,auto_linklocal></linkstate></up,broadcast,running,promisc,simplex,multicast></learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></performnud></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>

Machine boots ok, but network is not working, I don't get an ip from the dhcp server (which is on igb0). If I connect the cord that is connected to igb1 to another machine it connects to the lan successfully. If I set a static ip on the virtual nic and try to ping my gw ip I can see the ping requests, but no ping replies, using tcpdump on the physical host bridge0 interface.

# tcpdump -i bridge0 -n icmp
tcpdump: WARNING: bridge0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bridge0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:59:26.383788 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 34, length 64
10:59:27.392437 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 35, length 64
10:59:28.392378 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 36, length 64
10:59:29.391348 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 37, length 64
10:59:30.390402 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 38, length 64
10:59:31.389978 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 39, length 64
10:59:32.389386 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 40, length 64
10:59:33.388541 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 41, length 64
10:59:34.387731 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 42, length 64
10:59:35.387474 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 43, length 64
10:59:36.386352 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 44, length 64
10:59:37.385366 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 45, length 64
10:59:38.385192 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 46, length 64
10:59:39.384444 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 47, length 64
10:59:40.383546 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 48, length 64
10:59:41.383055 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 49, length 64
^C
16 packets captured
36 packets received by filter
0 packets dropped by kernel

But I don't see anything on the igb0 interface (where 10.11.12.1 ip is assigned):


# tcpdump -i igb0 -n "host 10.11.12.25"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:00:19.455243 ARP, Request who-has 10.11.12.1 tell 10.11.12.25, length 46
11:00:19.455265 ARP, Reply 10.11.12.1 is-at 00:0d:b9:41:60:94, length 28
^C
2 packets captured
37382 packets received by filter
0 packets dropped by kernel

Nor do I see the pings on the igb1 interface:


# tcpdump -i igb1 -n icmp
tcpdump: WARNING: igb1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 65535 bytes
^C
0 packets captured
21 packets received by filter
0 packets dropped by kernel

So I guess I have either something wrong with my bridge, or that the firewall is eating up my traffic, but I'm to much of a BSD noob to say for sure.

Do you know what I'm doing wrong?

jimp

Do you see anything blocked in the firewall log?

You'll probably need to do one of a couple possible things:

1. Create an interface group in the GUI and then make sure your tap interface gets added to the group, then add firewall rules on the group tab.
-or-
2. System > Advanced, Tunables tab, change the bridge filtering sysctls to filter on the bridge not the members, then assign the bridge0 interface and add rules to it. You'll probably actually want to make the bridge interface in the pfSense GUI with your existing interface on it, and add the tap interface later on programmatically. Otherwise when you reboot you'll be in for a bad time.

azzid

Creating the bridge with only igb1 as a member in the webUI was enough to get DHCP working, or so it seemed, worked once.

To start filtering somewhat properly I had to assign tap0 and bridge0 to OPT2 and OPT3. That way I could also add tap0 to the bridge using the gui.

I changed filtering from interfaces to the bridge as suggested:

Then added a firewall rule to allow traffic on the bridge. And now I have ping working on LAN, but not getting NAT:ed out to WAN for some reason.

Unsure what will happen on reboot too, as tap0 probably won't be re-created.

azzid

Couldn't figure out how to set the firewall rules for proper filtering.

But!

Setting the following seem to have side-stepped the problem entirely:

jimp

Setting both to 0 means you can't filter anything involving that bridge, which is highly undesirable.

Don't assign the tap interface in the GUI, try using an earlyshellcmd to create the tap interface and and then a regular shellcmd to addm it to the bridge.

Both types of shellcmd entries can be editing using the shellcmd package.