Bhyve networking blocked by pfsense?



  • I'm running pfsense on one of these: http://www.pcengines.ch/apu2c4.htm

    I'm trying to get a linux vm running under bhyve on it. The machine has three physical network interfaces (igb{0,1,2}), and I use one for WAN and one for LAN for the standard firewall stuff, so I have the third interface (igb1) which I can dedicate to bhyve.

    I've been using a "vanilla" FreeBSD VM to create a linux disk image, so I already have a system to boot, but I am unable to get working networking for the VM under pfsense.

    I use the following command to start the VM:

    grub-bhyve -r hd0,msdos1 -m device.map -d /grub -M 1024 ubuntu && \
    bhyve -c 1 -m 1024M -H -P -A -l com1,stdio -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,ubuntu.img ubuntu
    

    tap0 and igb1 are both members of bridge0, which I create like:

    ifconfig tap0 create up
    ifconfig bridge0 create up
    ifconfig bridge0 addm igb1
    ifconfig bridge0 addm tap0
    

    Interfaces, as described by ifconfig:

    
    igb1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=400b8 <vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso>ether 00:0d:b9:41:60:95
    	inet6 fe80::20d:b9ff:fe41:6095%igb1 prefixlen 64 scopeid 0x2 
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    bridge0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    	ether 02:a4:4a:32:c1:00
    	nd6 options=1 <performnud>id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    	member: tap0 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 9 priority 128 path cost 2000000
    	member: igb1 flags=143 <learning,discover,autoedge,autoptp>ifmaxaddr 0 port 2 priority 128 path cost 20000
    tap0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
    	options=80000 <linkstate>ether 00:bd:ad:21:e8:00
    	inet6 fe80::2bd:adff:fe21:e800%tap0 prefixlen 64 scopeid 0x9 
    	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect
    	status: active
    	Opened by PID 66171</performnud,auto_linklocal></linkstate></up,broadcast,running,promisc,simplex,multicast></learning,discover,autoedge,autoptp></learning,discover,autoedge,autoptp></performnud></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast> 
    

    Machine boots ok, but network is not working, I don't get an ip from the dhcp server (which is on igb0). If I connect the cord that is connected to igb1 to another machine it connects to the lan successfully. If I set a static ip on the virtual nic and try to ping my gw ip I can see the ping requests, but no ping replies, using tcpdump on the physical host bridge0 interface.

    # tcpdump -i bridge0 -n icmp
    tcpdump: WARNING: bridge0: no IPv4 address assigned
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on bridge0, link-type EN10MB (Ethernet), capture size 65535 bytes
    10:59:26.383788 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 34, length 64
    10:59:27.392437 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 35, length 64
    10:59:28.392378 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 36, length 64
    10:59:29.391348 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 37, length 64
    10:59:30.390402 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 38, length 64
    10:59:31.389978 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 39, length 64
    10:59:32.389386 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 40, length 64
    10:59:33.388541 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 41, length 64
    10:59:34.387731 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 42, length 64
    10:59:35.387474 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 43, length 64
    10:59:36.386352 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 44, length 64
    10:59:37.385366 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 45, length 64
    10:59:38.385192 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 46, length 64
    10:59:39.384444 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 47, length 64
    10:59:40.383546 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 48, length 64
    10:59:41.383055 IP 10.11.12.25 > 10.11.12.1: ICMP echo request, id 1177, seq 49, length 64
    ^C
    16 packets captured
    36 packets received by filter
    0 packets dropped by kernel
    
    

    But I don't see anything on the igb0 interface (where 10.11.12.1 ip is assigned):

    
    # tcpdump -i igb0 -n "host 10.11.12.25"
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on igb0, link-type EN10MB (Ethernet), capture size 65535 bytes
    11:00:19.455243 ARP, Request who-has 10.11.12.1 tell 10.11.12.25, length 46
    11:00:19.455265 ARP, Reply 10.11.12.1 is-at 00:0d:b9:41:60:94, length 28
    ^C
    2 packets captured
    37382 packets received by filter
    0 packets dropped by kernel
    

    Nor do I see the pings on the igb1 interface:

    
    # tcpdump -i igb1 -n icmp
    tcpdump: WARNING: igb1: no IPv4 address assigned
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on igb1, link-type EN10MB (Ethernet), capture size 65535 bytes
    ^C
    0 packets captured
    21 packets received by filter
    0 packets dropped by kernel
    
    

    So I guess I have either something wrong with my bridge, or that the firewall is eating up my traffic, but I'm to much of a BSD noob to say for sure.

    Do you know what I'm doing wrong?


  • Rebel Alliance Developer Netgate

    Do you see anything blocked in the firewall log?

    You'll probably need to do one of a couple possible things:

    1. Create an interface group in the GUI and then make sure your tap interface gets added to the group, then add firewall rules on the group tab.
    -or-
    2. System > Advanced, Tunables tab, change the bridge filtering sysctls to filter on the bridge not the members, then assign the bridge0 interface and add rules to it. You'll probably actually want to make the bridge interface in the pfSense GUI with your existing interface on it, and add the tap interface later on programmatically. Otherwise when you reboot you'll be in for a bad time.



  • Creating the bridge with only igb1 as a member in the webUI was enough to get DHCP working, or so it seemed, worked once.

    To start filtering somewhat properly I had to assign tap0 and bridge0 to OPT2 and OPT3. That way I could also add tap0 to the bridge using the gui.

    I changed filtering from interfaces to the bridge as suggested:

    | Tunable Name | Description | Value | New |
    | net.link.bridge.pfil_bridge | Packet filter on the bridge interface | 1 |
    | net.link.bridge.pfil_member | Packet filter on the member interface | 0 |

    Then added a firewall rule to allow traffic on the bridge. And now I have ping working on LAN, but not getting NAT:ed out to WAN for some reason.

    Unsure what will happen on reboot too, as tap0 probably won't be re-created.



  • Couldn't figure out how to set the firewall rules for proper filtering.

    But!

    Setting the following seem to have side-stepped the problem entirely:

    | Tunable Name | Description | Value | New |
    | net.link.bridge.pfil_bridge | Packet filter on the bridge interface | 0 |
    | net.link.bridge.pfil_member | Packet filter on the member interface | 0 |


  • Rebel Alliance Developer Netgate

    Setting both to 0 means you can't filter anything involving that bridge, which is highly undesirable.

    Don't assign the tap interface in the GUI, try using an earlyshellcmd to create the tap interface and and then a regular shellcmd to addm it to the bridge.

    Both types of shellcmd entries can be editing using the shellcmd package.


Log in to reply