Solved: pfSense as bhyve guest only gives 60Mbit instead 200+

soupdiver

After some tinkering I have setup my pfSense box.
My setup:
DLS Modem (Vigor 160) in (full-bridge mode) -> Intel NUC -> LAN

On my Intel NUC I added a second NIC via USB-C and installed pfSense as a guest in bhyve.
I created two bridges and tap devices on my host machine and passed them into the VM.
I configured my WAN interface for PPPoE and entered my login info. pfSense gets an IP and is able to reach the internet. All good except one thing: When I test my internet speed I only receive around 60-70MBit/s instead of the 250 I should get.
When I login to the modem I can see that it syncs with the correct speed of 245/46.

I already tried to usually mentioned options:

Hardware Checksum Offloading on/off
Hardware TCP Segmentation Offloading on/off
Hardware Large Receive Offloading on/off
verify MTU (set to 1492 on the modem and inside pfSense)
set MSS (tried with 1452)
check system load when traffic is high

When flipping the offload features I could see some change in traffic but not dramatically. The range was ~45-60 Mbit/s.
MTU/MSS seems to have to effect or just very small.
Load of the system seams fine. Load of the VM is around 15% on the host. The machine itself has 4 cores + HT and I gave the VM 4 cores.

I'm not 100% where to continue digging. Since the PPPoE connection is established by pfSense I can't hook into it earlier I guess.

What I would try next is to setup the pppoe connection directly on the host and see if this changes my speed.

Any more ideas? In case it matters it is a Telekom VDSL 250Mbit connection I have.

stephenw10

Yes I would remove the bhyve virtualization and test pfSense directly with your connection first.

But you certainly want to have all hardware off-loading disabled in a VM.

Make sure you're not maxing out one CPU core, try running top -aSH while testing the throughput.

Steve

soupdiver

@stephenw10

I would remove the bhyve virtualization and test pfSense directly with your connection first.

Yea will try that

But you certainly want to have all hardware off-loading disabled in a VM.

yea, makes sense

Make sure you're not maxing out one CPU core, try running top -aSH while testing the throughput.

Already did, nothing was under heavy load. Max on a core maybe around 30%.

Ultimately I wanna run it as a VM so I can better make use of my NUC and can keep the base system stable. Already took out everything more than once :D

soupdiver

Ok, I setup my PPPoE connection directly on the host using native ppp.
Connection worked instantly (what a surprise) but the speed is still not great. I get around 100Mbit/s now. A bit better but still far from what it should be.

What else could it be?

stephenw10

PPPoE is limited in pfSebse/FreeBSD by the the fact it can only use as single NIC queue:
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

But unless your CPU is running at 10th normal speed it should easily pass more than 100Mbps.

Is the NIC actually linked at 1G?

Do you see any errors or collisions on it in Status > Interfaces? You will have to assign the parent NIC to see it there.

Steve

soupdiver

@stephenw10 yea it's a usb-c dongle that I used on my macbook.
Speed was fine there.
So the nic should be ok.

Yea, the CPU is an i7... not close to see a core at very high load.
On the hast it was not pfsense. vanilla FreeBSD with pppd.

Collisions and errors are 0 but maybe that's because I passed a tap device to the VM and not the real device.

I will see if I can test with a linux laptop and see if the result is different. If not maybe the modem is misconfigured... what I kind of doubt but who knows.

stephenw10

USB NICs are notoriously bad in pfSense/FreeBSD. If you can test with a different NIC type I would. Does it actually show that as linked at 1G? Some USB NICs do not report a link speed if they default to the cdce driver.

Steve

netblues

Also, your intel nuc does it actually use an intel lan card?
Checkit out.
Forget about using a usb nic on pfsense.
Even if it works, if it is disconnected, it will kick in the interface reconfiguration script.

Use a small managed lan switch, and do router on a stick port duplication. 5 ports model will do.

soupdiver

Also, your intel nuc does it actually use an intel lan card?

Is this important somehow?

Seems I come to the same conclusion. The USB NIC does not work very will in this setup.

I did some more testing:
Created a PPPoE connection from my macbook with the same usb ethernet adapter. I instantly get full speed.

Then I switched my NICs on the NUC. The USB-C NIC for LAN and the internal one for WAN.
This (kinda) seems to do the trick. I now have full download speed when I download something on the NUC. However if another client on the network downloads something the limit seems to be around 100Mbit/s.
CPU load does not show anything crazy. Even with the NUC using 200Mbit/s the CPU looks fine.

So it seems to come down to the USB interface somehow.

What I wonder is:
I used that same adapter before in my network (with macbook not the NUC) and never had any issues.
I did some more testing:
I transfered a file from my macbook to the NUC and there I easily get my 1GBit/s as expected for the LAN.

It seems the USB NIC in general is capable to deliver the needed bandwidth but in combination with PPPoe/pfSense something is odd.

From my understanding the NIC "simply" transfers Ethernet frames over the wire... I thought there should be no difference in what kind of data it transfers but there seems to be something different between LAN file transfer, PPPoE and vnet interface inside pfSense.
Maybe it's something with the driver on FreeBSD... no idea... but it starts to feel more esoteric now somehow

Use a small managed lan switch, and do router on a stick port duplication. 5 ports model will do.

I had to look up "router on a stick". How would this look like in my scenario
Create VLAN2 (LAN) and VLAN3(WAN) on the switch.
Attach my modem to the VLAN3 port and home switch to the VLAN2 port and have a trunk port that I connect my NUC to.
Inside the NUC I then create 2 vlan devices and pass them to pfSense instead of my tap0/1 interfaces that I had before?
That's how I understood the setup so far.
Since the managed switch is also my home switch, can I just connect a cable from VLAN2 port to another "normal" port on the same switch to connect the rest of my home network?

Thanks for all the input so far

netblues

@soupdiver If the nuc is using realtek chipset for the lan, then performance will be much less than 1 gbit.
Its not always the case, but just check it out.

As for the router on a stick, what you describe is what is needed.
And obviously you can expand this to more vlans terminating to pfsnse over the trunk port.

soupdiver

@netblues

If the nuc is using realtek chipset for the lan, then performance will be much less than 1 gbit.
Its not always the case, but just check it out.

So far I had no trouble with the performance of the NICs involved in this setup. But not sure how much this says since I just figured out using the NIC for PPPoE seems not the same as using it for a normal file transfer. hrhrhr

As for the router on a stick, what you describe is what is needed.

Cool thanks. Will try this in the next days.
Should I create the VLAN interfaces on the host machine and pass them to the VM or create the VLAN interfaces inside the VM
Any pros/cons for either approaches?

netblues

@soupdiver Both approaches work.
Since pf manages vlan trunks nicely its better to have everything in one place, instead of managing things at different places

Having less virtual interfaces on the host might also work better.

stephenw10

I mean I'm not a great believer in USB NICs in general. But it's speciffically in FreeBSD, and hence pfSense, where they can really give trouble.
Check when driver that is using, connect it and check the system log.
If it's using the generic cdce driver it's probably linked at 100Mbps and not reporting it. It may be possible to get a more specific driver that works a lot better.

But I would still go with a switch and VLANs over USB everu time.

Steve

soupdiver

ok, I tried out some updates to the setup and am nearly where I want to be but one last piece is missing.

I setup a VLAN for the connection from my modem to the NUC through my switch. I created the vlan interface and was able to open my ppp connection from the host system.
I added the vlan interface to the bridge where also the tap device is connected that I want to use for my pppoe connection from within pfsense.

This last piece however does not work. No error message or so, just a timeout from ppp. Meh.
I could not find much information about "creating a pppoe connection through a tap device and vlan".

I found a post from 2019:

Gave that a try and it works for a physical interface, but not over a VLAN.

Seems I have the same result.

Anything else I could try?

soupdiver

I did more testing...
I think I can confirm the statement that you can't do pppoe through a vlan/tap device.

I inverted my setup: I made all my switch ports VLAN ports and only two stay untagged. To those two I connected my NUC and the modem.
This way I could attach em0 instead vlan100 to the bridge that will be used for pppoe. pfsense instantly established the connection with full speed.

But now there is another issue: I can only reach the IP of pfsense from my NUC directly but not from other machines in my network.

I created vlan100 from em0 and assigned IP 192.168.42.2 to it. This is the IP of my NUC within my home network. Inside pfsense I set IP address 192.168.42.7. The IPs are all set correctly because from my NUC it all works fine but traffic from other machines in the network doesn't make it to pfsense. I can ping the NUC but not the VM guest and vice versa.

Always just 1 step missing...

stephenw10

Might need a diagram at this point. I'm not sure which parts are virtual there.

You absolutely can use PPPoE over a VLAN, I do it here. Also there are a lot of providers that require a VLAN to be used and if the modem doesn't add that pfSense can.

What you can't do is bridge the parent interface of a VLAN within pfSense. Doing so grabs the tagged traffic directly before and can be passed to the VLAN interface.

Steve

soupdiver

Might need a diagram at this point. I'm not sure which parts are virtual there.

Yea, it easily gets confusing when just using words, agree

You absolutely can use PPPoE over a VLAN, I do it here

Yes, I tried and this works. However what does not work is the following:

ISP -> VLAN7 -> Modem -> VLAN2 -> NUC -> em0

em0
   -> pass via bridge0 to connect LAN
   -> VLAN2 -> pass via brdige1 to do pppoe inside pfsense

My ISP uses VLAN7 for its VDSL. My Modem is configured vor VLAN7 and does remove the vlan tag.
My modem is connected to a vlan2 port on a switch.
NUC is connected via trunk port to the switch.
On the NUC I configured my em0 interface with a LAN address and also created a vlan interface which takes vlan2. Through vlan2 I want to open a PPPoE connection.

This all works fine. I can open the PPPoE connection on the NUC through the vlan device but I can't open a PPPoE connection from pfsense when I pass the vlan2 interface via bridge1.

What you can't do is bridge the parent interface of a VLAN within pfSense. Doing so grabs the tagged traffic directly before and can be passed to the VLAN interface.

Is this the same as me trying to pass the vlan2 interface via bridge1?

Uff... so confusing but thanks a lot for your effort!

soupdiver

Ok, I can finally report success

The issue was that I mixed vlans and untagged network traffic.
I thought it would be enough to only put the PPPoE connection from the Modem to the NUC in a vlan and keep the rest of my network as default/untagged. Seems this mixing and then operating on the em0 and the vlan2 interfacea causes problems. I now separated my home network into two vlans.

vlan2 => PPPoE Modem - NUC
vlan3 => the rest of my homenetwork

This way I can create vlan2 and vlan3 interfaces from em0 on the NUC directly, attach them to the corresponding bridge and start my pfsense VM. pfsense is then able to connect to my home network through vlan3 and successfully opens a PPPoE connection through vlan2.
I get full speed if my internet uplink.

I think I reached my goal.

What you can't do is bridge the parent interface of a VLAN within pfSense. Doing so grabs the tagged traffic directly before and can be passed to the VLAN interface.

That got me thinking and I tried the approach with 2 vlans.
Thanks everyone for your input!!

stephenw10

Nice! Yeah, I assumed those bridges you're talking about are in the virtual infrastructure so I'm not sure how they would behave. But I could certainly imagine that causing the same sort of problems that bridges in pfSense/FreeBSD do on interfaces with VLANs on.

Steve

soupdiver

ok, still not 100% done here.
I recognised that my upload speed is crippled. But only for network clients not the pfSense machine itself.

From NUC directly

python3.7 speedtest.py --server 2495
Retrieving speedtest.net configuration...
Testing from Deutsche Telekom AG (xxx)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 9.997 ms
Testing download speed................................................................................
Download: 182.63 Mbit/s
Testing upload speed......................................................................................................
Upload: 36.49 Mbit/s

From inside pfsense:

python3.7 speedtest.py --server 2495
Retrieving speedtest.net configuration...
Testing from Deutsche Telekom AG (xxx)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 46.646 ms
Testing download speed................................................................................
Download: 166.56 Mbit/s
Testing upload speed......................................................................................................
Upload: 27.15 Mbit/s

From network client:

speedtest-cli --server 2495
Retrieving speedtest.net configuration...
Testing from Deutsche Telekom AG (xxx)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 10.589 ms
Testing download speed................................................................................
Download: 183.40 Mbit/s
Testing upload speed................................................................................................
Upload: 2.67 Mbit/s

For me this doesn't really make sense since I get full-speed of the downlink. Any ideas? I couldn't find anything specific online but some people mentioned that it can be correlated to vlans but I'm not sure how. Since download also flows with full speed through the vlan.

All hardware offloading is disabled in Advanced => Networking and there are no (traffic shaping) firewall rules in place.

My MTU is set to 1492 and I also tried setting 1492 for MSS but this had no effect.