Solved: pfSense as bhyve guest only gives 60Mbit instead 200+

soupdiver

Ok, I setup my PPPoE connection directly on the host using native ppp.
Connection worked instantly (what a surprise) but the speed is still not great. I get around 100Mbit/s now. A bit better but still far from what it should be.

What else could it be?

stephenw10

PPPoE is limited in pfSebse/FreeBSD by the the fact it can only use as single NIC queue:
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

But unless your CPU is running at 10th normal speed it should easily pass more than 100Mbps.

Is the NIC actually linked at 1G?

Do you see any errors or collisions on it in Status > Interfaces? You will have to assign the parent NIC to see it there.

Steve

soupdiver

@stephenw10 yea it's a usb-c dongle that I used on my macbook.
Speed was fine there.
So the nic should be ok.

Yea, the CPU is an i7... not close to see a core at very high load.
On the hast it was not pfsense. vanilla FreeBSD with pppd.

Collisions and errors are 0 but maybe that's because I passed a tap device to the VM and not the real device.

I will see if I can test with a linux laptop and see if the result is different. If not maybe the modem is misconfigured... what I kind of doubt but who knows.

stephenw10

USB NICs are notoriously bad in pfSense/FreeBSD. If you can test with a different NIC type I would. Does it actually show that as linked at 1G? Some USB NICs do not report a link speed if they default to the cdce driver.

Steve

netblues

Also, your intel nuc does it actually use an intel lan card?
Checkit out.
Forget about using a usb nic on pfsense.
Even if it works, if it is disconnected, it will kick in the interface reconfiguration script.

Use a small managed lan switch, and do router on a stick port duplication. 5 ports model will do.

soupdiver

Also, your intel nuc does it actually use an intel lan card?

Is this important somehow?

Seems I come to the same conclusion. The USB NIC does not work very will in this setup.

I did some more testing:
Created a PPPoE connection from my macbook with the same usb ethernet adapter. I instantly get full speed.

Then I switched my NICs on the NUC. The USB-C NIC for LAN and the internal one for WAN.
This (kinda) seems to do the trick. I now have full download speed when I download something on the NUC. However if another client on the network downloads something the limit seems to be around 100Mbit/s.
CPU load does not show anything crazy. Even with the NUC using 200Mbit/s the CPU looks fine.

So it seems to come down to the USB interface somehow.

What I wonder is:
I used that same adapter before in my network (with macbook not the NUC) and never had any issues.
I did some more testing:
I transfered a file from my macbook to the NUC and there I easily get my 1GBit/s as expected for the LAN.

It seems the USB NIC in general is capable to deliver the needed bandwidth but in combination with PPPoe/pfSense something is odd.

From my understanding the NIC "simply" transfers Ethernet frames over the wire... I thought there should be no difference in what kind of data it transfers but there seems to be something different between LAN file transfer, PPPoE and vnet interface inside pfSense.
Maybe it's something with the driver on FreeBSD... no idea... but it starts to feel more esoteric now somehow

Use a small managed lan switch, and do router on a stick port duplication. 5 ports model will do.

I had to look up "router on a stick". How would this look like in my scenario
Create VLAN2 (LAN) and VLAN3(WAN) on the switch.
Attach my modem to the VLAN3 port and home switch to the VLAN2 port and have a trunk port that I connect my NUC to.
Inside the NUC I then create 2 vlan devices and pass them to pfSense instead of my tap0/1 interfaces that I had before?
That's how I understood the setup so far.
Since the managed switch is also my home switch, can I just connect a cable from VLAN2 port to another "normal" port on the same switch to connect the rest of my home network?

Thanks for all the input so far

netblues

@soupdiver If the nuc is using realtek chipset for the lan, then performance will be much less than 1 gbit.
Its not always the case, but just check it out.

As for the router on a stick, what you describe is what is needed.
And obviously you can expand this to more vlans terminating to pfsnse over the trunk port.

soupdiver

@netblues

If the nuc is using realtek chipset for the lan, then performance will be much less than 1 gbit.
Its not always the case, but just check it out.

So far I had no trouble with the performance of the NICs involved in this setup. But not sure how much this says since I just figured out using the NIC for PPPoE seems not the same as using it for a normal file transfer. hrhrhr

As for the router on a stick, what you describe is what is needed.

Cool thanks. Will try this in the next days.
Should I create the VLAN interfaces on the host machine and pass them to the VM or create the VLAN interfaces inside the VM
Any pros/cons for either approaches?

netblues

@soupdiver Both approaches work.
Since pf manages vlan trunks nicely its better to have everything in one place, instead of managing things at different places

Having less virtual interfaces on the host might also work better.

stephenw10

I mean I'm not a great believer in USB NICs in general. But it's speciffically in FreeBSD, and hence pfSense, where they can really give trouble.
Check when driver that is using, connect it and check the system log.
If it's using the generic cdce driver it's probably linked at 100Mbps and not reporting it. It may be possible to get a more specific driver that works a lot better.

But I would still go with a switch and VLANs over USB everu time.

Steve

soupdiver

ok, I tried out some updates to the setup and am nearly where I want to be but one last piece is missing.

I setup a VLAN for the connection from my modem to the NUC through my switch. I created the vlan interface and was able to open my ppp connection from the host system.
I added the vlan interface to the bridge where also the tap device is connected that I want to use for my pppoe connection from within pfsense.

This last piece however does not work. No error message or so, just a timeout from ppp. Meh.
I could not find much information about "creating a pppoe connection through a tap device and vlan".

I found a post from 2019:

Gave that a try and it works for a physical interface, but not over a VLAN.

Seems I have the same result.

Anything else I could try?

soupdiver

I did more testing...
I think I can confirm the statement that you can't do pppoe through a vlan/tap device.

I inverted my setup: I made all my switch ports VLAN ports and only two stay untagged. To those two I connected my NUC and the modem.
This way I could attach em0 instead vlan100 to the bridge that will be used for pppoe. pfsense instantly established the connection with full speed.

But now there is another issue: I can only reach the IP of pfsense from my NUC directly but not from other machines in my network.

I created vlan100 from em0 and assigned IP 192.168.42.2 to it. This is the IP of my NUC within my home network. Inside pfsense I set IP address 192.168.42.7. The IPs are all set correctly because from my NUC it all works fine but traffic from other machines in the network doesn't make it to pfsense. I can ping the NUC but not the VM guest and vice versa.

Always just 1 step missing...

stephenw10

Might need a diagram at this point. I'm not sure which parts are virtual there.

You absolutely can use PPPoE over a VLAN, I do it here. Also there are a lot of providers that require a VLAN to be used and if the modem doesn't add that pfSense can.

What you can't do is bridge the parent interface of a VLAN within pfSense. Doing so grabs the tagged traffic directly before and can be passed to the VLAN interface.

Steve

soupdiver

Might need a diagram at this point. I'm not sure which parts are virtual there.

Yea, it easily gets confusing when just using words, agree

You absolutely can use PPPoE over a VLAN, I do it here

Yes, I tried and this works. However what does not work is the following:

ISP -> VLAN7 -> Modem -> VLAN2 -> NUC -> em0

em0
   -> pass via bridge0 to connect LAN
   -> VLAN2 -> pass via brdige1 to do pppoe inside pfsense

My ISP uses VLAN7 for its VDSL. My Modem is configured vor VLAN7 and does remove the vlan tag.
My modem is connected to a vlan2 port on a switch.
NUC is connected via trunk port to the switch.
On the NUC I configured my em0 interface with a LAN address and also created a vlan interface which takes vlan2. Through vlan2 I want to open a PPPoE connection.

This all works fine. I can open the PPPoE connection on the NUC through the vlan device but I can't open a PPPoE connection from pfsense when I pass the vlan2 interface via bridge1.

What you can't do is bridge the parent interface of a VLAN within pfSense. Doing so grabs the tagged traffic directly before and can be passed to the VLAN interface.

Is this the same as me trying to pass the vlan2 interface via bridge1?

Uff... so confusing but thanks a lot for your effort!

soupdiver

Ok, I can finally report success

The issue was that I mixed vlans and untagged network traffic.
I thought it would be enough to only put the PPPoE connection from the Modem to the NUC in a vlan and keep the rest of my network as default/untagged. Seems this mixing and then operating on the em0 and the vlan2 interfacea causes problems. I now separated my home network into two vlans.

vlan2 => PPPoE Modem - NUC
vlan3 => the rest of my homenetwork

This way I can create vlan2 and vlan3 interfaces from em0 on the NUC directly, attach them to the corresponding bridge and start my pfsense VM. pfsense is then able to connect to my home network through vlan3 and successfully opens a PPPoE connection through vlan2.
I get full speed if my internet uplink.

I think I reached my goal.

What you can't do is bridge the parent interface of a VLAN within pfSense. Doing so grabs the tagged traffic directly before and can be passed to the VLAN interface.

That got me thinking and I tried the approach with 2 vlans.
Thanks everyone for your input!!

stephenw10

Nice! Yeah, I assumed those bridges you're talking about are in the virtual infrastructure so I'm not sure how they would behave. But I could certainly imagine that causing the same sort of problems that bridges in pfSense/FreeBSD do on interfaces with VLANs on.

Steve

soupdiver

ok, still not 100% done here.
I recognised that my upload speed is crippled. But only for network clients not the pfSense machine itself.

From NUC directly

python3.7 speedtest.py --server 2495
Retrieving speedtest.net configuration...
Testing from Deutsche Telekom AG (xxx)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 9.997 ms
Testing download speed................................................................................
Download: 182.63 Mbit/s
Testing upload speed......................................................................................................
Upload: 36.49 Mbit/s

From inside pfsense:

python3.7 speedtest.py --server 2495
Retrieving speedtest.net configuration...
Testing from Deutsche Telekom AG (xxx)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 46.646 ms
Testing download speed................................................................................
Download: 166.56 Mbit/s
Testing upload speed......................................................................................................
Upload: 27.15 Mbit/s

From network client:

speedtest-cli --server 2495
Retrieving speedtest.net configuration...
Testing from Deutsche Telekom AG (xxx)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 10.589 ms
Testing download speed................................................................................
Download: 183.40 Mbit/s
Testing upload speed................................................................................................
Upload: 2.67 Mbit/s

For me this doesn't really make sense since I get full-speed of the downlink. Any ideas? I couldn't find anything specific online but some people mentioned that it can be correlated to vlans but I'm not sure how. Since download also flows with full speed through the vlan.

All hardware offloading is disabled in Advanced => Networking and there are no (traffic shaping) firewall rules in place.

My MTU is set to 1492 and I also tried setting 1492 for MSS but this had no effect.

stephenw10

Well at a basic level that points to the receive speed of the pfSense LAN interface.

If you previously swapped those it would have been the download to WAN that was affected, which does look to have been the case.

Steve

soupdiver

@stephenw10 said in pfSense as bhyve guest only gives 60Mbit instead 200+:

Well at a basic level that points to the receive speed of the pfSense LAN interface.

If you previously swapped those it would have been the download to WAN that was affected, which does look to have been the case.

Steve

Hmm not sure how to understand this.

In the very beginning I had the internal NIC and the USB dongle. I removed the USB dongle from the setup, so there is only 1 NIC involved now.

Download speed was capped 90Mbit/s previously and now I'm at 0.65 Mbit/s. Also seems very far off.

When I do the speed test from within the pfSense VM I have full-speed. So I think all the involved interfaces are ok.
The only difference seems to be when the traffic originates from outside the NUC.

soupdiver

Well at a basic level that points to the receive speed of the pfSense LAN interface.

Yea but only when uploading to the internet it seems. Downloads seems fine and it's the same port/interface/vlan.

I also tried to copy some files around in my network. No connection issues whatsoever. I can send/receive between my computer and the pfSense VM with full speed of 1GBit/s.

scp copy between pfSense and network client:
46% 1944MB 105.9MB/s 00:21 ETA

Summary:
client => internet: not ok
pfSense => internet: ok
NUC => internet: ok
client => pfSense: ok

Seems all individual parts are ok but something is odd when routing client traffic to the internet.