Solved: pfSense as bhyve guest only gives 60Mbit instead 200+
-
Also, your intel nuc does it actually use an intel lan card?
Is this important somehow?
Seems I come to the same conclusion. The USB NIC does not work very will in this setup.
I did some more testing:
Created a PPPoE connection from my macbook with the same usb ethernet adapter. I instantly get full speed.Then I switched my NICs on the NUC. The USB-C NIC for LAN and the internal one for WAN.
This (kinda) seems to do the trick. I now have full download speed when I download something on the NUC. However if another client on the network downloads something the limit seems to be around 100Mbit/s.
CPU load does not show anything crazy. Even with the NUC using 200Mbit/s the CPU looks fine.So it seems to come down to the USB interface somehow.
What I wonder is:
I used that same adapter before in my network (with macbook not the NUC) and never had any issues.
I did some more testing:
I transfered a file from my macbook to the NUC and there I easily get my 1GBit/s as expected for the LAN.It seems the USB NIC in general is capable to deliver the needed bandwidth but in combination with PPPoe/pfSense something is odd.
From my understanding the NIC "simply" transfers Ethernet frames over the wire... I thought there should be no difference in what kind of data it transfers but there seems to be something different between LAN file transfer, PPPoE and vnet interface inside pfSense.
Maybe it's something with the driver on FreeBSD... no idea... but it starts to feel more esoteric now somehow
Use a small managed lan switch, and do router on a stick port duplication. 5 ports model will do.
I had to look up "router on a stick". How would this look like in my scenario
Create VLAN2 (LAN) and VLAN3(WAN) on the switch.
Attach my modem to the VLAN3 port and home switch to the VLAN2 port and have a trunk port that I connect my NUC to.
Inside the NUC I then create 2 vlan devices and pass them to pfSense instead of my tap0/1 interfaces that I had before?
That's how I understood the setup so far.
Since the managed switch is also my home switch, can I just connect a cable from VLAN2 port to another "normal" port on the same switch to connect the rest of my home network?Thanks for all the input so far
-
@soupdiver If the nuc is using realtek chipset for the lan, then performance will be much less than 1 gbit.
Its not always the case, but just check it out.As for the router on a stick, what you describe is what is needed.
And obviously you can expand this to more vlans terminating to pfsnse over the trunk port. -
If the nuc is using realtek chipset for the lan, then performance will be much less than 1 gbit.
Its not always the case, but just check it out.So far I had no trouble with the performance of the NICs involved in this setup. But not sure how much this says since I just figured out using the NIC for PPPoE seems not the same as using it for a normal file transfer. hrhrhr
As for the router on a stick, what you describe is what is needed.
Cool thanks. Will try this in the next days.
Should I create the VLAN interfaces on the host machine and pass them to the VM or create the VLAN interfaces inside the VM
Any pros/cons for either approaches? -
@soupdiver Both approaches work.
Since pf manages vlan trunks nicely its better to have everything in one place, instead of managing things at different placesHaving less virtual interfaces on the host might also work better.
-
I mean I'm not a great believer in USB NICs in general.
But it's speciffically in FreeBSD, and hence pfSense, where they can really give trouble.
Check when driver that is using, connect it and check the system log.
If it's using the generic cdce driver it's probably linked at 100Mbps and not reporting it. It may be possible to get a more specific driver that works a lot better.But I would still go with a switch and VLANs over USB everu time.
Steve
-
ok, I tried out some updates to the setup and am nearly where I want to be but one last piece is missing.
I setup a VLAN for the connection from my modem to the NUC through my switch. I created the vlan interface and was able to open my ppp connection from the host system.
I added the vlan interface to the bridge where also the tap device is connected that I want to use for my pppoe connection from within pfsense.This last piece however does not work. No error message or so, just a timeout from ppp. Meh.
I could not find much information about "creating a pppoe connection through a tap device and vlan".I found a post from 2019:
Gave that a try and it works for a physical interface, but not over a VLAN.
Seems I have the same result.
Anything else I could try?
-
I did more testing...
I think I can confirm the statement that you can't do pppoe through a vlan/tap device.I inverted my setup: I made all my switch ports VLAN ports and only two stay untagged. To those two I connected my NUC and the modem.
This way I could attach em0 instead vlan100 to the bridge that will be used for pppoe. pfsense instantly established the connection with full speed.But now there is another issue: I can only reach the IP of pfsense from my NUC directly but not from other machines in my network.
I created vlan100 from em0 and assigned IP 192.168.42.2 to it. This is the IP of my NUC within my home network. Inside pfsense I set IP address 192.168.42.7. The IPs are all set correctly because from my NUC it all works fine but traffic from other machines in the network doesn't make it to pfsense. I can ping the NUC but not the VM guest and vice versa.
Always just 1 step missing...
-
Might need a diagram at this point. I'm not sure which parts are virtual there.
You absolutely can use PPPoE over a VLAN, I do it here. Also there are a lot of providers that require a VLAN to be used and if the modem doesn't add that pfSense can.
What you can't do is bridge the parent interface of a VLAN within pfSense. Doing so grabs the tagged traffic directly before and can be passed to the VLAN interface.
Steve
-
Might need a diagram at this point. I'm not sure which parts are virtual there.
Yea, it easily gets confusing when just using words, agree
You absolutely can use PPPoE over a VLAN, I do it here
Yes, I tried and this works. However what does not work is the following:
ISP -> VLAN7 -> Modem -> VLAN2 -> NUC -> em0em0 -> pass via bridge0 to connect LAN -> VLAN2 -> pass via brdige1 to do pppoe inside pfsenseMy ISP uses VLAN7 for its VDSL. My Modem is configured vor VLAN7 and does remove the vlan tag.
My modem is connected to a vlan2 port on a switch.
NUC is connected via trunk port to the switch.
On the NUC I configured my em0 interface with a LAN address and also created a vlan interface which takes vlan2. Through vlan2 I want to open a PPPoE connection.This all works fine. I can open the PPPoE connection on the NUC through the vlan device but I can't open a PPPoE connection from pfsense when I pass the vlan2 interface via bridge1.
What you can't do is bridge the parent interface of a VLAN within pfSense. Doing so grabs the tagged traffic directly before and can be passed to the VLAN interface.
Is this the same as me trying to pass the vlan2 interface via bridge1?
Uff... so confusing but thanks a lot for your effort!
-
Ok, I can finally report success
The issue was that I mixed vlans and untagged network traffic.
I thought it would be enough to only put the PPPoE connection from the Modem to the NUC in a vlan and keep the rest of my network as default/untagged. Seems this mixing and then operating on the em0 and the vlan2 interfacea causes problems. I now separated my home network into two vlans.vlan2 => PPPoE Modem - NUC
vlan3 => the rest of my homenetworkThis way I can create vlan2 and vlan3 interfaces from em0 on the NUC directly, attach them to the corresponding bridge and start my pfsense VM. pfsense is then able to connect to my home network through vlan3 and successfully opens a PPPoE connection through vlan2.
I get full speed if my internet uplink.I think I reached my goal.
What you can't do is bridge the parent interface of a VLAN within pfSense. Doing so grabs the tagged traffic directly before and can be passed to the VLAN interface.
That got me thinking and I tried the approach with 2 vlans.
Thanks everyone for your input!! -
Nice! Yeah, I assumed those bridges you're talking about are in the virtual infrastructure so I'm not sure how they would behave. But I could certainly imagine that causing the same sort of problems that bridges in pfSense/FreeBSD do on interfaces with VLANs on.
Steve
-
ok, still not 100% done here.
I recognised that my upload speed is crippled. But only for network clients not the pfSense machine itself.From NUC directly
python3.7 speedtest.py --server 2495 Retrieving speedtest.net configuration... Testing from Deutsche Telekom AG (xxx)... Retrieving speedtest.net server list... Retrieving information for the selected server... Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 9.997 ms Testing download speed................................................................................ Download: 182.63 Mbit/s Testing upload speed...................................................................................................... Upload: 36.49 Mbit/sFrom inside pfsense:
python3.7 speedtest.py --server 2495 Retrieving speedtest.net configuration... Testing from Deutsche Telekom AG (xxx)... Retrieving speedtest.net server list... Retrieving information for the selected server... Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 46.646 ms Testing download speed................................................................................ Download: 166.56 Mbit/s Testing upload speed...................................................................................................... Upload: 27.15 Mbit/sFrom network client:
speedtest-cli --server 2495 Retrieving speedtest.net configuration... Testing from Deutsche Telekom AG (xxx)... Retrieving speedtest.net server list... Retrieving information for the selected server... Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 10.589 ms Testing download speed................................................................................ Download: 183.40 Mbit/s Testing upload speed................................................................................................ Upload: 2.67 Mbit/sFor me this doesn't really make sense since I get full-speed of the downlink. Any ideas? I couldn't find anything specific online but some people mentioned that it can be correlated to vlans but I'm not sure how. Since download also flows with full speed through the vlan.
All hardware offloading is disabled in Advanced => Networking and there are no (traffic shaping) firewall rules in place.
My MTU is set to 1492 and I also tried setting 1492 for MSS but this had no effect.
-
Well at a basic level that points to the receive speed of the pfSense LAN interface.
If you previously swapped those it would have been the download to WAN that was affected, which does look to have been the case.
Steve
-
@stephenw10 said in pfSense as bhyve guest only gives 60Mbit instead 200+:
Well at a basic level that points to the receive speed of the pfSense LAN interface.
If you previously swapped those it would have been the download to WAN that was affected, which does look to have been the case.
Steve
Hmm not sure how to understand this.
In the very beginning I had the internal NIC and the USB dongle. I removed the USB dongle from the setup, so there is only 1 NIC involved now.
Download speed was capped 90Mbit/s previously and now I'm at 0.65 Mbit/s. Also seems very far off.
When I do the speed test from within the pfSense VM I have full-speed. So I think all the involved interfaces are ok.
The only difference seems to be when the traffic originates from outside the NUC. -
Well at a basic level that points to the receive speed of the pfSense LAN interface.
Yea but only when uploading to the internet it seems. Downloads seems fine and it's the same port/interface/vlan.
I also tried to copy some files around in my network. No connection issues whatsoever. I can send/receive between my computer and the pfSense VM with full speed of 1GBit/s.
scp copy between pfSense and network client:
46% 1944MB 105.9MB/s 00:21 ETASummary:
client => internet: not ok
pfSense => internet: ok
NUC => internet: ok
client => pfSense: okSeems all individual parts are ok but something is odd when routing client traffic to the internet.
-
@soupdiver said in pfSense as bhyve guest only gives 60Mbit instead 200+:
Ok, I can finally report success
haha that was too quick.
It seems I still have all kind of weird issues. I hope there is a common root cause somewhere.Issues:
- slow uplink from any network client except from the NUC itself but the speed for clients varies between 0.1 Mbits/ and 6 Mbit/s. Cable/wifi seems not to matter
- IPv6 issues, I get a prefix and the clients get a v6 address but incoming ICMP fails and accessing netflix via v6 also fails
- "interesting" Spotify issue: One computer can access Spotify while the other can only play music that has already been started or be used as a remote device and then start playing tracks. Starting a track on the computer directly doesn't work. Another machine in the same wlan/network works without issues
As soon as I switched from my NUC/pfSense setup to my FritzBox again everything works perfectly again. So I assume it must be something with the pfSense box but I have no clue since it feels so random. I'm grateful for ideas where I can continue digging.
-
Ah, I'm sorry I conflated some threads there I think!
Hmm, that level of throttling is usually something low lever like a speed/duplex mismatch or ecven bad hardware but obviously that cannot be the case in a VM.
I would think it's got to be a VLAN or MTU config problem with how that's behaving.
How are you testing from the client to pfSense dircetly? iperf? I would suggest that if not.
Steve
-
I would think it's got to be a VLAN or MTU config problem with how that's behaving.
I tried the usual MTUs of 1500 and 1492. When set to 1500 I saw a log that my provider set it to 1492. When I set that value the log disappeared. Sounds reasonable.
Anything special for MTU in combination with a vlan?
How are you testing from the client to pfSense dircetly? iperf? I would suggest that if not.
I did a scp file transfer and got full speed all the time.
-
VLAN tags add an additional 4 bytes to the packets. Normally not an issue but if something in the path is really at 1500B and path MTU is broken it might be.
Try setting it to 1300 to be sure.Did you test SCP both ways between the client and pfSense?
Steve
-
@stephenw10 said in pfSense as bhyve guest only gives 60Mbit instead 200+:
Did you test SCP both ways between the client and pfSense?
yea tried it both ways
Will test a pfSense setup natively on the NUC tomorrow and see what that brings.
