Solved: pfSense as bhyve guest only gives 60Mbit instead 200+
-
ok, still not 100% done here.
I recognised that my upload speed is crippled. But only for network clients not the pfSense machine itself.From NUC directly
python3.7 speedtest.py --server 2495 Retrieving speedtest.net configuration... Testing from Deutsche Telekom AG (xxx)... Retrieving speedtest.net server list... Retrieving information for the selected server... Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 9.997 ms Testing download speed................................................................................ Download: 182.63 Mbit/s Testing upload speed...................................................................................................... Upload: 36.49 Mbit/sFrom inside pfsense:
python3.7 speedtest.py --server 2495 Retrieving speedtest.net configuration... Testing from Deutsche Telekom AG (xxx)... Retrieving speedtest.net server list... Retrieving information for the selected server... Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 46.646 ms Testing download speed................................................................................ Download: 166.56 Mbit/s Testing upload speed...................................................................................................... Upload: 27.15 Mbit/sFrom network client:
speedtest-cli --server 2495 Retrieving speedtest.net configuration... Testing from Deutsche Telekom AG (xxx)... Retrieving speedtest.net server list... Retrieving information for the selected server... Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 10.589 ms Testing download speed................................................................................ Download: 183.40 Mbit/s Testing upload speed................................................................................................ Upload: 2.67 Mbit/sFor me this doesn't really make sense since I get full-speed of the downlink. Any ideas? I couldn't find anything specific online but some people mentioned that it can be correlated to vlans but I'm not sure how. Since download also flows with full speed through the vlan.
All hardware offloading is disabled in Advanced => Networking and there are no (traffic shaping) firewall rules in place.
My MTU is set to 1492 and I also tried setting 1492 for MSS but this had no effect.
-
Well at a basic level that points to the receive speed of the pfSense LAN interface.
If you previously swapped those it would have been the download to WAN that was affected, which does look to have been the case.
Steve
-
@stephenw10 said in pfSense as bhyve guest only gives 60Mbit instead 200+:
Well at a basic level that points to the receive speed of the pfSense LAN interface.
If you previously swapped those it would have been the download to WAN that was affected, which does look to have been the case.
Steve
Hmm not sure how to understand this.
In the very beginning I had the internal NIC and the USB dongle. I removed the USB dongle from the setup, so there is only 1 NIC involved now.
Download speed was capped 90Mbit/s previously and now I'm at 0.65 Mbit/s. Also seems very far off.
When I do the speed test from within the pfSense VM I have full-speed. So I think all the involved interfaces are ok.
The only difference seems to be when the traffic originates from outside the NUC. -
Well at a basic level that points to the receive speed of the pfSense LAN interface.
Yea but only when uploading to the internet it seems. Downloads seems fine and it's the same port/interface/vlan.
I also tried to copy some files around in my network. No connection issues whatsoever. I can send/receive between my computer and the pfSense VM with full speed of 1GBit/s.
scp copy between pfSense and network client:
46% 1944MB 105.9MB/s 00:21 ETASummary:
client => internet: not ok
pfSense => internet: ok
NUC => internet: ok
client => pfSense: okSeems all individual parts are ok but something is odd when routing client traffic to the internet.
-
@soupdiver said in pfSense as bhyve guest only gives 60Mbit instead 200+:
Ok, I can finally report success
haha that was too quick.
It seems I still have all kind of weird issues. I hope there is a common root cause somewhere.Issues:
- slow uplink from any network client except from the NUC itself but the speed for clients varies between 0.1 Mbits/ and 6 Mbit/s. Cable/wifi seems not to matter
- IPv6 issues, I get a prefix and the clients get a v6 address but incoming ICMP fails and accessing netflix via v6 also fails
- "interesting" Spotify issue: One computer can access Spotify while the other can only play music that has already been started or be used as a remote device and then start playing tracks. Starting a track on the computer directly doesn't work. Another machine in the same wlan/network works without issues
As soon as I switched from my NUC/pfSense setup to my FritzBox again everything works perfectly again. So I assume it must be something with the pfSense box but I have no clue since it feels so random. I'm grateful for ideas where I can continue digging.
-
Ah, I'm sorry I conflated some threads there I think!
Hmm, that level of throttling is usually something low lever like a speed/duplex mismatch or ecven bad hardware but obviously that cannot be the case in a VM.
I would think it's got to be a VLAN or MTU config problem with how that's behaving.
How are you testing from the client to pfSense dircetly? iperf? I would suggest that if not.
Steve
-
I would think it's got to be a VLAN or MTU config problem with how that's behaving.
I tried the usual MTUs of 1500 and 1492. When set to 1500 I saw a log that my provider set it to 1492. When I set that value the log disappeared. Sounds reasonable.
Anything special for MTU in combination with a vlan?
How are you testing from the client to pfSense dircetly? iperf? I would suggest that if not.
I did a scp file transfer and got full speed all the time.
-
VLAN tags add an additional 4 bytes to the packets. Normally not an issue but if something in the path is really at 1500B and path MTU is broken it might be.
Try setting it to 1300 to be sure.Did you test SCP both ways between the client and pfSense?
Steve
-
@stephenw10 said in pfSense as bhyve guest only gives 60Mbit instead 200+:
Did you test SCP both ways between the client and pfSense?
yea tried it both ways
Will test a pfSense setup natively on the NUC tomorrow and see what that brings.
-
Will test a pfSense setup natively on the NUC tomorrow and see what that brings.
So I installed pfSense directly on the NUC without any virtualisation and everything works as it should.
Uplink, Downlink, v4, v6... all works.While this is good... now I have my NUC occupied just by pfSense. Not really what I wanted but at least I have it working for the first time.
I still have absolutely no clue what could be issue with my crippled uplink since all the single parts of the connection seemed fine.
Gonna give it another try at some point I guess -
Mmm, some virtualisation quirk...
You might try something other than bhyve. I run Proxmox (KVM) on a NUC here, works great.
Steve
-
Pfsense on centos8 kvm on nuc here
Works also great :) -
Ok, I think I can report success again
I reinstalled everything a couple of times to try out some different things. I tried a vale switch but that also just gave me 100Mbit/s in local throughput.
Afterwards I thought again and tried the basic vlan setup again that I had in the beginning because I couldn't find a clue what was wrong.
So I set it all up again...- pfSense insdie bhve vm
- nuc connected to trunk port
- split vlans on host and pass vlans through tap devices to vm
- configure PPPoE inside pfSense
So far so good and everything was fine. I got IPv4 connectivity and full-speed.
Then I enabled IPv6 and my weird issues started again. Spotify won't play tracks and crippled upload speed. Not sure why I did not made the connection to IPv6 before.
Then I found a thread in this forum: https://forum.netgate.com/topic/72148/slow-upload-speed-test-through-ipv6-solvedI'm not 100% what "Large Segment Offload" it but disabling it solved my issue.
ifconfig em0 -lroSo yea... I might reached the end of my journey haha.
-
Nice catch.
