Solved: pfSense as bhyve guest only gives 60Mbit instead 200+

soupdiver

@soupdiver said in pfSense as bhyve guest only gives 60Mbit instead 200+:

Ok, I can finally report success

haha that was too quick.
It seems I still have all kind of weird issues. I hope there is a common root cause somewhere.

Issues:

slow uplink from any network client except from the NUC itself but the speed for clients varies between 0.1 Mbits/ and 6 Mbit/s. Cable/wifi seems not to matter
IPv6 issues, I get a prefix and the clients get a v6 address but incoming ICMP fails and accessing netflix via v6 also fails
"interesting" Spotify issue: One computer can access Spotify while the other can only play music that has already been started or be used as a remote device and then start playing tracks. Starting a track on the computer directly doesn't work. Another machine in the same wlan/network works without issues

As soon as I switched from my NUC/pfSense setup to my FritzBox again everything works perfectly again. So I assume it must be something with the pfSense box but I have no clue since it feels so random. I'm grateful for ideas where I can continue digging.

stephenw10

Ah, I'm sorry I conflated some threads there I think!

Hmm, that level of throttling is usually something low lever like a speed/duplex mismatch or ecven bad hardware but obviously that cannot be the case in a VM.

I would think it's got to be a VLAN or MTU config problem with how that's behaving.

How are you testing from the client to pfSense dircetly? iperf? I would suggest that if not.

Steve

soupdiver

I would think it's got to be a VLAN or MTU config problem with how that's behaving.

I tried the usual MTUs of 1500 and 1492. When set to 1500 I saw a log that my provider set it to 1492. When I set that value the log disappeared. Sounds reasonable.

Anything special for MTU in combination with a vlan?

How are you testing from the client to pfSense dircetly? iperf? I would suggest that if not.

I did a scp file transfer and got full speed all the time.

stephenw10

VLAN tags add an additional 4 bytes to the packets. Normally not an issue but if something in the path is really at 1500B and path MTU is broken it might be.
Try setting it to 1300 to be sure.

Did you test SCP both ways between the client and pfSense?

Steve

soupdiver

@stephenw10 said in pfSense as bhyve guest only gives 60Mbit instead 200+:

Did you test SCP both ways between the client and pfSense?

yea tried it both ways

Will test a pfSense setup natively on the NUC tomorrow and see what that brings.

soupdiver

Will test a pfSense setup natively on the NUC tomorrow and see what that brings.

So I installed pfSense directly on the NUC without any virtualisation and everything works as it should. Uplink, Downlink, v4, v6... all works.

While this is good... now I have my NUC occupied just by pfSense. Not really what I wanted but at least I have it working for the first time.

I still have absolutely no clue what could be issue with my crippled uplink since all the single parts of the connection seemed fine.
Gonna give it another try at some point I guess

stephenw10

Mmm, some virtualisation quirk...

You might try something other than bhyve. I run Proxmox (KVM) on a NUC here, works great.

Steve

netblues

Pfsense on centos8 kvm on nuc here
Works also great :)

soupdiver

Ok, I think I can report success again

I reinstalled everything a couple of times to try out some different things. I tried a vale switch but that also just gave me 100Mbit/s in local throughput.

Afterwards I thought again and tried the basic vlan setup again that I had in the beginning because I couldn't find a clue what was wrong.
So I set it all up again...

pfSense insdie bhve vm
nuc connected to trunk port
split vlans on host and pass vlans through tap devices to vm
configure PPPoE inside pfSense

So far so good and everything was fine. I got IPv4 connectivity and full-speed.
Then I enabled IPv6 and my weird issues started again. Spotify won't play tracks and crippled upload speed. Not sure why I did not made the connection to IPv6 before.
Then I found a thread in this forum: https://forum.netgate.com/topic/72148/slow-upload-speed-test-through-ipv6-solved

I'm not 100% what "Large Segment Offload" it but disabling it solved my issue.
ifconfig em0 -lro

So yea... I might reached the end of my journey haha.

stephenw10

Nice catch.