Solved: pfSense as bhyve guest only gives 60Mbit instead 200+

soupdiver · Dec 28, 2020, 1:59 PM

Might need a diagram at this point. I'm not sure which parts are virtual there.

Yea, it easily gets confusing when just using words, agree

You absolutely can use PPPoE over a VLAN, I do it here

Yes, I tried and this works. However what does not work is the following:

ISP -> VLAN7 -> Modem -> VLAN2 -> NUC -> em0

em0
   -> pass via bridge0 to connect LAN
   -> VLAN2 -> pass via brdige1 to do pppoe inside pfsense

My ISP uses VLAN7 for its VDSL. My Modem is configured vor VLAN7 and does remove the vlan tag.
My modem is connected to a vlan2 port on a switch.
NUC is connected via trunk port to the switch.
On the NUC I configured my em0 interface with a LAN address and also created a vlan interface which takes vlan2. Through vlan2 I want to open a PPPoE connection.

This all works fine. I can open the PPPoE connection on the NUC through the vlan device but I can't open a PPPoE connection from pfsense when I pass the vlan2 interface via bridge1.

What you can't do is bridge the parent interface of a VLAN within pfSense. Doing so grabs the tagged traffic directly before and can be passed to the VLAN interface.

Is this the same as me trying to pass the vlan2 interface via bridge1?

Uff... so confusing but thanks a lot for your effort!

soupdiver · Dec 28, 2020, 3:15 PM

Ok, I can finally report success

The issue was that I mixed vlans and untagged network traffic.
I thought it would be enough to only put the PPPoE connection from the Modem to the NUC in a vlan and keep the rest of my network as default/untagged. Seems this mixing and then operating on the em0 and the vlan2 interfacea causes problems. I now separated my home network into two vlans.

vlan2 => PPPoE Modem - NUC
vlan3 => the rest of my homenetwork

This way I can create vlan2 and vlan3 interfaces from em0 on the NUC directly, attach them to the corresponding bridge and start my pfsense VM. pfsense is then able to connect to my home network through vlan3 and successfully opens a PPPoE connection through vlan2.
I get full speed if my internet uplink.

I think I reached my goal.

What you can't do is bridge the parent interface of a VLAN within pfSense. Doing so grabs the tagged traffic directly before and can be passed to the VLAN interface.

That got me thinking and I tried the approach with 2 vlans.
Thanks everyone for your input!!

stephenw10 · Dec 28, 2020, 5:43 PM

Nice! Yeah, I assumed those bridges you're talking about are in the virtual infrastructure so I'm not sure how they would behave. But I could certainly imagine that causing the same sort of problems that bridges in pfSense/FreeBSD do on interfaces with VLANs on.

Steve

soupdiver · Dec 29, 2020, 11:03 AM

ok, still not 100% done here.
I recognised that my upload speed is crippled. But only for network clients not the pfSense machine itself.

From NUC directly

python3.7 speedtest.py --server 2495
Retrieving speedtest.net configuration...
Testing from Deutsche Telekom AG (xxx)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 9.997 ms
Testing download speed................................................................................
Download: 182.63 Mbit/s
Testing upload speed......................................................................................................
Upload: 36.49 Mbit/s

From inside pfsense:

python3.7 speedtest.py --server 2495
Retrieving speedtest.net configuration...
Testing from Deutsche Telekom AG (xxx)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 46.646 ms
Testing download speed................................................................................
Download: 166.56 Mbit/s
Testing upload speed......................................................................................................
Upload: 27.15 Mbit/s

From network client:

speedtest-cli --server 2495
Retrieving speedtest.net configuration...
Testing from Deutsche Telekom AG (xxx)...
Retrieving speedtest.net server list...
Retrieving information for the selected server...
Hosted by IBH IT-Service GmbH (Dresden) [xxx km]: 10.589 ms
Testing download speed................................................................................
Download: 183.40 Mbit/s
Testing upload speed................................................................................................
Upload: 2.67 Mbit/s

For me this doesn't really make sense since I get full-speed of the downlink. Any ideas? I couldn't find anything specific online but some people mentioned that it can be correlated to vlans but I'm not sure how. Since download also flows with full speed through the vlan.

All hardware offloading is disabled in Advanced => Networking and there are no (traffic shaping) firewall rules in place.

My MTU is set to 1492 and I also tried setting 1492 for MSS but this had no effect.

stephenw10 · Dec 29, 2020, 12:46 PM

Well at a basic level that points to the receive speed of the pfSense LAN interface.

If you previously swapped those it would have been the download to WAN that was affected, which does look to have been the case.

Steve

soupdiver · Dec 29, 2020, 12:52 PM

@stephenw10 said in pfSense as bhyve guest only gives 60Mbit instead 200+:

Well at a basic level that points to the receive speed of the pfSense LAN interface.

If you previously swapped those it would have been the download to WAN that was affected, which does look to have been the case.

Steve

Hmm not sure how to understand this.

In the very beginning I had the internal NIC and the USB dongle. I removed the USB dongle from the setup, so there is only 1 NIC involved now.

Download speed was capped 90Mbit/s previously and now I'm at 0.65 Mbit/s. Also seems very far off.

When I do the speed test from within the pfSense VM I have full-speed. So I think all the involved interfaces are ok.
The only difference seems to be when the traffic originates from outside the NUC.

soupdiver · Dec 29, 2020, 1:09 PM

Well at a basic level that points to the receive speed of the pfSense LAN interface.

Yea but only when uploading to the internet it seems. Downloads seems fine and it's the same port/interface/vlan.

I also tried to copy some files around in my network. No connection issues whatsoever. I can send/receive between my computer and the pfSense VM with full speed of 1GBit/s.

scp copy between pfSense and network client:
46% 1944MB 105.9MB/s 00:21 ETA

Summary:
client => internet: not ok
pfSense => internet: ok
NUC => internet: ok
client => pfSense: ok

Seems all individual parts are ok but something is odd when routing client traffic to the internet.

soupdiver · Dec 29, 2020, 2:26 PM

@soupdiver said in pfSense as bhyve guest only gives 60Mbit instead 200+:

Ok, I can finally report success

haha that was too quick.
It seems I still have all kind of weird issues. I hope there is a common root cause somewhere.

Issues:

slow uplink from any network client except from the NUC itself but the speed for clients varies between 0.1 Mbits/ and 6 Mbit/s. Cable/wifi seems not to matter
IPv6 issues, I get a prefix and the clients get a v6 address but incoming ICMP fails and accessing netflix via v6 also fails
"interesting" Spotify issue: One computer can access Spotify while the other can only play music that has already been started or be used as a remote device and then start playing tracks. Starting a track on the computer directly doesn't work. Another machine in the same wlan/network works without issues

As soon as I switched from my NUC/pfSense setup to my FritzBox again everything works perfectly again. So I assume it must be something with the pfSense box but I have no clue since it feels so random. I'm grateful for ideas where I can continue digging.

stephenw10 · Dec 29, 2020, 6:12 PM

Ah, I'm sorry I conflated some threads there I think!

Hmm, that level of throttling is usually something low lever like a speed/duplex mismatch or ecven bad hardware but obviously that cannot be the case in a VM.

I would think it's got to be a VLAN or MTU config problem with how that's behaving.

How are you testing from the client to pfSense dircetly? iperf? I would suggest that if not.

Steve

soupdiver · Dec 29, 2020, 6:17 PM

I would think it's got to be a VLAN or MTU config problem with how that's behaving.

I tried the usual MTUs of 1500 and 1492. When set to 1500 I saw a log that my provider set it to 1492. When I set that value the log disappeared. Sounds reasonable.

Anything special for MTU in combination with a vlan?

How are you testing from the client to pfSense dircetly? iperf? I would suggest that if not.

I did a scp file transfer and got full speed all the time.

stephenw10 · Dec 29, 2020, 10:37 PM

VLAN tags add an additional 4 bytes to the packets. Normally not an issue but if something in the path is really at 1500B and path MTU is broken it might be.
Try setting it to 1300 to be sure.

Did you test SCP both ways between the client and pfSense?

Steve

soupdiver · Dec 29, 2020, 11:54 PM

@stephenw10 said in pfSense as bhyve guest only gives 60Mbit instead 200+:

Did you test SCP both ways between the client and pfSense?

yea tried it both ways

Will test a pfSense setup natively on the NUC tomorrow and see what that brings.

soupdiver · Dec 30, 2020, 2:01 PM

Will test a pfSense setup natively on the NUC tomorrow and see what that brings.

So I installed pfSense directly on the NUC without any virtualisation and everything works as it should. Uplink, Downlink, v4, v6... all works.

While this is good... now I have my NUC occupied just by pfSense. Not really what I wanted but at least I have it working for the first time.

I still have absolutely no clue what could be issue with my crippled uplink since all the single parts of the connection seemed fine.
Gonna give it another try at some point I guess

stephenw10 · Dec 31, 2020, 2:41 PM

Mmm, some virtualisation quirk...

You might try something other than bhyve. I run Proxmox (KVM) on a NUC here, works great.

Steve

netblues · Dec 31, 2020, 4:58 AM

Pfsense on centos8 kvm on nuc here
Works also great :)

soupdiver · Jan 4, 2021, 12:49 PM

Ok, I think I can report success again

I reinstalled everything a couple of times to try out some different things. I tried a vale switch but that also just gave me 100Mbit/s in local throughput.

Afterwards I thought again and tried the basic vlan setup again that I had in the beginning because I couldn't find a clue what was wrong.
So I set it all up again...

pfSense insdie bhve vm
nuc connected to trunk port
split vlans on host and pass vlans through tap devices to vm
configure PPPoE inside pfSense

So far so good and everything was fine. I got IPv4 connectivity and full-speed.
Then I enabled IPv6 and my weird issues started again. Spotify won't play tracks and crippled upload speed. Not sure why I did not made the connection to IPv6 before.
Then I found a thread in this forum: https://forum.netgate.com/topic/72148/slow-upload-speed-test-through-ipv6-solved

I'm not 100% what "Large Segment Offload" it but disabling it solved my issue.
ifconfig em0 -lro

So yea... I might reached the end of my journey haha.

stephenw10 · Jan 4, 2021, 1:40 PM

Nice catch.