10 Gbps issue with pfSense on Proxmox

Nyxtorm

Hello,

I'm currently trying to understand a strange behavior with my pfSense CE 2.6 virtualized on Proxmox VE 7.4-3. I've been tearing my hair out for days now.

My ISP router (Freebox Delta w/ 10G-EPON, Free ISP in France) is wired directly to my Proxmox hypervisor via a passive 10Gbps DAC.

So I have a Linux bridge (vmbr2) connected to the corresponding SFP+ port.

Attached to this bridge are my pfSense WAN and two test VMs. For each of my tests, I download directly from my ISP's router, which is capable of generating data on the fly to test local speeds.

I put my tests in Pastebin because I'm marked as a spammer: https://pastebin.com/raw/qxAUGynQ

What I tried:

Test all possibilities with Hardware Checksum Offloading, TSO and LRO
Test with or without PCIe passthrough on pfSense, maybe a small difference but really not a big deal
Reinstall clean pfSense (test with CE 2.6, Plus 23.01 and Plus 23.05 versions)
Test of E1000, Realtek and VMX drivers
Test with multiqueue (4 or 8) with 8 vCPU, no difference and I can reach 10Gbps with only 2 vCPU and without multiqueue without any problem.
Test with OPNsense, throughputs significantly higher, but no major difference.
Test with i440fx and q35 machine
Test with Jumbo frames (MTU 9000)

Proxmox handles the 10Gbps link very well, since I can reach them without any problem as soon as I stop going through pfSense. I've also run a number of speedtest tests, which show the same behavior towards external servers: speeds seem to be "throttled" as soon as I go through my clean pfSense virtual machine (clean installation with no modifications).

I thought it was a FreeBSD limitation, but a FreeBSD installation perfectly exploits the 10Gbps link.

So I'm a bit lost, I'm throwing a bottle into the sea in case someone has the same experience and, more importantly, the solution.

My config :

ISP :
- Freebox Delta 10G-EPON 8Gbps/700Mbps
Hypervisor :
- Proxmox VE 7.4-3
- AMD Ryzen 7 5700G
- 64 GB RAM
- Motherboard MAG B550M MORTAR WIFI
- 1 To NVMe
- 2 x SFP+ 1/2.5/10Gbps (BCM57810S)
- 1 x RJ45 2.5 Gbps (RTL8125B, not used)

A brief overview of the network part concerned : pfsense_issue

Sorry for my English and thanks for your help ! :)

Nyxtorm

New tests :

Test new VM Untangle NG Firewall (Arista) : I'm able to use 10Gbps, including through the router, but... there is a bug if IPv6 gateway is on fe80::/10, no default route created)
Test new VM pfSense with OS set to "Other" and UEFI, same problematic behavior

Cool_Corona

Plug your pfsense WAN directly into the ISP modem LAN and bridge the modem.

Test again.

NollipfSense

@Nyxtorm Agree with Cool_Corona...please use this guide and make sure WAN passthrough to pfSense: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html

MikeFromOz

I can't find my cheat-sheet but this should do:

Procedure to bridge out the Freebox Delta: https://assistance.free.fr/articles/passer-votre-freebox-en-mode-bridge-utilisation-avancee-747
Procedure including pfsense: https://www.osnet.eu/en/node/752

Please be aware that:
-Free doesn't guarantee you'll get the advertised speed... I generally maxed out at 8.5GB, requiring ~3 simultaneous downloads to get there (individual streams maxed at 4.3GB from memory). I had a suspicion that I could get better single thread downloads with a CPU with higher single thread performance but I never had the hardware to test.
-Free shares bandwidth between subscribers e.g. it might be 10GB to your building, not your living room
-Internet speeds routinely change through the day with Free
-Ping times are much higher than with Orange (which made it feel slower than 1GB with Orange)

Nyxtorm

@MikeFromOz,

My problem wasn't a problem with global speed, but a huge difference in upload speed between the SFP+ connector and the RJ45 connector, a very abnormal behavior.
My problem was solved by simply changing ISP after weeks of waiting for answers from Free. I deliberately left dev tickets open, I never got any answer from Free, other than a high level of support who informed me that they were not trained at all regarding this very little used SFP+ LAN port, and the vast majority of customers use the classic RJ45 port, and some of those who use the SFP+ port don't make precise measurements. Since my first post, I've done a lot of tests and ended up with a direct single-mode fiber cable with two 10GBase-LR LC modules, no change, always inconsistent data displayed by FreeboxOS, and as soon as the SFP+ link was negotiated to 10Gbps, the problems started.

Free wasn't interested in looking for a solution, so I changed ISPs.

Sorry for my English.

MikeFromOz

@Nyxtorm

Absolutely no issues on language... you're perfectly understandable.

I was running... Freebox Delta SFP+ ----- DAC ---- SFP+ Chelsio NIC --- PFSense---- SFP+ Chelsio NIC ----- DAC ---- Mikrotik Switch

It was solid at 10Gb/s although (the act of) plugging the SFP+ connector into the Freebox would cause it to reboot. (not enough power for the connector?). I saw the same behaviour when plugging my NAS into the SFP+ port on the freebox via a DAC.

Free's support was poo but the developers of the freebox did put out some great material. There were also some good forum posts.

If I was going to pick another FR ISP I would go with Orange 1 giga. I didn't like using PPPOE but it worked. They were offering 2.5 giga but you couldn't get rid of the livebox (which is total poo) like you could with 1 giga.

Nyxtorm

Hello @MikeFromOz,

Indeed, I went to Orange with the Max 2 Gbps/800 Mbps offer, which is extremely stable, in my area anyway.

I use the 2.5 Gbps port of the Livebox 6 with pfSense to benefit from the 2 Gbps, even if I have a ready installation with an ONU SFP GPON 2.5 Gbps module because I sometimes had problems with the link re-establishing during a physical disconnection test. I totally agree that Free's hardware is very good and much more "flexible".