10 Gbps issue with pfSense on Proxmox

[Nyxtorm]

Hello,

I'm currently trying to understand a strange behavior with my pfSense CE 2.6 virtualized on Proxmox VE 7.4-3. I've been tearing my hair out for days now.

My ISP router (Freebox Delta w/ 10G-EPON, Free ISP in France) is wired directly to my Proxmox hypervisor via a passive 10Gbps DAC.

So I have a Linux bridge (vmbr2) connected to the corresponding SFP+ port.

Attached to this bridge are my pfSense WAN and two test VMs. For each of my tests, I download directly from my ISP's router, which is capable of generating data on the fly to test local speeds.

I put my tests in Pastebin because I'm marked as a spammer: https://pastebin.com/raw/qxAUGynQ

What I tried:

• Test all possibilities with Hardware Checksum Offloading, TSO and LRO
• Test with or without PCIe passthrough on pfSense, maybe a small difference but really not a big deal
• Reinstall clean pfSense (test with CE 2.6, Plus 23.01 and Plus 23.05 versions)
• Test of E1000, Realtek and VMX drivers
• Test with multiqueue (4 or 8) with 8 vCPU, no difference and I can reach 10Gbps with only 2 vCPU and without multiqueue without any problem.
• Test with OPNsense, throughputs significantly higher, but no major difference.
• Test with i440fx and q35 machine
• Test with Jumbo frames (MTU 9000)

Proxmox handles the 10Gbps link very well, since I can reach them without any problem as soon as I stop going through pfSense. I've also run a number of speedtest tests, which show the same behavior towards external servers: speeds seem to be "throttled" as soon as I go through my clean pfSense virtual machine (clean installation with no modifications).

I thought it was a FreeBSD limitation, but a FreeBSD installation perfectly exploits the 10Gbps link.

So I'm a bit lost, I'm throwing a bottle into the sea in case someone has the same experience and, more importantly, the solution.

My config :

• ISP :
  • Freebox Delta 10G-EPON 8Gbps/700Mbps
• Hypervisor :
  • Proxmox VE 7.4-3
  • AMD Ryzen 7 5700G
  • 64 GB RAM
  • Motherboard MAG B550M MORTAR WIFI
  • 1 To NVMe
  • 2 x SFP+ 1/2.5/10Gbps (BCM57810S)
  • 1 x RJ45 2.5 Gbps (RTL8125B, not used)

A brief overview of the network part concerned :

Sorry for my English and thanks for your help ! :)

[Nyxtorm]

New tests :

• Test new VM Untangle NG Firewall (Arista) : I'm able to use 10Gbps, including through the router, but... there is a bug if IPv6 gateway is on fe80::/10, no default route created)
• Test new VM pfSense with OS set to "Other" and UEFI, same problematic behavior

[Cool_Corona]

Plug your pfsense WAN directly into the ISP modem LAN and bridge the modem.

Test again.

[NollipfSense]

@Nyxtorm Agree with Cool_Corona...please use this guide and make sure WAN passthrough to pfSense: https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-proxmox-ve.html

[MikeFromOz]

I can't find my cheat-sheet but this should do:

Procedure to bridge out the Freebox Delta: https://assistance.free.fr/articles/passer-votre-freebox-en-mode-bridge-utilisation-avancee-747
Procedure including pfsense: https://www.osnet.eu/en/node/752

Please be aware that:
-Free doesn't guarantee you'll get the advertised speed... I generally maxed out at 8.5GB, requiring ~3 simultaneous downloads to get there (individual streams maxed at 4.3GB from memory). I had a suspicion that I could get better single thread downloads with a CPU with higher single thread performance but I never had the hardware to test.
-Free shares bandwidth between subscribers e.g. it might be 10GB to your building, not your living room
-Internet speeds routinely change through the day with Free
-Ping times are much higher than with Orange (which made it feel slower than 1GB with Orange)

[Nyxtorm]

@MikeFromOz,

My problem wasn't a problem with global speed, but a huge difference in upload speed between the SFP+ connector and the RJ45 connector, a very abnormal behavior.
My problem was solved by simply changing ISP after weeks of waiting for answers from Free. I deliberately left dev tickets open, I never got any answer from Free, other than a high level of support who informed me that they were not trained at all regarding this very little used SFP+ LAN port, and the vast majority of customers use the classic RJ45 port, and some of those who use the SFP+ port don't make precise measurements. Since my first post, I've done a lot of tests and ended up with a direct single-mode fiber cable with two 10GBase-LR LC modules, no change, always inconsistent data displayed by FreeboxOS, and as soon as the SFP+ link was negotiated to 10Gbps, the problems started.

Free wasn't interested in looking for a solution, so I changed ISPs.

Sorry for my English.

[MikeFromOz]

@Nyxtorm

Absolutely no issues on language... you're perfectly understandable.

I was running... Freebox Delta SFP+ ----- DAC ---- SFP+ Chelsio NIC --- PFSense---- SFP+ Chelsio NIC ----- DAC ---- Mikrotik Switch

It was solid at 10Gb/s although (the act of) plugging the SFP+ connector into the Freebox would cause it to reboot. (not enough power for the connector?). I saw the same behaviour when plugging my NAS into the SFP+ port on the freebox via a DAC.

Free's support was poo but the developers of the freebox did put out some great material. There were also some good forum posts.

If I was going to pick another FR ISP I would go with Orange 1 giga. I didn't like using PPPOE but it worked. They were offering 2.5 giga but you couldn't get rid of the livebox (which is total poo) like you could with 1 giga.

[Nyxtorm]

Hello @MikeFromOz,

Indeed, I went to Orange with the Max 2 Gbps/800 Mbps offer, which is extremely stable, in my area anyway.

I use the 2.5 Gbps port of the Livebox 6 with pfSense to benefit from the 2 Gbps, even if I have a ready installation with an ONU SFP GPON 2.5 Gbps module because I sometimes had problems with the link re-establishing during a physical disconnection test. I totally agree that Free's hardware is very good and much more "flexible".