Extreme slow internet speed pfsense over proxmox

[Derelict]

@skogs said in Extreme slow internet speed pfsense over proxmox:

Sorry if I sound like a putz but you did try checking the boxes for all three of these right?:
Disable hardware checksum offload
Disable hardware TCP segmentation offload
Disable hardware large receive offload

Almost certainly this...

[skogs]

lol

[abhishekakt]

I already tried these.. no help. Something is not right. when I run iperf on vm connected to same vswitch with pfsense, it never exceeds 300 Mbps and goes does as 80 mbps. It is slower than my physical LAN

[digdug3]

1. Use virtio
2. Use the checkboxes as stated above and reboot VM after setting them
3. Disabled "firewall" in the VM in Proxmox
4. Use 2.4.4-p3 for now. 2.4.5 has issues

[skogs]

So for giggles I did do some extra testing last week. Re-did the pfsense on a tiny physical machine again. Pretty much no matter what I did the physical instance drastically outperformed the virtual.

When it was virtualized I was also mirroring off the traffic to a third virtual NIC for monitoring on the inside of the firewall anything going out. In the physical environment this is taken care of by a physical switch. Turning this off did improve my data processing somewhat, but not entirely. Lets face it; a virtual network adaptor resides in memory and shouldn't really slow things down much.

I don't have half the network bandwidth you guys seem to have.
Physical: averaging 65-82Mbps on what is advertised as a 100M line.
Virtualized: depending on settings anywhere from 30Mbps to 45Mbps.

Physical is using little Dual Core Celeron N3160 at 1.6Ghz
Virtualized is using little Quad Core Ryzen 1200 at 3Ghz
The ryzen would generally outperform the Celeron a bit less than 3x with a vastly better single process speed and multi process. Kind of odd. I've never seen this large of a performance hit on Proxmox before.

[digdug3]

I'm getting 100/100 on proxmox, the same as I'm getting when connected physical.
What NICs are you using. I only use Intel based ones for Proxmox.
Also make sure Proxmox is updated.

[skogs]

Confirm the proxmox is using crappy Realtek RTL8111/8168/8411 style.
So is the bare metal.

[dogtreatfairy]

So I was in the same boat, and couldn't figure this out. I was about ready to buy another dedicated computer just to be my pfSense box. I'm running pfSense 2.5 inside Proxmox 6.3-3. I know it sounds stupid, but try upping your pfSense VM RAM to 6GB (6144MB). This tripled my connection speed and I'm back to full speed on my WAN. Inside pfSense, it was showing my RAM usage as ~20% at 4GB, so it makes no sense to me why this fixed my problem.

I also disabled the hardware checksum, tcp, and large receive offload options under System > Advanced > Networking.

My VM Setup

• RAM: 6.00 GB
• CPU: Host (i5-3470), 1 Socket, 4 Cores, +pcid,+aes, +ssbd
• BIOS: Default (SeaBIOS)
• Machine: Default (i440fx)
• SCSI Cont: VirtIO SCSI
• HDD: SCSI, 32GB volume.
• PCI Device: PCI pass through for my HP N364T 4-port gigabit NIC off ebay

Look through this to enable IOMMU for the PCI-E Pass Through.
Proxmox PCI-E Passthrough

[skogs]

@dogtreatfairy I agree that makes no sense. :)
I definitely think it was disabling hardware offload options rather than the RAM.

[Derelict]

@skogs Yes those must be disabled in proxmox/KVM. Quite normal to have to disable those in any virtual environment..

[tibere86]

@derelict Is it recommended to disable hardware checksum if the WAN is on a passthrough-ed NIC and the LAN is on a virtual NIC?

[Derelict]

@tibere86 I have never tried it on a passthrough NIC. But any time there is traffic THROUGH a VM it is wise to disable those offloads. You can certain'y see what works for you in your environment but you'll at least know what to disable if throughput tanks.

[abhishekakt]

So I did few testing and found it was pfsense package causing slow down. Bandwidthd, Darkstat, pftopng.. you name it. The worst part is, even if you remove these packages, you will never get original speed back. Here's how I did the testing:

Fresh install pfsense. Install and run iperf3 between Proxmox host (or any vm running under same host) and pfsense. I get full gigabit speed. Which is also strange the VMBR switch is sharing same kernel so between vm's I should get 10 Gigbit (theoretical) speed which I get with other linux vm's but not pfsense. Anyway, continuing my test. '

I install Bandwidthd and bam, speed drops to 200 Mbps in iperf3 test between pfsense and proxmox host. even if you remove the package from pfsense, you will never get same gigabit speed again ever. So just to prove this theory, I reinstalled pfsense over 100 times each time with different combinations. At last, I figured I could use snapshot to avoid reinstallling and reconfiguring pfsense.

So bottomline, these bandwidth morning package just kills pfsense throughput.

I have 2 Internet link 300 Mbps each.

[dogtreatfairy]

So, I would like to admit I'm an idiot. My problem was I couldn't get full speed on Wi-Fi. Well, turns out my computer was defaulting to connecting to my 2.4GHz network. When I upped the ram and restarted the pfSense VM, for some reason my computer reconnected to the 5GHz network.

On the 2.4GHz network my WAN tops out at 80-90 Mbps
On the 5 GHz network, the WAN tops out at my rated max of 230 Mbps

So, I didn't actually have a problem... just thought I did and managed to convince myself of it.

[osidosi]

Thank you "digdug3"
My production setup is PfSense virtualised on Proxmox, with fileover and balance gateway configuration. Max connected devices on our facility vary between 100-210.
HP DL380 G7
HP 4 port NIC 1Gbit (3 x WAN, planning to add 4th as Vodafone Radiolink backup WAN)
HP 2 Port 10Gbit SFP+, (LAN and Storage connections)
PVE 8,
The bandwidth of the WAN ports was sometimes slowing down to 7 mbit/s. The other bad situation was randomly packages losings and high latencies. I traing to do everiting described on forums but this article really help me. The magic word was "VirtIO"
What make difference for me is changing virtualizet network drivers from "Intel E1000" to "VırtlO" on PfSense virtual machine.

Befeore that
1- I was thicked these setting under System/Advanced/Networking
*Hardware Checksum Offloading
*Hardware TCP Segmentation Offloading
*Hardware Large Receive Offloading
*hn ALTQ support
2- Increase the Memory to 12Gb
3- Checked every cable and FO connections
4- Tested every DSL speed separately connected with pc directly to modems (speeds were as they should have been )
5- Get some additional support from ISP
6- Disable every firewall rule exept basic ones, disable, pfblockerng, disable pftopng, disable openvpn, destroy gateway group and enable only one WAN port a time.
7- Disable auto negotation and strict to different speeds
with no success.

Turning off the settings in step 1- after switching to VirtIO didn't make a big difference.
Hope this help some one.

[osidosi]

@digdug3 Thank you, your suggestion still works.
link text