Extreme slow internet speed pfsense over proxmox


  • Hello

    I migrated to proxmox ve over a year now from hyperv and I just love the product. I have small home system running one single proxmox on Intel Core i5 with 16 GB RAM.

    I have firewall and few other vms running.

    I was using Sophos XG but moved to PFSENSE when they added subscription to every possible feature. Sophos XG was giving good throughout but with Proxmox, pfsense throughput dropped quite badly. I have gone through Proxmox and Netgate suggested settings and turned off all Hardware offloading. It has fixed the upload but still have slow internet speed. I have 300 Mbps up down link and I get around 150 with pfsense. PFSense vm has 2 cores and 4 GB RAM. Hardware usage are quite low.

    I also tried spinning fresh PFSense VM and even tried PFSENSE developer build 2.5 but same problem.

    I have Intel Gigabit nic. I tried using all network adapter options from Proxmox VirtIO, Intel E1000, VMware and even realteck one but no help.

    I also tried other forum post suggesting disable TX on vmbr0 and actual ethernet port in /etc/network/interfaces but no help.

    I can tell its not Proxmox issue as if I spin up Untangle vm, I get full throughput. I read somewhere FreeBSD does not work well with proxmox. When I use pfsense in dedicated hardware, speed works just fine. I guess it has to do with the FreeBSD.

    I dont use traffic shaping.

    Here's my iperf result between pfsense vm and ubuntu vm connected to vmbr0 which shows it is indeed problem with pfsense vm:

    ubuntu@zm:~$ iperf3 -c asa
    Connecting to host asa, port 5201
    [ 4] local 10.12.47.43 port 43808 connected to 10.12.47.1 port 5201
    [ ID] Interval Transfer Bandwidth Retr Cwnd
    [ 4] 0.00-1.00 sec 19.7 MBytes 165 Mbits/sec 0 452 KBytes
    [ 4] 1.00-2.00 sec 32.4 MBytes 272 Mbits/sec 0 755 KBytes
    [ 4] 2.00-3.00 sec 35.0 MBytes 293 Mbits/sec 0 823 KBytes
    [ 4] 3.00-4.00 sec 32.2 MBytes 270 Mbits/sec 0 823 KBytes
    [ 4] 4.00-5.00 sec 36.2 MBytes 305 Mbits/sec 0 923 KBytes
    [ 4] 5.00-6.00 sec 43.7 MBytes 366 Mbits/sec 0 929 KBytes
    [ 4] 6.00-7.00 sec 35.5 MBytes 298 Mbits/sec 0 977 KBytes
    [ 4] 7.00-8.00 sec 35.9 MBytes 301 Mbits/sec 0 977 KBytes
    [ 4] 8.00-9.00 sec 33.5 MBytes 281 Mbits/sec 0 977 KBytes
    [ 4] 9.00-10.04 sec 31.2 MBytes 253 Mbits/sec 0 977 KBytes


    [ ID] Interval Transfer Bandwidth Retr
    [ 4] 0.00-10.04 sec 335 MBytes 280 Mbits/sec 0 sender
    [ 4] 0.00-10.04 sec 333 MBytes 278 Mbits/sec receiver

    iperf between two ubuntu vm's connected to same vmbr0 switch give better results:

    ubuntu@nextcloud:~$ iperf3 -c zm
    Connecting to host zm, port 5201
    [ 4] local 10.12.47.50 port 42630 connected to 10.12.47.43 port 5201
    [ ID] Interval Transfer Bandwidth Retr Cwnd
    [ 4] 0.00-1.00 sec 419 MBytes 3.51 Gbits/sec 0 3.15 MBytes
    [ 4] 1.00-2.00 sec 437 MBytes 3.67 Gbits/sec 1 3.15 MBytes
    [ 4] 2.00-3.00 sec 422 MBytes 3.55 Gbits/sec 1 3.15 MBytes
    [ 4] 3.00-4.00 sec 526 MBytes 4.40 Gbits/sec 0 3.15 MBytes
    [ 4] 4.00-5.00 sec 522 MBytes 4.39 Gbits/sec 0 3.15 MBytes
    [ 4] 5.00-6.00 sec 474 MBytes 3.97 Gbits/sec 0 3.15 MBytes
    [ 4] 6.00-7.00 sec 314 MBytes 2.64 Gbits/sec 1 3.15 MBytes
    [ 4] 7.00-8.00 sec 258 MBytes 2.16 Gbits/sec 0 3.15 MBytes
    [ 4] 8.00-9.00 sec 412 MBytes 3.46 Gbits/sec 1 3.15 MBytes
    [ 4] 9.00-10.00 sec 439 MBytes 3.68 Gbits/sec 0 3.15 MBytes


    [ ID] Interval Transfer Bandwidth Retr
    [ 4] 0.00-10.00 sec 4.13 GBytes 3.54 Gbits/sec 4 sender
    [ 4] 0.00-10.00 sec 4.12 GBytes 3.54 Gbits/sec receiver

    iperf from Proxmox host to ubuntu vm works well as expected:

    root@proxmox:~# iperf3 -c zm
    Connecting to host zm, port 5201
    [ 5] local 10.12.47.10 port 42520 connected to 10.12.47.43 port 5201
    [ ID] Interval Transfer Bitrate Retr Cwnd
    [ 5] 0.00-1.00 sec 756 MBytes 6.34 Gbits/sec 1 3.13 MBytes
    [ 5] 1.00-2.00 sec 905 MBytes 7.59 Gbits/sec 0 3.13 MBytes
    [ 5] 2.00-3.00 sec 729 MBytes 6.11 Gbits/sec 0 3.13 MBytes
    [ 5] 3.00-4.00 sec 834 MBytes 6.99 Gbits/sec 1 3.13 MBytes
    [ 5] 4.00-5.00 sec 849 MBytes 7.12 Gbits/sec 0 3.13 MBytes
    [ 5] 5.00-6.00 sec 625 MBytes 5.24 Gbits/sec 5 3.13 MBytes
    [ 5] 6.00-7.00 sec 638 MBytes 5.35 Gbits/sec 0 3.13 MBytes
    [ 5] 7.00-8.00 sec 860 MBytes 7.21 Gbits/sec 0 3.13 MBytes
    [ 5] 8.00-9.00 sec 882 MBytes 7.40 Gbits/sec 0 3.13 MBytes
    [ 5] 9.00-10.00 sec 810 MBytes 6.80 Gbits/sec 0 3.13 MBytes


    [ ID] Interval Transfer Bitrate Retr
    [ 5] 0.00-10.00 sec 7.70 GBytes 6.62 Gbits/sec 7 sender
    [ 5] 0.00-10.00 sec 7.70 GBytes 6.62 Gbits/sec receiver

    iperf from Promox host to pfsense gets slow results:

    root@proxmox:~# iperf3 -c asa
    Connecting to host asa, port 5201
    [ 5] local 10.12.47.10 port 34060 connected to 10.12.47.1 port 5201
    [ ID] Interval Transfer Bitrate Retr Cwnd
    [ 5] 0.00-1.00 sec 46.6 MBytes 391 Mbits/sec 223 184 KBytes
    [ 5] 1.00-2.00 sec 59.3 MBytes 497 Mbits/sec 79 341 KBytes
    [ 5] 2.00-3.00 sec 53.6 MBytes 449 Mbits/sec 53 426 KBytes
    [ 5] 3.00-4.00 sec 43.9 MBytes 369 Mbits/sec 0 495 KBytes
    [ 5] 4.00-5.00 sec 42.6 MBytes 357 Mbits/sec 95 544 KBytes
    [ 5] 5.00-6.00 sec 42.7 MBytes 358 Mbits/sec 0 601 KBytes
    [ 5] 6.00-7.00 sec 49.5 MBytes 415 Mbits/sec 90 646 KBytes
    [ 5] 7.00-8.00 sec 46.2 MBytes 388 Mbits/sec 43 691 KBytes
    [ 5] 8.00-9.00 sec 51.2 MBytes 430 Mbits/sec 48 742 KBytes
    [ 5] 9.00-10.00 sec 35.0 MBytes 294 Mbits/sec 0 778 KBytes


    [ ID] Interval Transfer Bitrate Retr
    [ 5] 0.00-10.00 sec 471 MBytes 395 Mbits/sec 631 sender
    [ 5] 0.00-10.01 sec 467 MBytes 391 Mbits/sec receiver

    I am not sure what could be the problem. I know lot of people use pfsense on proxmox so would appreciate if anyone can help.

    Thanks


  • Obviously I can't prove that this is your issue, but a week or so ago I thought I had a good idea and made the proxmox vm's virtual MAC match the physical host interface. It was not a good idea. Very similar slow responses...everything ~seemed~ up and proper, but it was just not coming anywhere near functional.


  • That can't be the reason in my setup. As I mentioned, I tried spinning few pfsense instances with all random MAC addresses and has same issue.

    I also spin up FreeBSD vm and got better performance so I am sure it has to do with pfsense


  • Sorry if I sound like a putz but you did try checking the boxes for all three of these right?:
    Disable hardware checksum offload
    Disable hardware TCP segmentation offload
    Disable hardware large receive offload

    I did indeed jack that up once a while back too.

    Less likely; but still possible on power savings section turn on/check Enable PowerD; and all options set to Hiadaptive or Max.

    I've also come to believe strongly that turning on the ramdisk option both saves ssd wear and increases throughput a bit.

    Apologies if I'm not helpful and you've already tried these several times.

  • LAYER 8 Netgate

    @skogs said in Extreme slow internet speed pfsense over proxmox:

    Sorry if I sound like a putz but you did try checking the boxes for all three of these right?:
    Disable hardware checksum offload
    Disable hardware TCP segmentation offload
    Disable hardware large receive offload

    Almost certainly this... 👆


  • lol


  • I already tried these.. no help. Something is not right. when I run iperf on vm connected to same vswitch with pfsense, it never exceeds 300 Mbps and goes does as 80 mbps. It is slower than my physical LAN


    1. Use virtio
    2. Use the checkboxes as stated above and reboot VM after setting them
    3. Disabled "firewall" in the VM in Proxmox
    4. Use 2.4.4-p3 for now. 2.4.5 has issues

  • So for giggles I did do some extra testing last week. Re-did the pfsense on a tiny physical machine again. Pretty much no matter what I did the physical instance drastically outperformed the virtual.

    When it was virtualized I was also mirroring off the traffic to a third virtual NIC for monitoring on the inside of the firewall anything going out. In the physical environment this is taken care of by a physical switch. Turning this off did improve my data processing somewhat, but not entirely. Lets face it; a virtual network adaptor resides in memory and shouldn't really slow things down much.

    I don't have half the network bandwidth you guys seem to have.
    Physical: averaging 65-82Mbps on what is advertised as a 100M line.
    Virtualized: depending on settings anywhere from 30Mbps to 45Mbps.

    Physical is using little Dual Core Celeron N3160 at 1.6Ghz
    Virtualized is using little Quad Core Ryzen 1200 at 3Ghz
    The ryzen would generally outperform the Celeron a bit less than 3x with a vastly better single process speed and multi process. Kind of odd. I've never seen this large of a performance hit on Proxmox before.


  • I'm getting 100/100 on proxmox, the same as I'm getting when connected physical.
    What NICs are you using. I only use Intel based ones for Proxmox.
    Also make sure Proxmox is updated.


  • Confirm the proxmox is using crappy Realtek RTL8111/8168/8411 style.
    So is the bare metal.