Installing pfSense on XG105Rev3
-
Hi there,
Installed pfSense 2.5.2 on my Sophos XG105Rev3 and i experience a slow iperf connection between pfsense and a host (multiple host test in different subnet:
Connecting to host 192.168.178.1, port 5201 [ 4] local 192.168.178.110 port 60951 connected to 192.168.178.1 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 78.0 MBytes 0.65 Gbits/sec [ 4] 1.00-2.00 sec 80.4 MBytes 0.67 Gbits/sec [ 4] 2.00-3.00 sec 80.4 MBytes 0.67 Gbits/sec [ 4] 3.00-4.00 sec 80.5 MBytes 0.68 Gbits/sec [ 4] 4.00-5.00 sec 80.6 MBytes 0.68 Gbits/sec [ 4] 5.00-6.00 sec 80.5 MBytes 0.68 Gbits/sec [ 4] 6.00-7.00 sec 81.0 MBytes 0.68 Gbits/sec [ 4] 7.00-8.00 sec 80.6 MBytes 0.68 Gbits/sec [ 4] 8.00-9.00 sec 78.1 MBytes 0.65 Gbits/sec [ 4] 9.00-10.00 sec 77.2 MBytes 0.65 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 797 MBytes 0.67 Gbits/sec sender [ 4] 0.00-10.00 sec 797 MBytes 0.67 Gbits/sec receiverThe sophos hast 4 GBit Connections and in the pfSense it tells me that its 1000M Full dublex.
The Nics are Intel IGB`s and I didnt tune that much expect that I configured some limiters.The XG105Rev3 has 8Gigabyte of RAM, 64 GB SSD and a dual Core Intel Atom CPU.
The Hosts are connected via a 1 GBit 5-Port switch. -
@sysadminfromhell Limiters?
Most probably this is the reason for slow performance
Try disabling and retest -
@netblues hi there, thanks for the reply.
Still the same result:Connecting to host 192.168.178.1, port 5201 [ 4] local 192.168.178.110 port 64268 connected to 192.168.178.1 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 66.5 MBytes 0.56 Gbits/sec [ 4] 1.00-2.00 sec 70.5 MBytes 0.59 Gbits/sec [ 4] 2.00-3.00 sec 69.1 MBytes 0.58 Gbits/sec [ 4] 3.00-4.00 sec 67.8 MBytes 0.57 Gbits/sec [ 4] 4.00-5.00 sec 66.5 MBytes 0.56 Gbits/sec [ 4] 5.00-6.00 sec 68.5 MBytes 0.57 Gbits/sec [ 4] 6.00-7.00 sec 65.1 MBytes 0.55 Gbits/sec [ 4] 7.00-8.00 sec 67.9 MBytes 0.57 Gbits/sec [ 4] 8.00-9.00 sec 65.5 MBytes 0.55 Gbits/sec [ 4] 9.00-10.00 sec 66.9 MBytes 0.56 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 674 MBytes 0.57 Gbits/sec sender [ 4] 0.00-10.00 sec 674 MBytes 0.57 Gbits/sec receiver -
What Atom CPU is that? 600Mbps might be it's limit when testing to/from the firewall directly. Running iperf takes a lot of CPU cycles, it's always better to test through the firewall between two other hosts.
Steve
-
@stephenw10 thanks for your reply.
now its getting really weird:
[ 4] local 192.168.178.110 port 58111 connected to 172.16.24.101 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.01 sec 384 KBytes 3.12 Mbits/sec [ 4] 1.01-2.01 sec 384 KBytes 3.12 Mbits/sec [ 4] 2.01-3.01 sec 256 KBytes 2.11 Mbits/sec [ 4] 3.01-4.01 sec 128 KBytes 1.05 Mbits/sec [ 4] 4.01-5.00 sec 384 KBytes 3.17 Mbits/sec [ 4] 5.00-6.01 sec 384 KBytes 3.13 Mbits/sec [ 4] 6.01-7.01 sec 256 KBytes 2.09 Mbits/sec [ 4] 7.01-8.01 sec 256 KBytes 2.09 Mbits/sec [ 4] 8.01-9.00 sec 384 KBytes 3.18 Mbits/sec [ 4] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 2.88 MBytes 2.41 Mbits/sec sender [ 4] 0.00-10.00 sec 2.64 MBytes 2.21 Mbits/sec receiver -
@stephenw10 said in Installing pfSense on XG105Rev3:
What Atom CPU is that? 600Mbps might be it's limit when testing to/from the firewall directly. Running iperf takes a lot of CPU cycles, it's always better to test through the firewall between two other hosts.
Steve
its a Intel(R) Atom(TM) Processor E3930 @ 1.30GHz
-
@stephenw10 i checked speedtest on both PCs (Linux Servers) in the Subnet 172.16.24.0/24:
Speedtest by Ookla Server: Contabo - Dusseldorf (id = 35154) ISP: Deutsche Telekom AG Latency: 5.59 ms (0.13 ms jitter) Download: 86.83 Mbps (data used: 85.5 MB ) Upload: 28.84 Mbps (data used: 21.3 MB ) Packet Loss: Not available.Speedtest by Ookla Server: Contabo - Dusseldorf (id = 35154) ISP: Deutsche Telekom AG Latency: 18.87 ms (0.57 ms jitter) Download: 87.27 Mbps (data used: 42.6 MB ) Upload: 29.15 Mbps (data used: 14.6 MB ) Packet Loss: Not available.Also on my PC in 192.168.178.0/24:
The routing between the networks just seems slow.
-
@sysadminfromhell
i Guess we're done here because:iPerf from 192.168.178.0/24 -> 172.16.24.0/24 (Proxmox hosts) = slow
iPerf from 172.16.24.0/24 -> 192.168.178,0/24 (Clients) => fast
iPerf from 172.16.24.0/24 -> 172.16.24.0/24 = fast
i reset the Switch and see what happens. Maybe sometimes this helps. -
@sysadminfromhell
and were back in, i connected my windows PC directly to the PFsense and get this:----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.178.2, port 53372 [ 5] local 172.16.24.5 port 5201 connected to 192.168.178.2 port 53373 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.01 sec 6.50 MBytes 54.1 Mbits/sec [ 5] 1.01-2.00 sec 6.25 MBytes 52.8 Mbits/sec [ 5] 2.00-3.01 sec 5.62 MBytes 46.6 Mbits/sec [ 5] 3.01-4.00 sec 6.25 MBytes 52.9 Mbits/sec [ 5] 4.00-5.01 sec 5.62 MBytes 47.0 Mbits/sec [ 5] 5.01-6.00 sec 6.38 MBytes 53.9 Mbits/sec [ 5] 6.00-7.01 sec 6.12 MBytes 50.6 Mbits/sec [ 5] 7.01-8.00 sec 6.12 MBytes 51.9 Mbits/sec [ 5] 8.00-9.01 sec 6.50 MBytes 54.4 Mbits/sec [ 5] 9.01-10.00 sec 6.25 MBytes 52.6 Mbits/sec [ 5] 11.00-12.00 sec 6.62 MBytes 55.7 Mbits/sec [ 5] 12.00-13.01 sec 6.50 MBytes 53.8 Mbits/sec [ 5] 13.01-14.01 sec 6.25 MBytes 52.9 Mbits/sec [ 5] 14.01-15.01 sec 6.50 MBytes 54.3 Mbits/sec [ 5] 15.01-16.01 sec 6.75 MBytes 56.3 Mbits/sec [ 5] 16.01-17.01 sec 5.62 MBytes 47.6 Mbits/sec [ 5] 17.01-18.01 sec 6.88 MBytes 57.4 Mbits/sec [ 5] 18.01-19.00 sec 6.75 MBytes 56.9 Mbits/sec [ 5] 19.00-20.00 sec 6.75 MBytes 56.6 Mbits/sec [ 5] 20.00-21.01 sec 6.75 MBytes 56.4 Mbits/sec [ 5] 21.01-22.01 sec 7.00 MBytes 58.4 Mbits/sec [ 5] 22.01-23.00 sec 7.00 MBytes 59.2 Mbits/sec [ 5] 23.00-24.00 sec 7.00 MBytes 58.7 Mbits/sec [ 5] 24.00-25.01 sec 7.25 MBytes 60.4 Mbits/sec [ 5] 25.01-26.01 sec 7.38 MBytes 62.2 Mbits/sec [ 5] 26.01-27.00 sec 7.50 MBytes 63.2 Mbits/sec [ 5] 27.00-28.01 sec 2.50 MBytes 20.8 Mbits/sec [ 5] 28.01-29.00 sec 6.88 MBytes 58.3 Mbits/sec [ 5] 29.00-30.00 sec 7.62 MBytes 63.7 Mbits/sec [ 5] 30.00-30.01 sec 0.00 Bytes 0.00 bits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-30.01 sec 194 MBytes 54.1 Mbits/sec sender [ 5] 0.00-30.01 sec 0.00 Bytes 0.00 bits/sec receiver ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- iperf3: interrupt - the server has terminated PS C:\Users\Andre\Downloads\iperf-3.1.3-win64> .\iperf3.exe -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.178.2, port 53377 [ 5] local 172.16.24.5 port 5201 connected to 192.168.178.2 port 53378 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.01 sec 3.62 MBytes 30.2 Mbits/sec [ 5] 1.01-2.00 sec 7.62 MBytes 64.3 Mbits/sec [ 5] 2.00-3.00 sec 7.00 MBytes 58.5 Mbits/sec [ 5] 3.00-4.00 sec 7.75 MBytes 65.0 Mbits/sec [ 5] 4.00-5.00 sec 7.62 MBytes 63.9 Mbits/sec [ 5] 5.00-6.01 sec 7.50 MBytes 62.7 Mbits/sec [ 5] 6.01-7.01 sec 8.00 MBytes 67.3 Mbits/sec [ 5] 7.01-8.00 sec 8.00 MBytes 67.5 Mbits/sec [ 5] 8.00-9.00 sec 8.12 MBytes 67.9 Mbits/sec [ 5] 9.00-10.00 sec 8.00 MBytes 67.3 Mbits/sec [ 5] 10.00-11.01 sec 8.25 MBytes 68.9 Mbits/sec [ 5] 11.01-12.00 sec 8.50 MBytes 71.6 Mbits/sec [ 5] 12.00-13.01 sec 8.62 MBytes 71.4 Mbits/sec [ 5] 13.01-14.00 sec 8.62 MBytes 73.0 Mbits/sec [ 5] 14.00-15.00 sec 8.88 MBytes 74.5 Mbits/sec [ 5] 15.00-16.00 sec 8.88 MBytes 74.5 Mbits/sec [ 5] 16.00-17.00 sec 9.12 MBytes 76.4 Mbits/sec [ 5] 17.00-18.00 sec 9.38 MBytes 78.6 Mbits/sec [ 5] 18.00-19.01 sec 8.62 MBytes 72.0 Mbits/sec [ 5] 19.01-20.01 sec 9.38 MBytes 78.5 Mbits/sec [ 5] 20.01-21.00 sec 9.50 MBytes 80.4 Mbits/sec [ 5] 21.00-22.01 sec 9.50 MBytes 79.5 Mbits/sec [ 5] 22.01-23.00 sec 9.50 MBytes 79.7 Mbits/sec [ 5] 23.00-24.00 sec 9.62 MBytes 81.0 Mbits/sec [ 5] 24.00-25.00 sec 10.0 MBytes 83.6 Mbits/sec [ 5] 25.00-26.01 sec 10.1 MBytes 84.8 Mbits/sec [ 5] 26.01-27.01 sec 10.2 MBytes 85.8 Mbits/sec [ 5] 27.01-28.00 sec 10.5 MBytes 88.6 Mbits/sec [ 5] 28.00-29.00 sec 10.5 MBytes 88.1 Mbits/sec [ 5] 29.00-30.00 sec 10.8 MBytes 89.9 Mbits/sec [ 5] 30.00-30.01 sec 0.00 Bytes 0.00 bits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-30.01 sec 262 MBytes 73.2 Mbits/sec sender [ 5] 0.00-30.01 sec 0.00 Bytes 0.00 bits/sec receiverSome somehow the port is slow?
-
@sysadminfromhell
i changed a few things in the /boot/loader.conf as mentioned in a different Post with a Intel i211 NIC:kern.cam.boot_delay=10000 # agree with Intel license terms legal.intel_ipw.license_ack=1 legal.intel_iwi.license_ack=1 # This is the magic. If you don't set this, queues won't be utilized properly. # Allow multiple processes for receive/transmit processing. hw.igb.0.rx_process_limit="-1" hw.igb.0.tx_process_limit="-1" hw.igb.1.rx_process_limit="-1" hw.igb.1.tx_process_limit="-1" hw.igb.2.rx_process_limit="-1" hw.igb.2.tx_process_limit="-1" hw.igb.3.rx_process_limit="-1" hw.igb.3.tx_process_limit="-1" #net.pf.states_hashsize=2097152 hw.igb.0.rxd=4096 hw.igb.0.txd=4096 hw.igb.1.rxd=4096 hw.igb.1.txd=4096 hw.igb.2.rxd=4096 hw.igb.2.txd=4096 hw.igb.3.rxd=4096 hw.igb.3.txd=4096 #net.inet.tcp.syncache.hashsize="1024" #net.inet.tcp.syncache.bucketlimit="100" kern.ipc.nmbclusters="1000000" kern.ipc.nmbjumbop="524288" kern.ipc.nmbjumbo9="524288" autoboot_delay="3" net.link.ifqmaxlen="128"but this didnt change a thing (maybe outdated?)
I updated to 2.6.0 (2.6.0.r.20220124.1828) as someone mentioned better beformance with Nic I211 as in the XG105Rev3.But at all the performance between the interfaces remains slow.
I mean in one way (igb1->igb2) the other way seems okay.
Whats weird is, that when i iperf3 agains the IP of igb2 its also slow. So the Routing between igb1->igb2 seems to be slow.maybe someone has an idea in part of "Intel Nic I211" or something, I dont know :(
-
Mmm, well I'd expect at least 600Mbps from that. Try running
top -aSHat the same time and see if one of the cores is maxed out. -
@stephenw10
Hi there, thanks for the reply, I tried that but the cpu is not maxing out. It sticks at around 20% while the test is running -
On both cores?
-
@stephenw10
Hi there, will check this again soon but yes afaik it was on both cores 20% usage. -
Hmm, I would expect more usage than that on that CPU at 600Mbps.
-
@stephenw10 hi there i checked again and I am at 91%. sorry for the misinformation around there.
-
@stephenw10 but the main problem remains: why is the network speed between 2 networks (2 nics) so slow?
when i check iperf3 from igb1(192.168.178.0/24) to igb2(172.16.24.0/24) its super super slow (3,1 MBit/s) and the CPU usage on the pfSense is low (about 5%, peek to 20/25%).
Is there something i can do/tune/change in the setup? -
That sort of throttling is almost certainly one of the NICs not linked correctly. When you see less that 100Mbps on what should be a Gigabit link it's usually because the autonegotiation has failed and something is running at the fallback default of 100M half-duplex. So check the ifconfig output on pfSense and check the link speed at each host.
Steve
-
@stephenw10 i forced it via pfsense to 1000Tbase Full duplex but still the same issue:
Connecting to host 172.16.24.100, port 5201 [ 4] local 192.168.178.110 port 52362 connected to 172.16.24.100 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.01 sec 256 KBytes 2.08 Mbits/sec [ 4] 1.01-2.00 sec 384 KBytes 3.17 Mbits/sec [ 4] 2.00-3.01 sec 128 KBytes 1.04 Mbits/sec [ 4] 3.01-4.00 sec 256 KBytes 2.11 Mbits/sec [ 4] 4.00-5.01 sec 512 KBytes 4.16 Mbits/sec [ 4] 5.01-6.00 sec 256 KBytes 2.11 Mbits/sec [ 4] 6.00-7.01 sec 256 KBytes 2.08 Mbits/sec [ 4] 7.01-8.00 sec 128 KBytes 1.06 Mbits/sec [ 4] 8.00-9.01 sec 384 KBytes 3.12 Mbits/sec [ 4] 9.01-10.01 sec 384 KBytes 3.17 Mbits/sec [ 4] 10.01-11.01 sec 512 KBytes 4.16 Mbits/sec [ 4] 11.01-12.01 sec 384 KBytes 3.17 Mbits/sec [ 4] 12.01-13.02 sec 384 KBytes 3.12 Mbits/sec [ 4] 13.02-14.01 sec 256 KBytes 2.11 Mbits/sec [ 4] 14.01-15.00 sec 512 KBytes 4.22 Mbits/sec [ 4] 15.00-16.01 sec 640 KBytes 5.20 Mbits/sec [ 4] 16.01-17.00 sec 256 KBytes 2.11 Mbits/sec [ 4] 17.00-18.01 sec 384 KBytes 3.12 Mbits/sec [ 4] 18.01-19.00 sec 256 KBytes 2.11 Mbits/sec [ 4] 19.00-20.01 sec 256 KBytes 2.08 Mbits/sec [ 4] 20.01-21.00 sec 384 KBytes 3.17 Mbits/sec [ 4] 21.00-22.01 sec 512 KBytes 4.16 Mbits/sec [ 4] 22.01-23.01 sec 384 KBytes 3.17 Mbits/sec [ 4] 23.01-24.01 sec 256 KBytes 2.08 Mbits/sec [ 4] 24.01-25.01 sec 384 KBytes 3.17 Mbits/sec [ 4] 25.01-26.00 sec 384 KBytes 3.17 Mbits/sec [ 4] 26.00-27.01 sec 384 KBytes 3.12 Mbits/sec [ 4] 27.01-28.00 sec 384 KBytes 3.17 Mbits/sec [ 4] 28.00-29.01 sec 384 KBytes 3.12 Mbits/sec [ 4] 29.01-30.00 sec 256 KBytes 2.11 Mbits/sec [ 4] 30.00-31.01 sec 384 KBytes 3.12 Mbits/sec [ 4] 31.01-32.00 sec 384 KBytes 3.17 Mbits/sec [ 4] 32.00-33.01 sec 512 KBytes 4.16 Mbits/sec [ 4] 33.01-34.00 sec 512 KBytes 4.22 Mbits/sec [ 4] 34.00-35.01 sec 256 KBytes 2.08 Mbits/sec [ 4] 35.01-36.01 sec 128 KBytes 1.06 Mbits/sec [ 4] 36.01-37.01 sec 128 KBytes 1.04 Mbits/sec [ 4] 37.01-38.01 sec 640 KBytes 5.28 Mbits/sec [ 4] 38.01-39.00 sec 384 KBytes 3.17 Mbits/sec [ 4] 39.00-40.01 sec 384 KBytes 3.12 Mbits/sec [ 4] 40.01-41.00 sec 512 KBytes 4.22 Mbits/sec [ 4] 41.00-42.01 sec 256 KBytes 2.08 Mbits/sec [ 4] 42.01-43.00 sec 256 KBytes 2.11 Mbits/sec [ 4] 43.00-44.01 sec 384 KBytes 3.12 Mbits/sec [ 4] 44.01-45.00 sec 512 KBytes 4.22 Mbits/sec [ 4] 45.00-46.01 sec 640 KBytes 5.20 Mbits/sec [ 4] 46.01-47.01 sec 256 KBytes 2.11 Mbits/sec [ 4] 47.01-48.01 sec 128 KBytes 1.04 Mbits/sec [ 4] 48.01-49.01 sec 640 KBytes 5.28 Mbits/sec [ 4] 49.01-50.00 sec 256 KBytes 2.10 Mbits/sec [ 4] 50.00-51.01 sec 384 KBytes 3.13 Mbits/sec [ 4] 51.01-52.00 sec 384 KBytes 3.17 Mbits/sec [ 4] 52.00-53.01 sec 256 KBytes 2.08 Mbits/sec [ 4] 53.01-54.00 sec 128 KBytes 1.06 Mbits/sec [ 4] 54.00-55.01 sec 512 KBytes 4.16 Mbits/sec [ 4] 55.01-56.00 sec 384 KBytes 3.17 Mbits/sec [ 4] 56.00-57.01 sec 640 KBytes 5.20 Mbits/sec [ 4] 57.01-58.00 sec 256 KBytes 2.11 Mbits/sec [ 4] 58.00-59.01 sec 512 KBytes 4.16 Mbits/sec [ 4] 59.01-60.01 sec 128 KBytes 1.06 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-60.01 sec 21.1 MBytes 2.95 Mbits/sec sender [ 4] 0.00-60.01 sec 21.0 MBytes 2.93 Mbits/sec receiverEDIT: all my hosts are at 1GBit (1000M) the whole time. My switches even say, that the Uplink to the router (pfsense) is 1000M. I forced it but still no change.
-
@sysadminfromhell Well, have you checked the obvious?
Cables/patch cords.
There isn't much to tune in any case and intel cards and pfsense are considered perfect matches
