Speed inbound outbound mismatch

mattiols

Hello everyone, today I configured an appliance with a PPPOE connection to a TIM ONT, with the TIM modem I reach 800Mbps in Download and 300 in Upload, with pfsense instead I have a maximum of 270Mbps and the usual 300 in Upload.
I made several tests, and I realized that LAN traffic also behaves in the same way, the inbound speed reaches a maximum of 35.7 MBytes / sec while the outbound one arrives correctly at 112 MBytes / sec, in addition during the outbound test the cpu remains stable between 20 and 26% of use, while during the inbound tests it reaches 50/60%.
I don't use limits or traffic shaping, what could be causing this unexpected behavior?

### INBOUND
[*********~]$ iperf3 -c 192.168.1.1 -t 60 -i 5 -f MB/s
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.191 port 55234 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-5.00   sec   180 MBytes  35.9 MBytes/sec    0    274 KBytes      
[  5]   5.00-10.00  sec   178 MBytes  35.7 MBytes/sec    0    274 KBytes      
[  5]  10.00-15.00  sec   178 MBytes  35.7 MBytes/sec    0    274 KBytes      
[  5]  15.00-20.00  sec   179 MBytes  35.7 MBytes/sec    0    274 KBytes      
[  5]  20.00-25.00  sec   177 MBytes  35.5 MBytes/sec    0    274 KBytes      
[  5]  25.00-30.00  sec   177 MBytes  35.4 MBytes/sec    0    290 KBytes      
[  5]  30.00-35.00  sec   178 MBytes  35.6 MBytes/sec    0    290 KBytes      
[  5]  35.00-40.00  sec   178 MBytes  35.6 MBytes/sec    0    290 KBytes      
[  5]  40.00-45.00  sec   178 MBytes  35.5 MBytes/sec    0    290 KBytes      
[  5]  45.00-50.00  sec   178 MBytes  35.6 MBytes/sec    0    290 KBytes      
[  5]  50.00-55.00  sec   174 MBytes  34.8 MBytes/sec    0    290 KBytes      
[  5]  55.00-60.00  sec   178 MBytes[2.5.2-RELEASE]

### OUTBOUND
[2.5.2-RELEASE][*******]/root: iperf3 -c 192.168.1.191 -t 60 -i 5 -f MB/s
Connecting to host 192.168.1.191, port 5201
[  5] local 192.168.1.1 port 8600 connected to 192.168.1.191 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-5.00   sec   561 MBytes   112 MBytes/sec    0   1.25 MBytes      
[  5]   5.00-10.00  sec   560 MBytes   112 MBytes/sec    0   1.77 MBytes      
[  5]  10.00-15.00  sec   560 MBytes   112 MBytes/sec    0   2.17 MBytes      
[  5]  15.00-20.00  sec   559 MBytes   112 MBytes/sec    0   2.50 MBytes      
[  5]  20.00-25.00  sec   559 MBytes   112 MBytes/sec    0   2.79 MBytes      
[  5]  25.00-30.00  sec   559 MBytes   112 MBytes/sec    0   3.00 MBytes      
[  5]  30.00-35.00  sec   559 MBytes   112 MBytes/sec    0   3.00 MBytes      
[  5]  35.00-40.00  sec   560 MBytes   112 MBytes/sec    0   3.00 MBytes      
[  5]  40.00-45.00  sec   559 MBytes   112 MBytes/sec    0   3.00 MBytes      
[  5]  45.00-50.00  sec   559 MBytes   112 MBytes/sec    0   3.00 MBytes      
[  5]  50.00-55.00  sec   559 MBytes   112 MBytes/sec    0   3.00 MBytes      
[  5]  55.00-60.00  sec   559 MBytes   112 MBytes/sec    0   3.00 MBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  6.56 GBytes   112 MBytes/sec    0             sender
[  5]   0.00-60.82  sec  6.56 GBytes   110 MBytes/sec                  receiver
  35.5 MBytes/sec    0    290 KBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec  2.08 GBytes  35.6 MBytes/sec    0             sender
[  5]   0.00-60.01  sec  2.08 GBytes  35.5 MBytes/sec                  receiver

Below the screenshots captured during the tests

speedtest:

Ipef:

Thanks to all

stephenw10

What hardware are you using?

You may be hitting this:
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

Steve

mattiols

@stephenw10 Hi Steve, first of all thx for your answer! It's an appliance fanless and rackmount bought on Amazon, below the specs:

### CPU_INFO
[2.5.2-RELEASE][******]/root: dmidecode --string processor-version
Intel(R) Atom(TM) CPU D525   @ 1.80GHz              
[2.5.2-RELEASE][******]/root: dmidecode --string processor-frequency
1800 MHz

### RAM_INFO
[2.5.2-RELEASE][******]/root: dmidecode -t 16
# dmidecode 3.3
Scanning /dev/mem for entry point.
SMBIOS 2.6 present.

Handle 0x000C, DMI type 16, 15 bytes
Physical Memory Array
        Location: System Board Or Motherboard
        Use: System Memory
        Error Correction Type: None
        Maximum Capacity: 4 GB
        Error Information Handle: Not Provided
        Number Of Devices: 2

### NIC_INFO
[2.5.2-RELEASE][******]/root: dmesg | egrep "em[0-3]:(.*port)" | sort | uniq
em0: <Intel(R) PRO/1000 Network Connection> port 0xbc00-0xbc1f mem 0xfe8e0000-0xfe8fffff,0xfe8dc000-0xfe8dffff irq 16 at device 0.0 on pci1
em1: <Intel(R) PRO/1000 Network Connection> port 0xcc00-0xcc1f mem 0xfe9e0000-0xfe9fffff,0xfe9dc000-0xfe9dffff irq 17 at device 0.0 on pci2
em2: <Intel(R) PRO/1000 Network Connection> port 0xdc00-0xdc1f mem 0xfeae0000-0xfeafffff,0xfeadc000-0xfeadffff irq 18 at device 0.0 on pci3
em3: <Intel(R) PRO/1000 Network Connection> port 0xec00-0xec1f mem 0xfebe0000-0xfebfffff,0xfebdc000-0xfebdffff irq 19 at device 0.0 on pci4

### VMSTAT_INFO
[2.5.2-RELEASE][*****]/root: vmstat 
procs  memory       page                    disks     faults         cpu
r b w  avm   fre   flt  re  pi  po    fr   sr md0 ad0   in    sy    cs us sy id
0 0 0 977M  3.3G   841   0   0   1   940   17   0   0  738  2923  1750  1  2 97

Can i provided specific sysctl params if can help you?

Thx
Andrea

stephenw10

Ah, an Atom D525 will be CPU limited there. It's been a while since I tested one but when they were quite common they were good for 600-700Mbps if they have Intel NICs.
But that's without the PPPoE limitation where only one CPU core can be used for receive traffic.

If you run at the console whilst testing: top -aSH
You will probably see one CPU core is at 100%. You might be able to improve it by applying the system tunable from the docs page but you will never see the full 800Mbps.

Steve

mattiols

@stephenw10 said in Speed inbound outbound mismatch:

Ah, an Atom D525 will be CPU limited there. It's been a while since I tested one but when they were quite common they were good for 600-700Mbps if they have Intel NICs.
But that's without the PPPoE limitation where only one CPU core can be used for receive traffic.
If you run at the console whilst testing: top -aSH
You will probably see one CPU core is at 100%. You might be able to improve it by applying the system tunable from the docs page but you will never see the full 800Mbps.
Steve

Hi, thanks for your answer I know the PPPOE limit based on single CPU used, but on LAN interface i've the same problem, what is the difference from inbound and outbound in terms of cpu utilizzation?
which kind of tuning did you suggest for this situation?
You suggest to upgrade the CPU compatible with socket BGA559

Thanks

stephenw10

@mattiols said in Speed inbound outbound mismatch:

but on LAN interface i've the same problem

Is that between two devices on the same LAN? Between two internal interfaces?
I would expect to see in the 600-700Mps range between two local subnets unless you're running Squid or Snort or something else CPU heavy.

I still recommend running top -aSH at the console whilst testing to see how the load is distributed. If you post that output here we can review it.

@mattiols said in Speed inbound outbound mismatch:

which kind of tuning did you suggest for this situation?

What is suggested here: https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

Go to Sys > Adv > System Tunables and create a newtunable: net.isr.dispatch.
Set it's value to deferred.

The Atom D525 is now a very old CPU. It cannot be upgraded, it's soldered to the board in every box I've seen. (BGA).
We used to sell a firewall using that CPU, the FW-7541, but we stopped selling in 2014!
https://www.netgate.com/support/product-lifecycle

Steve

mattiols

@stephenw10 said in Speed inbound outbound mismatch:

Is that between two devices on the same LAN? Between two internal interfaces?
I would expect to see in the 600-700Mps range between two local subnets unless you're running Squid or Snort or something else CPU heavy.

On LAN interface i've configured an LACP, if you see my first post you can see the iperf3 test between the FW and another physical host inside my network (squid and snort disabled).
If you want i can reproduce the same test bypassing LAG connecting direclty my laptop on FW to another interface.

last pid: 90823;  load averages:  1.22,  0.40,  0.19                                                      up 0+03:08:11  16:33:59
200 threads:   7 running, 164 sleeping, 29 waiting
CPU:  0.9% user,  0.0% nice, 29.0% system, 25.1% interrupt, 45.0% idle
Mem: 37M Active, 97M Inact, 262M Wired, 92M Buf, 3496M Free
Swap: 4096M Total, 4096M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   12 root        -72    -     0B   480K CPU3     3   1:42  99.64% [intr{swi1: netisr 3}]
72907 root        102    0    15M  6324K CPU1     1   0:46  96.73% iperf3 -s
   11 root        155 ki31     0B    64K CPU2     2 184:12  74.62% [idle{idle: cpu2}]
   11 root        155 ki31     0B    64K RUN      3 183:22  54.52% [idle{idle: cpu3}]
   11 root        155 ki31     0B    64K RUN      0 183:37  40.76% [idle{idle: cpu0}]
    0 root        -76    -     0B   576K -        2   1:30  19.56% [kernel{if_io_tqg_2}]
   11 root        155 ki31     0B    64K RUN      1 182:12   6.85% [idle{idle: cpu1}]
   12 root        -72    -     0B   480K WAIT     3   2:13   2.32% [intr{swi1: netisr 1}]
    0 root        -76    -     0B   576K -        3   0:17   1.80% [kernel{if_io_tqg_3}]
    0 root        -76    -     0B   576K -        0   1:28   0.93% [kernel{if_io_tqg_0}]
   12 root        -72    -     0B   480K WAIT     2   2:24   0.81% [intr{swi1: netisr 0}]
   12 root        -72    -     0B   480K WAIT     3   0:42   0.56% [intr{swi1: netisr 2}]
 6452 root         20    0    24M    13M nanslp   0   0:21   0.26% /usr/local/sbin/pcscd{pcscd}
    0 root        -76    -     0B   576K -        1   0:16   0.22% [kernel{if_io_tqg_1}]
48959 root         20    0    13M  3820K CPU0     0   0:04   0.20% top -aSH

This morning I've apply this tuning:

[2.5.2-RELEASE][******]/root: cat /boot/loader.conf.local 
net.isr.dispatch=deferred
net.inet.ip.intr_queue_maxlen=10000

Furthermore I tried to disable the flow control for em interface, but i lose 250Mbps in upload!

SteveITS

@mattiols Are you testing on pfSense itself, it looks like? Same speed if testing from a LAN PC to the Internet?

mattiols

@steveits yes i've tested the speed from both (server and FW)

[2.5.2-RELEASE][******]/root: speedtest 
Retrieving speedtest.net configuration...
Testing from Telecom Italia (62.211.29.5)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by TIM SpA (Venice) [258.76 km]: 30.021 ms
Testing download speed................................................................................
Download: 221.20 Mbit/s
Testing upload speed......................................................................................................
Upload: 266.56 Mbit/s

Leaving out the problem of WAN speed with PPPOE, I don't understand why from the FW to a server I reach 100MBps, while from a server to the FW at most 35 / 40MBps,
the internal network is good I tried between 2 servers on the same network and they always send and receive around 100MBps

stephenw10

Testing either to or from the firewall itself is not a good test as uses significant CPU cycles just to run iperf.

To see the real throughput of your hardware you need to test between two hosts on separate internal interfaces. The LACP LAGG is unlikely to make a significant difference.

Steve