@jnz07 I replaced mine with a 550-t2 to avoid future issues

I agree, if it was software the behavior would be identical on all ports.

There are many subtle ways electronics can fail under load, it's difficult to speculate about why, but usually boils down to heat or some kind of electrical interference which can't be compensated for at high load vs low load.

@stephenw10 It solved one issue but now I need to find another but its not related to the nic

Yes, it's that. As I linked above. 😉

Because you are seeing almost 100% packet loss at anywhere close to those speeds.

Either you're testing completely wrong or the connection is very broken.

If it's the later knowing what value the packet loss seems to kick in at is useful.

Steve

@bingo600
Thanks for the help the custom DNS resolver worked

https://forum.netgate.com/topic/149988/installing-pfsense-on-f5-big-ip-5050

https://forum.netgate.com/topic/154364/pfsense-on-f5-big-ip5050-series

https://forum.netgate.com/topic/66067/f5-networks-big-ip-1500

All tails of failure!

Hmm, not sure what that is then.

It's a Realtek NIC that requires the newer driver. In 2.5.0 do this: https://forum.netgate.com/post/946519

Steve

The package is in the 2.5.0 repo so you can do this: https://forum.netgate.com/post/946519

Steve

Please note that on post #1 the iperf server and client was running into the APU2 devices thus the traffic isn't the best achieveable.

I' ve repeated tha same tests in a better scenario that leaves the APU2 devices to only handle the traffic generated elsewhere.

1 2 ┌────────┐ ┌───────────┐ ┌───────────┐ ┌────────┐ │ │ │ │ │ │ │ │ │ ├─┐ │ APU2 │ │ APU2 │ ┌─┤ │ ├────────┤ │ │ │ │ │ │ ├────────┤ │┼┼┼┼┼┼┼┼│ │ └─┬───────┬─┘ └─┬───────┬─┘ │ │┼┼┼┼┼┼┼┼│ │┼┼┼┼┼┼┼┼│ │ │ │ │ │ │ │┼┼┼┼┼┼┼┼│ └────────┘ │ │ │ WAN │ │ │ └────────┘ iperf server └────┘ └──────────┘ └────┘ iperf client LAN1 LAN2

APU2 (AMD GX-412TC 1 GHz quad core - 4GB RAM - 3 Intel I210 NIC)

BIOS v4.13.0.3 mainline

AES-NI CPU Crypto: Yes (inactive) (same results if activated so this option has no effect)

System tested:

pfSense 2.4.5 (default)

pfSense 2.4.5 (modified)
/boot/loader.conf.local
legal.intel_igb.license_ack="1"
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"
Advanced -> tunables
net.inet.ip.redirect: 0
net.inet6.ip6.redirect: 0

pfSense 2.5.0 (default)

pfSense 2.5.0 (modified)
Advanced -> tunables
net.inet.ip.redirect: 0
net.inet6.ip6.redirect: 0

LAN1 to LAN2 (1 connection)

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 592 MBytes 497 Mbits/sec sender
[ 5] 0.00-10.00 sec 590 MBytes 495 Mbits/sec receiver

pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 589 MBytes 494 Mbits/sec sender
[ 5] 0.00-10.00 sec 587 MBytes 492 Mbits/sec receiver

pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 466 MBytes 391 Mbits/sec sender
[ 5] 0.00-10.00 sec 464 MBytes 389 Mbits/sec receiver

pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 466 MBytes 391 Mbits/sec sender
[ 5] 0.00-10.00 sec 465 MBytes 390 Mbits/sec receiver

LAN1 to LAN2 (10 connections)

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 13.3 MBytes 11.1 Mbits/sec sender
[ 5] 0.00-10.03 sec 12.0 MBytes 10.1 Mbits/sec receiver
[ 7] 0.00-10.00 sec 51.0 MBytes 42.8 Mbits/sec sender
[ 7] 0.00-10.03 sec 50.8 MBytes 42.5 Mbits/sec receiver
[ 9] 0.00-10.00 sec 326 MBytes 273 Mbits/sec sender
[ 9] 0.00-10.03 sec 325 MBytes 272 Mbits/sec receiver
[ 11] 0.00-10.00 sec 32.4 MBytes 27.2 Mbits/sec sender
[ 11] 0.00-10.03 sec 32.0 MBytes 26.8 Mbits/sec receiver
[ 13] 0.00-10.00 sec 3.24 MBytes 2.72 Mbits/sec sender
[ 13] 0.00-10.03 sec 3.06 MBytes 2.56 Mbits/sec receiver
[ 15] 0.00-10.00 sec 358 MBytes 300 Mbits/sec sender
[ 15] 0.00-10.03 sec 357 MBytes 299 Mbits/sec receiver
[ 17] 0.00-10.00 sec 121 MBytes 102 Mbits/sec sender
[ 17] 0.00-10.03 sec 121 MBytes 101 Mbits/sec receiver
[ 19] 0.00-10.00 sec 36.2 MBytes 30.4 Mbits/sec sender
[ 19] 0.00-10.03 sec 36.0 MBytes 30.1 Mbits/sec receiver
[ 21] 0.00-10.00 sec 41.4 MBytes 34.7 Mbits/sec sender
[ 21] 0.00-10.03 sec 40.8 MBytes 34.1 Mbits/sec receiver
[ 23] 0.00-10.00 sec 51.7 MBytes 43.3 Mbits/sec sender
[ 23] 0.00-10.03 sec 49.8 MBytes 41.6 Mbits/sec receiver
[SUM] 0.00-10.00 sec 1.01 GBytes 867 Mbits/sec sender
[SUM] 0.00-10.03 sec 1.00 GBytes 860 Mbits/sec receiver

pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 112 MBytes 93.7 Mbits/sec sender
[ 5] 0.00-10.01 sec 111 MBytes 93.0 Mbits/sec receiver
[ 7] 0.00-10.00 sec 2.25 MBytes 1.89 Mbits/sec sender
[ 7] 0.00-10.01 sec 382 KBytes 313 Kbits/sec receiver
[ 9] 0.00-10.00 sec 129 MBytes 108 Mbits/sec sender
[ 9] 0.00-10.01 sec 129 MBytes 108 Mbits/sec receiver
[ 11] 0.00-10.00 sec 2.82 MBytes 2.37 Mbits/sec sender
[ 11] 0.00-10.01 sec 1.24 MBytes 1.04 Mbits/sec receiver
[ 13] 0.00-10.00 sec 160 MBytes 134 Mbits/sec sender
[ 13] 0.00-10.01 sec 158 MBytes 133 Mbits/sec receiver
[ 15] 0.00-10.00 sec 3.25 MBytes 2.73 Mbits/sec sender
[ 15] 0.00-10.01 sec 1.25 MBytes 1.05 Mbits/sec receiver
[ 17] 0.00-10.00 sec 236 MBytes 198 Mbits/sec sender
[ 17] 0.00-10.01 sec 235 MBytes 197 Mbits/sec receiver
[ 19] 0.00-10.00 sec 240 MBytes 202 Mbits/sec sender
[ 19] 0.00-10.01 sec 239 MBytes 201 Mbits/sec receiver
[ 21] 0.00-10.00 sec 231 MBytes 194 Mbits/sec sender
[ 21] 0.00-10.01 sec 230 MBytes 192 Mbits/sec receiver
[ 23] 0.00-10.00 sec 2.22 MBytes 1.86 Mbits/sec sender
[ 23] 0.00-10.01 sec 349 KBytes 286 Kbits/sec receiver
[SUM] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec sender
[SUM] 0.00-10.01 sec 1.08 GBytes 926 Mbits/sec receiver

pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 110 MBytes 92.3 Mbits/sec sender
[ 5] 0.00-10.01 sec 109 MBytes 91.6 Mbits/sec receiver
[ 7] 0.00-10.00 sec 111 MBytes 93.4 Mbits/sec sender
[ 7] 0.00-10.01 sec 111 MBytes 93.1 Mbits/sec receiver
[ 9] 0.00-10.00 sec 113 MBytes 94.8 Mbits/sec sender
[ 9] 0.00-10.01 sec 113 MBytes 94.5 Mbits/sec receiver
[ 11] 0.00-10.00 sec 109 MBytes 91.7 Mbits/sec sender
[ 11] 0.00-10.01 sec 109 MBytes 91.1 Mbits/sec receiver
[ 13] 0.00-10.00 sec 113 MBytes 95.0 Mbits/sec sender
[ 13] 0.00-10.01 sec 113 MBytes 94.6 Mbits/sec receiver
[ 15] 0.00-10.00 sec 113 MBytes 95.0 Mbits/sec sender
[ 15] 0.00-10.01 sec 113 MBytes 94.7 Mbits/sec receiver
[ 17] 0.00-10.00 sec 113 MBytes 94.6 Mbits/sec sender
[ 17] 0.00-10.01 sec 112 MBytes 94.3 Mbits/sec receiver
[ 19] 0.00-10.00 sec 113 MBytes 95.1 Mbits/sec sender
[ 19] 0.00-10.01 sec 113 MBytes 94.8 Mbits/sec receiver
[ 21] 0.00-10.00 sec 112 MBytes 93.8 Mbits/sec sender
[ 21] 0.00-10.01 sec 111 MBytes 93.1 Mbits/sec receiver
[ 23] 0.00-10.00 sec 112 MBytes 93.7 Mbits/sec sender
[ 23] 0.00-10.01 sec 111 MBytes 93.4 Mbits/sec receiver
[SUM] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec sender
[SUM] 0.00-10.01 sec 1.09 GBytes 935 Mbits/sec receiver

pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 113 MBytes 94.6 Mbits/sec sender
[ 5] 0.00-10.01 sec 112 MBytes 94.1 Mbits/sec receiver
[ 7] 0.00-10.00 sec 96.3 MBytes 80.8 Mbits/sec sender
[ 7] 0.00-10.01 sec 95.5 MBytes 80.0 Mbits/sec receiver
[ 9] 0.00-10.00 sec 115 MBytes 96.5 Mbits/sec sender
[ 9] 0.00-10.01 sec 115 MBytes 96.2 Mbits/sec receiver
[ 11] 0.00-10.00 sec 126 MBytes 106 Mbits/sec sender
[ 11] 0.00-10.01 sec 126 MBytes 105 Mbits/sec receiver
[ 13] 0.00-10.00 sec 116 MBytes 97.0 Mbits/sec sender
[ 13] 0.00-10.01 sec 115 MBytes 96.4 Mbits/sec receiver
[ 15] 0.00-10.00 sec 93.7 MBytes 78.6 Mbits/sec sender
[ 15] 0.00-10.01 sec 93.0 MBytes 77.9 Mbits/sec receiver
[ 17] 0.00-10.00 sec 110 MBytes 92.5 Mbits/sec sender
[ 17] 0.00-10.01 sec 109 MBytes 91.8 Mbits/sec receiver
[ 19] 0.00-10.00 sec 105 MBytes 87.8 Mbits/sec sender
[ 19] 0.00-10.01 sec 104 MBytes 87.2 Mbits/sec receiver
[ 21] 0.00-10.00 sec 127 MBytes 107 Mbits/sec sender
[ 21] 0.00-10.01 sec 127 MBytes 106 Mbits/sec receiver
[ 23] 0.00-10.00 sec 116 MBytes 97.6 Mbits/sec sender
[ 23] 0.00-10.01 sec 116 MBytes 97.1 Mbits/sec receiver
[SUM] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec sender
[SUM] 0.00-10.01 sec 1.09 GBytes 932 Mbits/sec receiver

OpenVPN (P2P PSK L2L) AES128-CBC SHA1

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 108 MBytes 90.4 Mbits/sec sender
[ 5] 0.00-10.01 sec 108 MBytes 90.2 Mbits/sec receiver

pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 102 MBytes 85.9 Mbits/sec sender
[ 5] 0.00-10.01 sec 102 MBytes 85.6 Mbits/sec receiver

pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 94.1 MBytes 78.9 Mbits/sec sender
[ 5] 0.00-10.02 sec 93.9 MBytes 78.6 Mbits/sec receiver

pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 88.6 MBytes 74.4 Mbits/sec sender
[ 5] 0.00-10.02 sec 88.4 MBytes 74.0 Mbits/sec receiver

OpenVPN (P2P PSK L2L) AES256-CBC SHA256

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 89.2 MBytes 74.8 Mbits/sec sender
[ 5] 0.00-10.02 sec 89.0 MBytes 74.6 Mbits/sec receiver

pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 85.6 MBytes 71.8 Mbits/sec sender
[ 5] 0.00-10.02 sec 85.4 MBytes 71.5 Mbits/sec receiver

pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 78.1 MBytes 65.5 Mbits/sec sender
[ 5] 0.00-10.02 sec 78.0 MBytes 65.3 Mbits/sec receiver

pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 71.5 MBytes 60.0 Mbits/sec sender
[ 5] 0.00-10.02 sec 71.4 MBytes 59.7 Mbits/sec receiver

OpenVPN (P2P TLS L2L) AES128-GCM

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 133 MBytes 112 Mbits/sec sender
[ 5] 0.00-10.00 sec 133 MBytes 112 Mbits/sec receiver

pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 139 MBytes 116 Mbits/sec sender
[ 5] 0.00-10.00 sec 139 MBytes 116 Mbits/sec receiver

pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 115 MBytes 96.6 Mbits/sec sender
[ 5] 0.00-10.01 sec 115 MBytes 96.2 Mbits/sec receiver

pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 122 MBytes 102 Mbits/sec sender
[ 5] 0.00-10.01 sec 122 MBytes 102 Mbits/sec receiver

OpenVPN (P2P TLS L2L) AES256-GCM

pfSense 2.4.5 (default)
[ 5] 0.00-10.00 sec 119 MBytes 100 Mbits/sec sender
[ 5] 0.00-10.00 sec 119 MBytes 100 Mbits/sec receiver

pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 120 MBytes 101 Mbits/sec sender
[ 5] 0.00-10.00 sec 120 MBytes 100 Mbits/sec receiver

pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 101 MBytes 84.5 Mbits/sec sender
[ 5] 0.00-10.00 sec 101 MBytes 84.3 Mbits/sec receiver

pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 107 MBytes 89.5 Mbits/sec sender
[ 5] 0.00-10.01 sec 107 MBytes 89.4 Mbits/sec receiver

IPSEC VPN L2L AES128-CBC SHA1

Use asynchronous mode to parallelize multiple cryptography jobs = OFF

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 85.0 MBytes 71.3 Mbits/sec sender
[ 5] 0.00-10.14 sec 84.2 MBytes 69.6 Mbits/sec receiver

pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 80.6 MBytes 67.6 Mbits/sec sender
[ 5] 0.00-10.00 sec 78.6 MBytes 65.9 Mbits/sec receiver

pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 77.0 MBytes 64.6 Mbits/sec sender
[ 5] 0.00-10.00 sec 75.3 MBytes 63.2 Mbits/sec receiver

pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 79.8 MBytes 67.0 Mbits/sec sender
[ 5] 0.00-10.00 sec 77.9 MBytes 65.3 Mbits/sec receiver

Use asynchronous mode to parallelize multiple cryptography jobs = ON

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 198 MBytes 166 Mbits/sec sender
[ 5] 0.00-10.05 sec 198 MBytes 166 Mbits/sec receiver

pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 193 MBytes 162 Mbits/sec sender
[ 5] 0.00-10.03 sec 193 MBytes 161 Mbits/sec receiver

pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 65.2 MBytes 54.7 Mbits/sec sender
[ 5] 0.00-10.15 sec 65.2 MBytes 53.9 Mbits/sec receiver

pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 62.5 MBytes 52.4 Mbits/sec sender
[ 5] 0.00-10.14 sec 62.5 MBytes 51.7 Mbits/sec receiver

IPSEC VPN L2L AES256-CBC SHA256

Use asynchronous mode to parallelize multiple cryptography jobs = OFF

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 77.9 MBytes 65.3 Mbits/sec sender
[ 5] 0.00-10.08 sec 76.5 MBytes 63.7 Mbits/sec receiver

pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 73.8 MBytes 61.9 Mbits/sec sender
[ 5] 0.00-10.00 sec 71.8 MBytes 60.2 Mbits/sec receiver

pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 70.8 MBytes 59.4 Mbits/sec sender
[ 5] 0.00-10.00 sec 68.8 MBytes 57.7 Mbits/sec receiver

pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 74.6 MBytes 62.6 Mbits/sec sender
[ 5] 0.00-10.00 sec 73.4 MBytes 61.5 Mbits/sec receiver

Use asynchronous mode to parallelize multiple cryptography jobs = ON

pfSense 2.4.5 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 181 MBytes 151 Mbits/sec sender
[ 5] 0.00-10.05 sec 181 MBytes 151 Mbits/sec receiver

pfSense 2.4.5 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 182 MBytes 152 Mbits/sec sender
[ 5] 0.00-10.09 sec 182 MBytes 151 Mbits/sec receiver

pfSense 2.5.0 (default)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 55.9 MBytes 46.9 Mbits/sec sender
[ 5] 0.00-10.24 sec 55.9 MBytes 45.8 Mbits/sec receiver

pfSense 2.5.0 (modified)
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 55.7 MBytes 46.7 Mbits/sec sender
[ 5] 0.00-10.33 sec 55.7 MBytes 45.3 Mbits/sec receiver

Conclusions

Switching from 2.4.5 to 2.5.0 (on APU2 platform) leads to performances loss about 10%-25% in almost all tests done.

The heavy loss on IPSEC VPN with the option "Use asynchronous mode to parallelize multiple cryptography jobs"=enabled definitely needs some investigations.

I would be curious to know if the same situation occurs on other hardware thus I hope that someone will post some results.

Hmm, weird. I guess I'd try a packet capture. See if anything is actually coming in at all.
Other than dhcp broadcast requests which might be a clue.
If setting the pcap in promiscuous mode suddenly allows traffic to pass again it's some ARP/MAC problem.
The time span you mentioned does sounds quite ARP like...

Nope, do not try to update the driver. That's the current FreeBSD driver version for that kernel.

You could upgrade to 2.5 if you want newer drivers.

It shouldn't matter though.

Steve

@viragomann Thanks.

@stephenw10 Ok then I skip this part. Thanks.

Too many variables for me. 😉

I'd have to test that to know for sure.

Steve

@stephenw10 said in Can't get Huawei 4g modem working:

Ah, nice!

So you ran at AT command in Windows/Linux?

Steve

I ran the AT command on a Windows box and then moved the modem over to the pfsense box.

Hmm, interesting. SafeXcel doesn't do 3DES but it might be used for everything else there.