HAproxy slow on WAN jagged throughput
-
@se4n_1 np, but issue with performance that you described is strange. Try install any nginx behind haproxy and return static 10gb file over http and then over https. If you said you have issues with ssl offloading then I recommend you dig to you dig in aes-ni and vcpu, but you saing that you have issues even with plain http, it really strange.
-
I have the devel version on pfsense. I can get full Gigabit without issues in my ISP and routing myself between VLANs.
The problem is the moment I get out of the ISP. I think with COVID-19 some throttling could been happening, but I got this issue before...
I tested with an installation of HAproxy of other user on Linux and go the same results.
I have to test further, I've got on my side Wireshark captures.
EDIT:
I just did a quick test, with a burst of 500Mbit on the client and then throttling.
-
For sure I see throttling, and bad peering. That's why I had a NAT setup so I could compare downloading the Ubuntu 20.04 image directly from apache2 and the same image via a http (no SSL) frontend to keep things as simple as possible. In principle there are only trivial differences between the packets coming and going through the NAT and HAproxy so I was esentially trying to pin the blame on HAproxy. It seems most people are using the devel package so I will give that a go, I am bullish on the chances of success given this seems very much a minority issue.
-
@se4n_1 did you use
option http-keep-alive? -
Yes I tried with both, and all of the other keepalive and timeout settings - makes no big difference - I also tried the different closing modes and tunnel mode and a TCP connection instead of http
-
Hello all, happy Sunday. I upgraded the package to the -devel version and I still see the same behavior - no change. Just to recap:
- These settings apply to my fronted:
Automaticaly generated, dont edit manually.
Generated on: 2020-08-30 19:10
global
maxconn 10000
log /var/run/log local0 info
stats socket /tmp/haproxy.socket level admin expose-fd listeners
uid 80
gid 80
nbproc 1
nbthread 4
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
server-state-file /tmp/haproxy_server_state
ssl-engine cryptodev
tune.ssl.cachesize 1000000cache webcache total-max-size 256 max-age 1800sbackend srv-frs_ipvANY
mode http
id 126
log global
# use mailers
# level err
email-alert mailers globalmailers
email-alert level err
email-alert from admin@yyy.com
email-alert to sysadmins@yyy.com
email-alert myhostname yyy.com
http-response set-header Strict-Transport-Security max-age=31536000;
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
option tcp-smart-connect
timeout check 5s
timeout tunnel 60000s
timeout connect 20s
timeout http-keep-alive 300s
timeout http-request 30s
timeout queue 20s
timeout server 50s
server srv-frs 10.192.3.54:80 id 127 check inter 10000 resolvers globalresolversfrontend http_test
bind 94.103.xx.yy 80 name 94.103.xx.yy:80
mode http
log global
option http-keep-alive
timeout client 30000
use_backend srv-frs_ipvANYI can see a download of around 1M/s via the proxy but if I NAT directly to the firewall in pfsense, I see a download of around 40M/s
Next step will be to create a virtual server for the proxy and take it off the firewall, that will be a big shame and I hope to avoid it!
-
@se4n_1 what you tried to get by:
cache webcache total-max-size 256 max-age 1800sThis isn't correct part of haproxy config as far I know.
On other side I doesn't see any of uncommon conf which can lead to any performance issues in plain http. For better https I recommend changessl-engine cryptodevto Intel RAND if CPU is support it. For investigate issues with plain http I already just to test recommended you host install any other NonApache backend, f.e.: nginx, iis, npm hserver, etc. and host one big file to test speed. -
Performance is the same with nginx backend. I think this may be a hardware issue, these are official Netgate XG7100s though :/
As an additional test I also set up a vanilla Ubuntu 20.04 and apt installed haproxy package, dropped in the sample config, NATd a pfsense port to it, and I get downloads of 40M, so it does not appear to be the HAproxy config either.
I dont have any rate limiting/qos/1:1 NAT on this firewall. The only thing special is it has CARP failover and the monitor CARP interface option in HAproxy. Could this be an issue? Actually looking at your config in your header it seems we may be running similar setup, 2x XG7100 presumably with CARP failover and pfsync?
Adendum2:
If I go into maintenance mode and use the second XG7100 I also get the same 1M speed. Wow this is really frustrating! The problem seems very ethereal. -
@se4n_1 I doesn't have carp (my pfsense are not in one place :P), but 100% this isn't case. I really recommend you write to netgate support as xg7100 must provide much more speed from haproxy (more then 1gbs).
-
Hello, so finally an update from me. Netgate and I tried everything we could think of but eventually suspicion fell on the ISP gateway router. I contacted the ISP and they did some tests for a while but eventually this weekend they replaced the gateway router with a new one and the speed issue has disappeared. I can now easily saturate the connection.
So in my case, this was a strange and not fully explained ISP issue that was handling traffic terminating on the WAN VIP differently to traffic NATd to LAN. Thanks for your assistance and sorry my answer will likely be of absolutely no use to anyone else.
-
@se4n_1 hi, actually your answer can help other people as it describes that ISP can also cause performance issues :)
-
I saw the post and redid a test on my side, and the same behaviour, not getting the throughput. I have to test again, but thad the same throtling on multiple ISP with different servers all with HAproxy...
-
@S_m not sure how you get this really. If I download file bigger than 1gb it easily take full bandwidth in my case.
-
@dragoangel I can get full bandwidth between VLANs, and saturate the link on the ISP side, but when I'm out of my ISP things jagged output.
I can have easy on my box with iPerfs or speedtest.org peaks of 950mbit download.. and upload.. So it's a mess not being able to use HA proxy to deliver high throughput.
-
@S_m Haproxy doesn't change any workflow comparing vlan and ISP, I think it obvious. And obvious where the issue located as problem start only at ISP.
-
@dragoangel the main issue is the jagged throughput only happens when traffic goes through HAproxy over internet. But if I put the traffic directly (port forward) goes perfectly fine at maximum speed.
-
@S_m what if you setup haproxy on Linux and nat it over pfsense?
-
@dragoangel after the first post and when COVID first started ... I tested another scenarios:
---> NAT nginx: better throughput!
---> HAproxy on Linux Server : Same behavour with other ISP
---> Another pfsense with HAproxy: same issues. -
@S_m if you have same result on multiple os (freebsd/linux) and with multiple ISP then ask haproxy directly on their forum, but this really sound strange
-
Yes, I need to test this on another country, on Spain I get this bad results. Trying to diagnose what really happened, The post is because of the problem happened on my pfsense.
That's why having someone test this too would be great.
