HAproxy slow on WAN jagged throughput

se4n_1

Hello all, happy Sunday. I upgraded the package to the -devel version and I still see the same behavior - no change. Just to recap:

These settings apply to my fronted:

Automaticaly generated, dont edit manually.

Generated on: 2020-08-30 19:10

global
maxconn 10000
log /var/run/log local0 info
stats socket /tmp/haproxy.socket level admin expose-fd listeners
uid 80
gid 80
nbproc 1
nbthread 4
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
server-state-file /tmp/haproxy_server_state
ssl-engine cryptodev
tune.ssl.cachesize 1000000

cache webcache
  total-max-size 256
  max-age 1800s

backend srv-frs_ipvANY
mode http
id 126
log global
# use mailers
# level err
email-alert mailers globalmailers
email-alert level err
email-alert from admin@yyy.com
email-alert to sysadmins@yyy.com
email-alert myhostname yyy.com
http-response set-header Strict-Transport-Security max-age=31536000;
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
option tcp-smart-connect
timeout check 5s
timeout tunnel 60000s
timeout connect 20s
timeout http-keep-alive 300s
timeout http-request 30s
timeout queue 20s
timeout server 50s
server srv-frs 10.192.3.54:80 id 127 check inter 10000 resolvers globalresolvers

frontend http_test
bind 94.103.xx.yy 80 name 94.103.xx.yy:80
mode http
log global
option http-keep-alive
timeout client 30000
use_backend srv-frs_ipvANY

I can see a download of around 1M/s via the proxy but if I NAT directly to the firewall in pfsense, I see a download of around 40M/s

Next step will be to create a virtual server for the proxy and take it off the firewall, that will be a big shame and I hope to avoid it!

dragoangel

@se4n_1 what you tried to get by:

cache webcache
  total-max-size 256
  max-age 1800s

This isn't correct part of haproxy config as far I know.
On other side I doesn't see any of uncommon conf which can lead to any performance issues in plain http. For better https I recommend change ssl-engine cryptodev to Intel RAND if CPU is support it. For investigate issues with plain http I already just to test recommended you host install any other NonApache backend, f.e.: nginx, iis, npm hserver, etc. and host one big file to test speed.

se4n_1

Performance is the same with nginx backend. I think this may be a hardware issue, these are official Netgate XG7100s though :/

As an additional test I also set up a vanilla Ubuntu 20.04 and apt installed haproxy package, dropped in the sample config, NATd a pfsense port to it, and I get downloads of 40M, so it does not appear to be the HAproxy config either.

I dont have any rate limiting/qos/1:1 NAT on this firewall. The only thing special is it has CARP failover and the monitor CARP interface option in HAproxy. Could this be an issue? Actually looking at your config in your header it seems we may be running similar setup, 2x XG7100 presumably with CARP failover and pfsync?

Adendum2:
If I go into maintenance mode and use the second XG7100 I also get the same 1M speed. Wow this is really frustrating! The problem seems very ethereal.

dragoangel

@se4n_1 I doesn't have carp (my pfsense are not in one place :P), but 100% this isn't case. I really recommend you write to netgate support as xg7100 must provide much more speed from haproxy (more then 1gbs).

se4n_1

Hello, so finally an update from me. Netgate and I tried everything we could think of but eventually suspicion fell on the ISP gateway router. I contacted the ISP and they did some tests for a while but eventually this weekend they replaced the gateway router with a new one and the speed issue has disappeared. I can now easily saturate the connection.

So in my case, this was a strange and not fully explained ISP issue that was handling traffic terminating on the WAN VIP differently to traffic NATd to LAN. Thanks for your assistance and sorry my answer will likely be of absolutely no use to anyone else.

dragoangel

@se4n_1 hi, actually your answer can help other people as it describes that ISP can also cause performance issues :)

S_m

I saw the post and redid a test on my side, and the same behaviour, not getting the throughput. I have to test again, but thad the same throtling on multiple ISP with different servers all with HAproxy...

dragoangel

@S_m not sure how you get this really. If I download file bigger than 1gb it easily take full bandwidth in my case.

S_m

@dragoangel I can get full bandwidth between VLANs, and saturate the link on the ISP side, but when I'm out of my ISP things jagged output.

I can have easy on my box with iPerfs or speedtest.org peaks of 950mbit download.. and upload.. So it's a mess not being able to use HA proxy to deliver high throughput.

dragoangel

@S_m Haproxy doesn't change any workflow comparing vlan and ISP, I think it obvious. And obvious where the issue located as problem start only at ISP.

S_m

@dragoangel the main issue is the jagged throughput only happens when traffic goes through HAproxy over internet. But if I put the traffic directly (port forward) goes perfectly fine at maximum speed.

dragoangel

@S_m what if you setup haproxy on Linux and nat it over pfsense?

S_m

@dragoangel after the first post and when COVID first started ... I tested another scenarios:
---> NAT nginx: better throughput!
---> HAproxy on Linux Server : Same behavour with other ISP
---> Another pfsense with HAproxy: same issues.

dragoangel

@S_m if you have same result on multiple os (freebsd/linux) and with multiple ISP then ask haproxy directly on their forum, but this really sound strange

S_m

Yes, I need to test this on another country, on Spain I get this bad results. Trying to diagnose what really happened, The post is because of the problem happened on my pfsense.

That's why having someone test this too would be great.

dragoangel

@S_m if you create post at haproxy community it will be good if you cross link posts to haproxy and from.

dragoangel

@S_m do you tried http/2? In theory it not help on one big file but still. Also you can try something like loader.io