HAproxy slow on WAN jagged throughput

dragoangel

@se4n_1 what you tried to get by:

cache webcache
  total-max-size 256
  max-age 1800s

This isn't correct part of haproxy config as far I know.
On other side I doesn't see any of uncommon conf which can lead to any performance issues in plain http. For better https I recommend change ssl-engine cryptodev to Intel RAND if CPU is support it. For investigate issues with plain http I already just to test recommended you host install any other NonApache backend, f.e.: nginx, iis, npm hserver, etc. and host one big file to test speed.

se4n_1

Performance is the same with nginx backend. I think this may be a hardware issue, these are official Netgate XG7100s though :/

As an additional test I also set up a vanilla Ubuntu 20.04 and apt installed haproxy package, dropped in the sample config, NATd a pfsense port to it, and I get downloads of 40M, so it does not appear to be the HAproxy config either.

I dont have any rate limiting/qos/1:1 NAT on this firewall. The only thing special is it has CARP failover and the monitor CARP interface option in HAproxy. Could this be an issue? Actually looking at your config in your header it seems we may be running similar setup, 2x XG7100 presumably with CARP failover and pfsync?

Adendum2:
If I go into maintenance mode and use the second XG7100 I also get the same 1M speed. Wow this is really frustrating! The problem seems very ethereal.

dragoangel

@se4n_1 I doesn't have carp (my pfsense are not in one place :P), but 100% this isn't case. I really recommend you write to netgate support as xg7100 must provide much more speed from haproxy (more then 1gbs).

se4n_1

Hello, so finally an update from me. Netgate and I tried everything we could think of but eventually suspicion fell on the ISP gateway router. I contacted the ISP and they did some tests for a while but eventually this weekend they replaced the gateway router with a new one and the speed issue has disappeared. I can now easily saturate the connection.

So in my case, this was a strange and not fully explained ISP issue that was handling traffic terminating on the WAN VIP differently to traffic NATd to LAN. Thanks for your assistance and sorry my answer will likely be of absolutely no use to anyone else.

dragoangel

@se4n_1 hi, actually your answer can help other people as it describes that ISP can also cause performance issues :)

S_m

I saw the post and redid a test on my side, and the same behaviour, not getting the throughput. I have to test again, but thad the same throtling on multiple ISP with different servers all with HAproxy...

dragoangel

@S_m not sure how you get this really. If I download file bigger than 1gb it easily take full bandwidth in my case.

S_m

@dragoangel I can get full bandwidth between VLANs, and saturate the link on the ISP side, but when I'm out of my ISP things jagged output.

I can have easy on my box with iPerfs or speedtest.org peaks of 950mbit download.. and upload.. So it's a mess not being able to use HA proxy to deliver high throughput.

dragoangel

@S_m Haproxy doesn't change any workflow comparing vlan and ISP, I think it obvious. And obvious where the issue located as problem start only at ISP.

S_m

@dragoangel the main issue is the jagged throughput only happens when traffic goes through HAproxy over internet. But if I put the traffic directly (port forward) goes perfectly fine at maximum speed.

dragoangel

@S_m what if you setup haproxy on Linux and nat it over pfsense?

S_m

@dragoangel after the first post and when COVID first started ... I tested another scenarios:
---> NAT nginx: better throughput!
---> HAproxy on Linux Server : Same behavour with other ISP
---> Another pfsense with HAproxy: same issues.

dragoangel

@S_m if you have same result on multiple os (freebsd/linux) and with multiple ISP then ask haproxy directly on their forum, but this really sound strange

S_m

Yes, I need to test this on another country, on Spain I get this bad results. Trying to diagnose what really happened, The post is because of the problem happened on my pfsense.

That's why having someone test this too would be great.

dragoangel

@S_m if you create post at haproxy community it will be good if you cross link posts to haproxy and from.

dragoangel

@S_m do you tried http/2? In theory it not help on one big file but still. Also you can try something like loader.io