Any known issues with HAproxy on 2.5.2?

stephenw10

Yes, just be aware it would be very easy to introduce asymmetry and I've seen that bite people many, many times!

If you really are seeing an issue in HAProxy then a pcap should prove it.

I would expect to see something logged though.

Was this working in the old network setup?

Steve

lewis

We had the proxy going for the past couple of years approximately.
During that time, we've had lots of complaints about 500/504 but always blamed our own resources, never once thinking it could be the proxy.

So to answer your question, there is really no way to know other than when I posted this, that was around the time we realized what was happening.

We had taken the proxy out of the mix to do some testing so it was off for maybe a week. Then when we re-enabled it, the timeout complaints started again which got me wondering what was going on. That's when I disabled it again and since then, the complaints stopped and we too were no longer getting them.

We know one problem was a back end one in that there was an issue with the database and it wasn't responding fast enough causing 504's but we were aware of those and could see them in the logs.

stephenw10

If something is responding with a 500 or 504 error that will be logged somewhere. That's not just a failure to respond at all. If that's HAProxt responding with that then I'd expect to see some other errors logged.

lewis

What I mean is that we know about the 500/504 errors because we see them on the LAN side when we have problems.

However, when users get them because they cannot reach the site, there aren't any errors that we've logged because we simply didn't think it was the load balancer.

We would have to set up some kind of test to see if we can log but that will take a little time. We ended up upgrading a bunch of things, adding hardware, the multi-firewall thing and so on. Since we could not find the problem, we simply blamed ourselves after weeks of searching.

It all got better yesterday when I removed the last server from the proxy.

Now I'm more concerned about this segfault thing I'm seeing and not being able to upgrade. That feels like imminent failure to me.

stephenw10

There's nothing in the system log following the upgrade attempt?

lewis

This post is deleted!

lewis

Ah, here we are.

May 16 21:31:06 sshd 79909 Accepted keyboard-interactive/pam for root from x.x.x.x port xxx ssh2
May 16 21:31:20 kernel pid 59117 (pkg-static), jid 0, uid 0: exited on signal 11 (core dumped)
May 16 21:31:26 kernel pid 87625 (pkg-static), jid 0, uid 0: exited on signal 11 (core dumped)

Reboot will be required!!
Proceed with upgrade? (y/N) y
>>> Removing vital flag from php74... done.
>>> Downloading upgrade packages...
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (201 candidates): ....
Child process pid=87625 terminated abnormally: Segmentation fault
pfSense - Netgate Device ID: xxx

Unrelated?
May 13 08:00:32 php-fpm 72646 /services_dhcp_edit.php: The command '/usr/sbin/arp -d '10.0.0.100'' returned exit code '1', the output was 'arp: writing to routing socket: No such file or directory'

I'm feeling a little nervous that this firewall is going to crash at some point.

stephenw10

The arp log is unrelated, something trying to remove an ARP entry that's already been removed.
That is unusual though. Do you have static ARP entries set?

Do you have Zabbix Agent installed? Specifically the obsolete 5_2 version?
If so you are probably hitting this: https://redmine.pfsense.org/issues/12796

Removing that before the upgrade should allow it.

Steve

lewis

I showed the arp log part because it's having a problem doing that which makes me nervous that the os might be getting messed up or something.

I do have static MAC/IP entries in the DHCP server. It's how I keep track of all the equipment. If first gets a DHCP IP which is how I identify it on the network so I enter a static entry into the DHCP server.

Yes, zabbix 5.2 is installed on this firewall. Removed.
The haproxy was a little out of date so that's updated now.

I'll try running the upgrade later today and see how it goes.

stephenw10

Static DHCP mappings are not the same as static ARP entries. You can enable static ARP on static dhcp mappings but it's almost always unnecessary and can cause problems.

https://docs.netgate.com/pfsense/en/latest/services/dhcp/ipv4.html#static-mappings

Steve

lewis

@stephenw10

Understood. Just saying I don't have any static ARP, just DHCP mappings I maintain.

I'll try the upgrade again tonight I hope.

lewis

Well, that worked, thanks so much. Feels a bit better seeing it upgrade and upgraded.

No idea how I'm going to test the proxy as I've decided to do something different. Have not gone back to it since finding the problem.