HAProxy LastChk fails with L4OUT (level4 timeout) to all backends located on the far side of IPSec Tunnel

[layla]

1. All backends on the near side of the IPSec Tunnel work properly.
   • The pfSense box that hosts the HAProxy's WebUI HAProxy Status Page
   • An ILO web page from a server directly attached to the local pfSense.
2. All backends on the far side of the IPSec Tunnel fail with L4OUT (level4 timeout).
3. All IP addresses resolve properly for all servers on each side of the IPSec Tunnel.

• All of these IP addresses can bi-direcitonally communicate with both pfSense firewalls
  • I can access the pfSense Web UIs on each firewall from either side of the tunnel..
  • I can ping the pfSense LAN IPs of each firewall from either side of the tunnel, from any of the servers.

4. The pfSense firewalls can both ping all of the server IPs in question properly from their respective LAN addresses.
5. The pfSense firewalls can ping each other, respectively.

For this reason, it seems like no static routes should need to be added.

The only things I can think of, though neither make much sense, are that maybe the HAProxy connections are attempting to initiate from 127.0.0.1? Or from the HAProxy firewall's PublicIP?

I tried making IPSec P2 entries to permit the HAProxy Firewall's localhost 127.0.0.1 and PublicIP to communicate with the far side of the IPSec tunnel and back, as nonsensical as this seems. This did not help, though it occurs to me that if this were the issue, further firewall rules might be necessary? But this feels like a rabbit hole.

Does anyone have any experience with this kind of issue? Any advice on how to make HAProxy be able to talk to servers on the other side of the IPSec tunnel?

Thanks in advance for any assistance you might have!

NOTE: This thread seemed to ask a similar/related question, but never had any resolution:
https://forum.netgate.com/topic/55964/ipsec-and-haproxy-on-the-fw-servers-on-the-other-side-of-the-tunnel

[layla]

Trying something really stupid, I seem to have solved it. This seems highly possibly to be a bug...

pi_dev_server2 is the pre-existing server backend, just renamed, and it uses the hostname as mentioned in my first post, which gets correctly resolved to 10.0.0.235 by HAProxy and uses port 80. There's no obvious reason why this doesn't work, but it doesn't, I've disabled it here for the test. But if you look at the original post, it fails with L4OUT.

Now all of the servers are working, as a result of that one change (?) or did something else suddenly start working? I'm not sure, but having pi_dev_server using an IP instead of a hostname seemed to make all of the others work properly, despite the fact that they are still using hostnames... Very very curious...