Server Load Balancer showing red, but appears to be working?

[snm777]

pfsense version 2.1.5
Packages installed:
Open-VM-Tools
Postfix Forwarder

The short version of my problem is that I have a load balancer configured with three servers in the pool, pfsense shows all three as down, but in fact it appears that the load balancing is working.  The Load Balancer logs only every have entries for the pool servers when relayd is restarted  - it does not appear to be logging any checks it does after restart, if it is checking.

I have tried setting up both a TCP and an ICMP monitor, I have verified that I see a three way TCP session established between the LAN interface of the pfsense and the pool servers, I have verified that the pfsense can ping the pool servers from the LAN interface, and every time I restart realyd, I get these messages in the Load Balancer log ( I sanitized the IP's)
Nov 3 14:47:48 relayd[54690]: host 184.x.x.1, check icmp (0ms), state unknown -> up, availability 100.00%
Nov 3 14:47:48 relayd[54690]: host 184.x.x.2, check icmp (0ms), state unknown -> up, availability 100.00%
Nov 3 14:47:48 relayd[54690]: host 184.x.x.3, check icmp (0ms), state unknown -> up, availability 100.00%
or this, if I have the check set to TCP:
Nov 3 15:19:11 relayd[30858]: host 184.x.x.1, check tcp (1ms), state unknown -> up, availability 100.00%
Nov 3 15:19:11 relayd[30858]: host 184.x.x.2, check tcp (1ms), state unknown -> up, availability 100.00%
Nov 3 15:19:11 relayd[30858]: host 184.x.x.3, check tcp (1ms), state unknown -> up, availability 100.00%

Just now discoverd that a few minutes after resarting relayd on pfsense, we lost access to the pfsense webconfigurator and packets to the load balancer dropped.  Had to reboot the pfsense firewall to get it all back.

About the setup:

here is a rough text flow of how this is set up:
<client on="" internet="">–---Request to public VIP-----[Router]–---Internet-----[Router]–---{WAN}pfsense hosting VIP, loadbanalcer setup{LAN}-----[Router]–---Subnet with the 3 pool servers

I have a load balancer setup with 3 servers in the pool, and doing a TCP check on a specific port, let's say port 11114 in this example.
My WAN has a publicly routeable IP, as does my LAN interface.  NAT is set to manual and there are no NAT rules so no NAT in any direction.

I have a Virtual Server set up in Firewall -> Virtual IP's, the IP is on the same subnet as the WAN IP, we DO have routing set up so that services on the Internet successfully reach the VIP.
The SAME IP address is configured udner Services -> Load Balancer -> Virtual Servers.  The specific port, 11114 is listed here too, as is my pool of IP's in the Virtual Server Pool, there is no Fall Back Pool, and the Relay Protocol is TCP.

If I go to Services -> Load Balancer -> Pools, all three of my backend servers are configured.  Now, the IP's of these servers are beyond a gateway on the LAN interface.  Pfsense is configured with the correct gateway and it has a route for the network these servers reside on, and it works.  The port, 11114, is configured here too.

We actually have TWO pfsense firewalls set up with this configuration, one in one city, another in another city about 250 miles away.  The only differences between the two are the IP addresses - rules, packages, pfsense version, all the same. Both exhibit the same strange issue.  What else can I look at or do to troubleshoot this?  Any additional logs I can post to give more information?</client>

[cmb]

What does the output of "relayctl show summary" look like?

[arrmo]

Hi,

Did you ever get an answer to this? FYI, I had the same question … all when green when you actually add the Pool to a Virtual Server (until then, no checks it seems -> stays red).