Load Balancer service allowing connections to disabled server
-
We have out Load Balancer service setup to hit our 3 nodes in round robin. Every so often I will take one node out to do some work on it. After unchecking/disabling the node, I hit save and the service effectively restarts. It appears that the service goes offline and then come back online. Leaving a small window of downtime (3-5 secs) and all is well. I am OK with this. I have accepted this as a norm and am not worried about it.
Yesterday we had only node 1 and node 2 enabled in the cluster. node 3 has been disabled for a few weeks while I have been diagnosing an issue.
We have lots of traffic hitting our site that records time punches, etc.
Again, node 3 is disabled.
When reloading the service (taking out node 1 for example), for a very brief moment, node 1,2,3 are all able to be hit from the outside. We see time punches with a server fingerprint from the 3rd node! that this has been disabled for a few weeks. How would it allow connections through?
It's very repeatable. Happens every time I hit the load balancer, from the outside, through the gateway,etc. while reloading the service.