503s on non-offloaded backends

[senseivita]

I'm getting 503s on some HTTP backends, not all. And, when I setup SNI, no backend works except the loopback to the offloading frontend.

If that wasn't clear:
HTTP/80 - some backends work, some don't -- both working and non-working backends have green health checks
HTTP-SNI/443 - nothing works -- all health checks are green
Offloading frontends/backends -- everything works perfectly -- all health checks are green

Turning off the health check doesn't make a difference. :(

Where can I get the logs from HAProxy? I want to try to fix it. Thanks!

[senseivita]

Since I wrote this I kept testing and discovered that there's something wrong with the software itself--I think; I've been using de dev version (haproxy18-1.8.23-ish) since forever so I thought it was my own fault for not using the official one, but, I downgraded to the official version (haproxy17-1.7.12-ish) and it got worse.

Now neither TLS termination/offloading nor SNI work. It shows something about the data not being complete:

Like if it were being corrupted somewhere. I tried different connections to the same result. I thought, maybe other tools like Suricata and ntopng were getting in the way but disabling them (and clearing the states) made no diff.

I wanted to send logs to help out devs but I have none. I forgot to set them. My bad. :)

When I switched back to the dev version things got working again but I've seen this tends to last like for a little while only. I've also observed that on the SNI front when all backends inevitably fail, the loopback backend (for the offloading front) is the only backend that works--as I mentioned earlier, offloading and http work fine.

I'll set up a logging server for the next time. :)