HAProxy issue: Frontend works on 51443, not on 443

sgw

I have an issue that took me a few hours yesterday and I haven't yet found the solution:

A customer ran a pfSense-CE-2.8.1 in a VM. Rather basic setup: WAN, LAN, some VLANs tagged on LAN. A HAproxy providing access to a few services, among them a gitlab VM, reachable via https://gitlab.some.tld

The WAN interface is plugged into a Fritzbox, german users know these ... I know, I would prefer to get rid of that FB, but the customer insists and needs it for VOIP purposes. The FB has a feature called "Exposed Host", which forwards all ports to a defined IP behind it, in our case the WAN iface of the pfSense.

Last week I migrated the VM to a physical appliance (there is a major overhaul taking place, new switches, firewall, server etc). I adjusted the interfaces etc and everything worked, including the web services behind HAproxy. Especially gitlab.

Sorry for making this too long ...

Yesterday morning I noticed gitlab was unreachable.

What I see:

when I telnet the WAN-IP on port 443 I get no reply
surfing or curl-ing https://gitlab.some.tld gives no reply
the DNS record is fine
the firewall allows WAN:443
I see traffic on the pfsense related to port 443 and my client IP (packet dump)
I see a haproxy-socket on the pfsense on port 443
I copied the frontend and modified it to port 51443 (plus fw allow): https://gitlab.some.tld:51443 works! (so the backend is OK also)

so the frontend for 443 is somehow dead (?)

I checked for NAT-rules or something that occupy 443, nothing (as far as I see). Reloaded filter rules.

I am definitely sure that this worked last week after the migration to the hardware. What do I miss, what can I do to pinpoint this issue?

So far I have avoided a plain reboot, maybe this would help, although it should be resolvable without that as well, right?

thanks for any ideas here ...

Gertjan

@sgw said in HAProxy issue: Frontend works on 51443, not on 443:

so the frontend for 443 is somehow dead (?)

When I ask my pfSense who/waht uses port 443 (on the WAN), I get this :

[25.07.1-RELEASE][root@pfSense.bhf.tld]/root: sockstat | grep '443'
root     lighttpd_p 83774 4   tcp4   10.10.10.1:443        *:*
root     lighttpd_p 83774 6   tcp6   ::10.10.10.1:443      *:*
root     nginx      28252 5   tcp4   *:443                 *:*
root     nginx      28252 6   tcp6   *:443                 *:*
root     nginx      28139 5   tcp4   *:443                 *:*
root     nginx      28139 6   tcp6   *:443                 *:*
root     nginx      27732 5   tcp4   *:443                 *:*
root     nginx      27732 6   tcp6   *:443                 *:*
....

and guess what : I don't have HAProxy installed.
Which means some one else is listening on that 443 TCP port, on every known pfSense interface, for IPv4 and IPv6.
It's the GUI web server.
Did you move it out of the way ?

sgw

@Gertjan good point, but not the solution ;-)
I didn't mention / forgot it: yes, the webGUI is on 50443 for a long time already.
Otherwise the HAproxy-FE would never have worked.

So unfortunately this is not my problem.

it looks like this:

172.20.0.2 is the WAN IP, sure. The various ports belong to various HAproxy-Frontends.

sgw

mysteriously works again after a gitlab-upgrade. strange ...