HAProxy Caused Total Network Outage - Dissecting What Went Wrong
-
Hello,
I have been going through the process of setting up HAProxy on my network, and I did something that caused a complete network outage, looking for some input on what exactly went wrong, so I can hopefully avoid similar issues in future.
Some pretext, cert generated in ACME, backend created, I had one frontend created and listening on WAN:443, SSL offloading, some basic ACLs, nothing fancy, I was making a secondary frontend for a backend that was meant to serve the internal traffic, I duplicated the original ACL, and heres where I beleive my mistake happened, I arbitrarily set the listen IP to "10.8.40.10:443", I do have a /24 interface in 10.8.40.x, and theres stuff in there, but there shouldn't have been any address collisions, rules, or anything in that interface/subnet regarding that IP, immedietly after applying my webUI locked up, and the entire router stopped responding on all interfaces. I locally KVM-ed into the router and nothing seemed out of the ordinary, I didnt get a chance to try pinging out of the router but nothing client side on any of the subnets was able to ping into the router, as it normally would have been able to; I attempted rolling back using the automatic last 30 changes feature, rebooted, and it didnt fix the issue, I went into the shell and manually uninstalled HAProxy, rebooted, and it didnt fix the issue, I ended up having to fully factory reset the box, and I'm just now starting to restore connectivity to everything, but I'm having to start all over from scratch.
What went wrong here? I'm at a total loss for what could cause such a catastrophic failure of the entire router from what was maybe one mistake in HAProxy? I don't even fully understand what issue I could have created, could I possibly have needed to create a VIP first?