HAProxy stopped accepting connections



  • It was OK, I checked constantly because I'm working on a little project, went to bed and I noticed one of the servers didn't respond. I went absolutely medieval on the thing, broke a million things instead of checking out if it was an isolated incident first...turns out it wasn't. I lost about 3w worth of data.

    Oh well, it's done. Now I'm positive it's HAProxy but the settings are all OK, certificates are valid I am running out of ideas.

    0_1540083080124_Screen_Shot_2018-10-20_at_18_25_25.png

    At first it quickly broke connections, or that might've been that the server I thought was bad, actually was; right after HAProxyt there's actually another proxy and that one seems alright, I hooked it up to the edge temporarily while I I try to sort this out.

    Suricata's been going a little cray these days, I'm starting to think someone may have come through. I get a lot of Asian and Eastern European IP addresses that the minute on of my addresses gets a DNS name it's on. 🤘🏼

    Is a common bug? If I maybe get around it by installing the other HAProxy version? Thanks !



  • @umademelosemyusernamepfsense said in HAProxy stopped accepting connections:

    the other HAProxy version

    What version do you use (exactly) ? Sure trying 'the other' is an option, as long as you don't need features that only exist in the newer version.?.

    But well there isn't much info to go on here. Sharing some stats/logs/configuration could all be helpful..



  • @piba It seems related to certificates, but, the whole system is acting weird, again related to certificates. I checked if they're OK and they are. I reinstalled the CAs and, issued new certificates and it still wouldn't work.

    Apart from that, system 2.4.4 starts to slow down and starts losing packets, I see tons of red and black from retransmissions in captures. I don't know how to make sense of packet captures myself, but, I know that's not supposed to happen on a working network. What I do to correct is not reinstall the system, but not using the console but actually deleting the disk and reinstalling from scratch. Restoring to factory won't fix issues; I found some commands, though, that apparently are just for that but they aren't for system 2.4.4. Modifying them a little go them working and the system seemed alright again, after restoring a backup back from September I just needed to delete an VPN interface before testing again HAProxy, and when I did the system hung again. So, after all this, it seems very likely HAProxy isn't the culprit but until I get a working stable system and back the hell out of it, I can't test again. :/

    Before 2.4.4 I could change and change and change stuff and it would route on for months, no biggie, but now it appears that changes affect something, maybe a database or something. "Storage!" I thought, maybe some disk-access/speed or file system issue, but the disk is local, flash-based, the system has plenty memory and I've gone back and forth between the ultra-resilient ZFS and U..HF..--something, the old one, and it doesn't seem to make a difference.

    Sorry for the missing details, though. I'll keep testing.