I was with a feedback as you like.

Thanks for the reply, Derelict. Sorry for my slow reply. I went in for surgery, and am just back on my feet again. It is indeed unusual. I can see the connection in the state table of the master node, with TIME_WAIT:TIME_WAIT, and 9/4 packets, but the browser tells me the connection was reset, and indeed telnet to the management port is denied as well. ARP of the pfSense VIP is correct on the pinging machine, and ARP of the pinging machine is correct on the pfSense box. It seems like pfSense is blocking the connections, even though it's been told to allow them through (my allow rule uses an alias that includes the VIP and both real IPs of the pfSense boxes). Very, very strange. I'll walk through the Network Connectivity process and see if it turns up anything.

In a last ditch effort to get things running again, I blew reset the config on FW2 and started over.  Since this is an HA pair, I just did the initial setup and had FW1 sync over the settings.  This seems to have fixed the problem.  The secondary FW is in BACKUP mode and the traffic is very minor (16KB/sec). Not sure what happened, but something must have gone wrong during the upgrade from 2.2.6 to 2.3.2.  I might consider doing the same on FW1 (reset to factory then sync from FW2). Thanks again for helping out!

How is your Outbound NAT configured?  I had a similar problem whereby my Outbound NAT was tied to the interface and not the VIP.  Here is what I did: Firewall -> NAT Click Outbound NAT Make sure your "NAT Address" for the LAN subnet is tied to the external CARP IP address instead of interface address Hope this helps.

Just a quick follow-up: we did our FW exchange last night and - at least on the CARP side - things went as expected -> VHIDs seem to work and failover just fine.

Thanks for the help! much appreciated. As I mentioned on the first post there is a toggle to ignore suspicious Mac but it does not produces any result. As it stands, I was forced to remove both drayteks from the network and plug the wans directly into one of my PFSENSE boxes, losing wan redundancy in the process. I am preparing two mini-ITX pc's do replace the drayteks. One for each wan, also running Pfsense, so I can regain wan redundancy on both pfsense boxes. cheers

It's a HP 2810-48G J9022A pfsense is connected to port 23. Should I select MODE tagged there? [image: switch.PNG] [image: switch.PNG_thumb]

@Francesco: That's fine. You weren't reading the details, exaggerating information needed and making things more difficult than they really are. No, he was trying to point out that you could have a dozen different things wrong with your lab setup which no one can easily sort out. e.g. something on the WAN side by default can't ping your LAN, so the fact that 77 whatever can't ping 66 whatever is probably irrelevant; the 'isp router' config is unknown, etc…  I'm not sure what you are trying to test with your methodology either, someone unplugging the WAN on the master seems an unlikely event. The HA failures I've dealt with usually involve failed hardware. If I was going to test, I'd pull power on the master and see what happens. Anyway, you are seeking free assistance from strangers on the Internet. If you don't want to work with someone who steps up, fine, but don't be offended if no one else wants to spend time trying to figure out what's wrong with your setup.

Hmm. A /29 is not 8 usable IP addresses unless it's routed to you. They kind of need to get their act together. Nope they don't. A pity but quite a few ISPs or Hosting Providers will give you 8 IPs but not route them in a clean way. Either some hack'n'slash P2P Host Routing is done or you get 8 single IPs from different segments. No one said those 8 addresses are from the same block. I know quite a few german (big) hosting companies working that way and it is annoying as hell from a networking perspective. So I won't get my hopes up until I read someone cleanly stating that it actually is a /29 IP block.

Thanks, that is really helpful.  I was looking at two providers on a single firewall replicated on both firewalls.  That is to say, I have a two distinct failure modes 1 - WAN failure (ie ISP is hacked and brought down for example), auto switch to backup provider from within the same (active) firewall [this is my present config] 2 - firewall failure, autoswitch to backup firewall for normal service. [this is what I hope to add to my failure redundancy] I will go get the book. Thank you again.

Just wondering if you ever got this solution working or not. We just stood up a pfSense box to replace our Brocade router that connects us to our 1 ISP via BGP. I think you have to put in a "neighbor" before your connection to your ISP will work, but I'm not 100% sure. Here is my config (IP addresses substituted to hide my real ones): AS 12345 fib-update yes holdtime 30 listen on 0.0.0.0 router-id 1.2.3.4 network 100.200.100.0/24 neighbor 1.2.3.3 { descr "ISP1" remote-as 4321 softreconfig in yes  local-address 1.2.3.4 } deny from any deny to any allow from 1.2.3.3 allow to 1.2.3.3 P.S. We originally had two ISP connections, two Brocade routers (each cost $20,000!!), and two Sonicwalls in HA mode (cost for 5 year lease on the Sonicwalls was $20,000). Now: We only have one ISP connection. I replaced the Sonicwalls with two Netgate C2758 pfSense boxes in HA setup for under $4,000. Our Brocade routers became obsolete because they can't be upgraded to handle today's full Internet routes size, so I used an old 1U server with pfSense & OpenBGP to replace them.

It is TCP now. TCP is unicast only. That won't work. I doubt it will be converted to anything that would support Multicast or broadcast, it's not meant to work that way. Eventually there will be a central management system that will make those kinds of hacks completely unnecessary.

Many, many thanks for this hint. :) I have been trying to figure out what went wrong, when i set the carp-IPs via developer shell. If the uniqid is not set you will not be able to set the Interface within RA-Advertisments. The address from which it should be send (e.g. LAN, CARP-IP) is simply missing. (2.3.2)

Interface Names & Order are identical in Status > Interfaces. I've tried both CARP and interface addresses and neither the CARP nor the Master responds to pings or is in the ARP cache. Nothing on the consoles either, other than login events. Your last paragraph mentions something that puzzles me too. The backup doesn't take over or detect the master is down. However when tested by disabling CARP on the Master the backup does take over. After rebooting the Master this morning I see a reconfiguration occurred on Saturday afternoon. That's the only entry between the reboots on Friday morning and this morning. I'll try disabling the firewall rule sync to see if that helps.

I am also experiencing this issue after I upgraded 4 boxes running various levels of 2.3.1 up to 2.3.2. Same environment: pfsense on ESXi 5.5/6.0 hosts. On my end though, It might be an issue with how Suricata is using the XMLRPC sync? Appears to be breaking once a rules refresh is complete. Suricata is setup to NOT ask the target slave to refresh their own rules. Haven't had any XMLRPC errors from the other pair of 2.3.2 VMs that DO NOT have Suricata Sep 7 00:31:49 php-fpm 7073 /rc.filter_synchronize: New alert found: A communications error occurred while attempting XMLRPC sync with username admin https://192.168.254.2:443. Sep 7 00:31:49 php-fpm 7073 /rc.filter_synchronize: A communications error occurred while attempting XMLRPC sync with username admin https://192.168.254.2:443. Sep 7 00:31:49 php-fpm 7073 /rc.filter_synchronize: XML_RPC_Client: Connection to RPC server 192.168.254.2:443 failed. Operation timed out 103 Sep 7 00:30:33 check_reload_status Syncing firewall Sep 7 00:30:33 php-cgi suricata_check_for_rule_updates.php: [Suricata] The Rules update has finished. Sep 7 00:30:29 php-cgi suricata_check_for_rule_updates.php: [Suricata] Snort GPLv2 Community Rules are up to date... Sep 7 00:30:28 php-cgi suricata_check_for_rule_updates.php: [Suricata] Snort VRT rules file update downloaded successfully. Sep 7 00:30:09 php-cgi suricata_check_for_rule_updates.php: [Suricata] There is a new set of Snort VRT rules posted. Downloading snortrules-snapshot-2983.tar.gz... Sep 7 00:30:08 php-cgi suricata_check_for_rule_updates.php: [Suricata] Emerging Threats Open rules file update downloaded successfully. Sep 7 00:30:06 php-cgi suricata_check_for_rule_updates.php: [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz... Sep 6 17:59:01 lonrogfw-bluesteel.voyageurtransportation.ca nginx: 2016/09/06 17:59:01 [error] 57076#100061: *2654 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.100.7, server: , request: "POST /widgets/widgets/ipsec.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.1.100.30", referrer: "https://10.1.100.30/" Sep 6 17:27:55 php-cgi rc.restart_webgui: Creating rrd update script Sep 6 17:27:53 rc.php-fpm_restart 54531 >>> Restarting php-fpm Sep 6 17:27:50 lonrogfw-bluesteel.voyageurtransportation.ca nginx: 2016/09/06 17:27:50 [alert] 28036#100081: *38216 kevent() reported about an closed connection (53: Software caused connection abort) while reading response header from upstream, client: 10.1.100.7, server: , request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket:", host: "10.1.100.30" Sep 6 17:27:47 login login on ttyv0 as root Sep 6 17:26:50 php-fpm 44517 /pkg_edit.php: [suricata] XMLRPC sync sending auto-SID conf files to https://192.168.254.2:443. Sep 6 17:26:50 php-fpm 44517 /pkg_edit.php: New alert found: A communications error occurred while attempting Suricata XMLRPC sync with https://192.168.254.2:443\. Failed to transfer file: enablesid-sample.conf Sep 6 17:26:50 php-fpm 44517 /pkg_edit.php: A communications error occurred while attempting Suricata XMLRPC sync with https://192.168.254.2:443\. Failed to transfer file: enablesid-sample.conf Sep 6 17:26:50 php-fpm 44517 /pkg_edit.php: XML_RPC_Client: Connection to RPC server 192.168.254.2:443 failed. Operation timed out 103 Sep 6 17:26:05 lonrogfw-bluesteel.voyageurtransportation.ca nginx: 2016/09/06 17:26:05 [error] 28036#100081: *38216 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.100.7, server: , request: "POST /pkg_edit.php?xml=suricata/suricata_sync.xml HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.1.100.30", referrer: "https://10.1.100.30/pkg_edit.php?xml=suricata/suricata_sync.xml" Sep 6 17:25:35 php-fpm 44517 /pkg_edit.php: [suricata] XMLRPC sync sending auto-SID conf files to https://192.168.254.2:443. Sep 6 17:25:35 php-fpm 44517 /pkg_edit.php: New alert found: A communications error occurred while attempting Suricata XMLRPC sync with https://192.168.254.2:443\. Failed to transfer file: dropsid-sample.conf Sep 6 17:25:35 php-fpm 44517 /pkg_edit.php: A communications error occurred while attempting Suricata XMLRPC sync with https://192.168.254.2:443\. Failed to transfer file: dropsid-sample.conf Sep 6 17:25:35 php-fpm 44517 /pkg_edit.php: XML_RPC_Client: Connection to RPC server 192.168.254.2:443 failed. Operation timed out 103 Sep 6 17:24:20 php-fpm 44517 /pkg_edit.php: [suricata] XMLRPC sync sending auto-SID conf files to https://192.168.254.2:443. Sep 6 17:24:20 php-fpm 44517 /pkg_edit.php: New alert found: A communications error occurred while attempting Suricata XMLRPC sync with https://192.168.254.2:443\. Failed to transfer file: disablesid-sample.conf Sep 6 17:24:20 php-fpm 44517 /pkg_edit.php: A communications error occurred while attempting Suricata XMLRPC sync with https://192.168.254.2:443\. Failed to transfer file: disablesid-sample.conf Sep 6 17:24:20 php-fpm 44517 /pkg_edit.php: XML_RPC_Client: Connection to RPC server 192.168.254.2:443 failed. Operation timed out 103 Sep 6 17:23:05 php-fpm 44517 /pkg_edit.php: [suricata] XMLRPC sync sending auto-SID conf files to https://192.168.254.2:443. Sep 6 17:23:05 php-fpm 44517 /pkg_edit.php: [suricata] XMLRPC sync is starting. Sep 6 17:23:05 check_reload_status Syncing firewall Sep 6 17:23:05 check_reload_status Syncing firewall Sep 6 12:31:22 php-fpm 49586 /rc.filter_synchronize: New alert found: A communications error occurred while attempting XMLRPC sync with username admin https://192.168.254.2:443. Sep 6 12:31:22 php-fpm 49586 /rc.filter_synchronize: A communications error occurred while attempting XMLRPC sync with username admin https://192.168.254.2:443. Sep 6 12:31:22 php-fpm 49586 /rc.filter_synchronize: XML_RPC_Client: Connection to RPC server 192.168.254.2:443 failed. Operation timed out 103 Sep 6 12:30:06 check_reload_status Syncing firewall Sep 6 12:30:06 php-cgi suricata_check_for_rule_updates.php: [Suricata] The Rules update has finished. Sep 6 12:30:06 php-cgi suricata_check_for_rule_updates.php: [Suricata] Snort GPLv2 Community Rules are up to date... Sep 6 12:30:06 php-cgi suricata_check_for_rule_updates.php: [Suricata] Snort VRT rules are up to date... Sep 6 12:30:04 php-cgi suricata_check_for_rule_updates.php: [Suricata] Emerging Threats Open rules are up to date... Sep 6 12:16:07 lonrogfw-bluesteel.voyageurtransportation.ca nginx: 2016/09/06 12:16:07 [error] 28036#100081: *12636 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.100.7, server: , request: "POST /widgets/widgets/ipsec.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.1.100.30", referrer: "https://10.1.100.30/"

Got it debugged by the switch provider and they made a new firmware for the switches were it works :)

No, I am not complaining about ISC, I know the server is widely used in many Linux/Unix environments. I just wanted to know if this was a missconfiguration on my side or normal behaviour.