100% Packet loss on primary firewall with HA Enabled (PFSync/CARP/NAT)



  • Trying to get HA working flawlessly although so far I seem to be hitting a wall with packet loss issues.
    Whenever the second firewall is connected it immediately causes the first one to hit 100% packet loss on both WAN and LAN, unplug and reconnect the Ethernet cables for the first firewall and the packet loss jumps over to the second firewall and back to 0% on the first.

    Two firewalls are connected together via Port 5 with PFSYNC set on that network
    Sync reports OK.

    Both LAN ports connect to the same switch.
    Both WAN ports are connected to an unmanaged switch which connects straight to the router.

    I'm sure i'm missing something obvious but I just can't see it now as I've been looking at it for too long.

    Virtual IP:

    192...2/32 (vhid: 3) WAN CARP WAN CARP 2
    10...251/32 (vhid: 2) LAN CARP LAN CARP
    192...3/32 (vhid: 1) WAN CARP WAN CARP

    NAT Mappings (Outbound NAT):

    Interface	Source	Source Port	Destination	Destination Port	NAT Address	NAT Port	Static Port	Description	Actions
    	WAN	any	*	109.*.*.*/32	*	192.*.*.2	*		Routed via ISP for NAT traffic on .2	  
    	WAN	any	*	93.*.*.*/32	*	192.*.*.2	*		Routed via ISP for NAT traffic on .2
    	WAN	any	*	*	*	192.*.*.3	*		WAN	  
    	LAN	any	*	*	*	10.*.*.251	*		DEFAULT LAN RULE	  
    

    CARP Status:
    Show as MASTER and Backup and full history of pfSync Nodes

    Firewall Rules (Relevant):

    WAN:

    States	Protocol	Source	Port	Destination	Port	Gateway	Queue	Schedule	Description	Actions
    	0 /84  IPv4 *	WAN net	*	WAN net	*	*	none	 		    
             BLOCK IPv4 *	*	*	*	*	*	none	 		    
    

    LAN:

    States	Protocol	Source	Port	Destination	Port	Gateway	Queue	Schedule	Description	Actions
               IPv4+6 *	*	*	*	*	*	none	 		
    

    SYNCPORT (PFSYNC):

    	States	Protocol	Source	Port	Destination	Port	Gateway	Queue	Schedule	Description	Actions
            0 /4.94 MiB  IPv4 *	*	*	*	*	*	none	 
    

    Gateways:

    	    Name	Default	Interface	Gateway	Monitor IP	Description	Actions
    	    
    	    GW_WAN (default)	Default (IPv4)	WAN	192.*.*.1	192.*.*.1	Interface wan Gateway	   
    	    GW_LAN		LAN	10.*.*.1	10.*.*.1	Interface lan Gateway	      
    

    Static Routes:

    Network Gateway Interface Description Actions
    10..5.0/24 GW_LAN - 10...1 LAN VLAN150
    10.
    .2.0/24 GW_LAN - 10...1 LAN VLAN 120
    10..10.0/24 GW_LAN - 10...1 LAN VLAN 180
    10.
    .3.0/24 GW_LAN - 10...1 LAN VLAN 130


  • LAYER 8 Netgate

    Why the /32s? CARP VIPs should have the same netmask as the interfaces they are on.

    https://www.netgate.com/docs/pfsense/book/highavailability/index.html


  • LAYER 8 Netgate

    Why GW_LAN?

    You should probably post screenshots instead of textual approximations so we know what it is we're actually looking at.



  • @derelict That i did not notice, flicking them all over to /24 bit.


  • LAYER 8 Netgate

    It sounds like you have address conflicts between the primary and secondary though.

    You need to configure the two nodes with their own addresses on every interface. The CARP VIP swings between them.

    Simple example:

    WAN
    192.0.2.1/29 ISP GATEWAY
    192.0.2.2/29 CARP WAN
    192.0.2.3/29 PRIMARY WAN
    192.0.2.4/29 SECONDARY WAN

    LAN
    192.168.100.1/24 CARP LAN
    192.168.100.2/24 PRIMARY LAN
    192.168.100.3/24 SECONDARY LAN



  • We have this setup:
    Firewall 1: WAN: 192...201/24
    Firewall 2: WAN: 192...202/24

    Reason for two CARP for WAN is due to rules on ISP end, specific traffic needs to go out via .2
    CARP WAN: 192...2/24
    CARP WAN2: 192...3/24

    ISP Gateway: 192...1/24

    I'll power up the second firewall later in the day although the change of subnet's for the CARP might be the fix.



  • After making the tweaks to the CARP subnets I'm still left in the same situation, one switch is permanently sat with 100% packet loss, the other 0%

    Any logs I can provide that'll help diagnose the issue?


  • LAYER 8 Netgate

    @jgzowski said in 100% Packet loss on primary firewall with HA Enabled (PFSync/CARP/NAT):

    After making the tweaks to the CARP subnets I'm still left in the same situation, one switch is permanently sat with 100% packet loss, the other 0%

    What do you mean one switch?

    What exactly are you testing and how?



  • @derelict Sorry, i meant to say Firewall.
    I disabled monitoring of gateway as it does seem to function as expected.

    Issue I'm seeing now though within the logs is:
    A communications error occurred while attempting to call XMLRPC method host_firmware_version
    Configuration from primary isn't replicating to the secondary.

    Followed the instructions exactly and have double checked them now many times. Only have sync settings set on the primary, firewall rules for sync port set up.


  • LAYER 8 Netgate

    @jgzowski said in 100% Packet loss on primary firewall with HA Enabled (PFSync/CARP/NAT):

    @derelict Sorry, i meant to say Firewall.
    I disabled monitoring of gateway as it does seem to function as expected.

    does or does not? Because it works fine.

    Issue I'm seeing now though within the logs is:
    A communications error occurred while attempting to call XMLRPC method host_firmware_version

    That works fine too. Can you ping the other side that you're syncing to? Can you Diagnostics > Test Port to it on your webgui port? Is the admin password the same as is set in the sync settings?

    Configuration from primary isn't replicating to the secondary.

    Followed the instructions exactly and have double checked them now many times. Only have sync settings set on the primary, firewall rules for sync port set up.

    If it was done exactly as documented it would be working. I'd check everything again.



  • both firewalls are working, the packet loss issue seems to be there still although i disabled monitoring of it as as soon as primary drops the secondary works and vice versa.

    Primary managed to update the configuration on the secondary after the secondary had a reboot, since reboot though it's back to doing:

    A communications error occurred while attempting to call XMLRPC method restore_config_section: @ 2019-01-30 16:56:19
    A communications error occurred while attempting to call XMLRPC method host_firmware_version: @ 2019-01-30 16:56:37

    Port test from SYNCPORT:
    Port test to host: 10.200.0.2 Port: 443 successful

    Using HTTPS for webgui

    Firewall logs from SYNCPORT:
    SYNCPORT tcp 10.200.0.1:40286 -> 10.200.0.2:443 FIN_WAIT_2:FIN_WAIT_2 0 / 0 0 B / 0 B
    SYNCPORT pfsync 10.200.0.1 -> 10.200.0.2 MULTIPLE:MULTIPLE 21.66 K / 577 23.69 MiB / 460 KiB
    SYNCPORT tcp 10.200.0.1:40286 -> 10.200.0.2:443 FIN_WAIT_2:FIN_WAIT_2 4 / 3 216 B / 164 B

    --- EDIT

    It now seems to be working, have not changed anything else but it works.


  • LAYER 8 Netgate

    Do you have State killing on gateway failure enabled in System > Advanced, Miscellaneous?



  • no, should i?


  • LAYER 8 Netgate

    No. Not unless you know you need it. It is commonly the cause of the XMLRPC sync state being killed, resulting in errors like you are seeing.

    There has to be a reason for what you are seeing. What are the rules on the sync interfaces on both nodes?

    Are you just using the admin user/password for this or did you create another user?

    Are you familiar with packet capturing? Capturing HTTPS traffic on the sync interfaces might yield a clue.



  • @derelict

    Rules on both firewalls for the syncport:

    States	Protocol	Source	Port	Destination	Port	Gateway	Queue	Schedule	Description	Actions
     	0 /0 B IPv4+6 *	SYNCPORT net	*	*	*	*	none	 		    
    

    I'm using the default user admin with the same password on each firewall.

    Recorded full packet capture, looked in Wireshark and can't see anything glaringly obvious. servers are talking to each other, passing key exchange/handshake followed by many SYN,ACK and Application Data. Communication is going both ways ending with a FIN, ACK from the primary server and an ACK from the secondary.



  • Think i've solved it. Had a NAT Outbound rule for any traffic to anywhere to use NAT Adddress CARP.
    Added mapping for source of the LAN and another for source of the SYNCPORT and instructed the SYNCPORT not to use NAT.

    Also made changes to DNS Resolver so that All interfaces resolve to the NAT CARP as DNS was set to 8.8.8.8 and 8.8.4.4


  • LAYER 8 Netgate

    Why would sync interface traffic ever have to go out the WAN?

    Yes, outbound NAT with source any is almost never right - especially to a CARP VIP.

    Traffic from Localhost should NAT to the interface address

    Traffic from inside hosts should:

    1. Use the local interface CARP VIP as their default gateway
    2. Have outbound NAT to the WAN CARP VIP set.

    Traffic from the sync interface should never need internet access.


Log in to reply