ARP packets "disappearing" on back to back link

renedk

Hi,

We've just updated our HA setup from 2.4 to 2.7.

This broke sync, and troubleshooting the issue has led to discovering a problem that I can't seem to figure out.

The two firewalls can't ping each other, tcpdumping on the interfaces (bce1) showed that fw1 sends arp request out, fw2 replies to it, but the answer never seems to reach fw1.
The link is back to back, so no switches involved.

[root@pfSense1 ~]# tcpdump -vv -i bce1 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bce1, link-type EN10MB (Ethernet), capture size 262144 bytes
13:47:50.740653 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.179.2 tell 192.168.179.1, length 28
13:47:51.739915 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.179.2 tell 192.168.179.1, length 28

[root@pfSense2 ~]# tcpdump -vv -i bce1 arp
tcpdump: listening on bce1, link-type EN10MB (Ethernet), capture size 262144 bytes
13:47:50.740513 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.179.2 tell 192.168.179.1, length 46
13:47:50.740521 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.179.2 is-at 5c:f3:fc:e4:6b:0e (oui Unknown), length 28
13:47:51.739773 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.179.2 tell 192.168.179.1, length 46
13:47:51.739781 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.179.2 is-at 5c:f3:fc:e4:6b:0e (oui Unknown), length 28

My first thought was that maybe some rules were blocking it, but as far as I can see, that doesn't seem to be the case.

[root@pfSense1 ~]# pfctl -vvsr | grep bce1
@42 scrub on bce1 inet all fragment reassemble
@43 scrub on bce1 inet6 all fragment reassemble
@276 block drop in on ! bce1 inet from 192.168.179.0/24 to any ridentifier 1000022470
@278 block drop in on bce1 inet6 from fe80::5ef3:fcff:fee4:698a to any ridentifier 1000022470
@688 pass on bce1 inet proto tcp from any to (self:131) port = domain flags S/SA keep state label "USER_RULE: Tillad DNS" label "id:1519110775" ridentifier 1519110775
@689 pass on bce1 inet proto udp from any to (self:131) port = domain keep state label "USER_RULE: Tillad DNS" label "id:1519110775" ridentifier 1519110775
@2285 pass in quick on bce1 inet all flags S/SA keep state label "USER_RULE" label "id:1688636855" ridentifier 1688636855

[root@pfSense2 ~]# pfctl -vvsr|grep bce1
@42 scrub on bce1 inet all fragment reassemble
@43 scrub on bce1 inet6 all fragment reassemble
@276 block drop in log on ! bce1 inet from 192.168.179.0/24 to any ridentifier 1000022470
@278 block drop in log on bce1 inet6 from fe80::5ef3:fcff:fee4:6b0e to any ridentifier 1000022470
@688 pass on bce1 inet proto tcp from any to (self:131) port = domain flags S/SA keep state label "USER_RULE: Tillad DNS" label "id:1519110775" ridentifier 1519110775
@689 pass on bce1 inet proto udp from any to (self:131) port = domain keep state label "USER_RULE: Tillad DNS" label "id:1519110775" ridentifier 1519110775
@2285 pass in quick on bce1 inet all flags S/SA keep state label "USER_RULE" label "id:1517396910" ridentifier 1517396910

The other interfaces between the firewalls also can't ping each other, but this one is connected back to back, so I've concentrated on troubleshooting this one, so it wouldn't be a switch issue.

The clients doesn't have any problem accessing both firewalls, and get ARP replies just fine.

So far I've come up short on figuring out the issue, so I could use some advice on where to troubleshoot next.