arp message about an IP in daily output

Understudy

So I posted to the freebsd forums this message.

Hi All,
So recently I had to make some changes to my network system. I put in a new router / firewall with a new IP address. The IP is within the subnet and the public gateway remains the same. However one of my servers is sending me this in the daily security run output.

nexus.brendhanhorne.com kernel log messages:
+arp: xxx.xxx.xxx.94 moved from 0c:c4:7a:59:7c:11 to 0c:c4:7a:59:7c:0f on bce1
+arp: xxx.xxx.xxx.94 moved from 0c:c4:7a:59:7c:11 to 0c:c4:7a:59:7c:0f on bce1
+arp: xxx.xxx.xxx.94 moved from 0c:c4:7a:59:7c:11 to 0c:c4:7a:59:7c:0f on bce1
+arp: xxx.xxx.xxx.94 moved from 0c:c4:7a:59:7c:0f to 0c:c4:7a:59:7c:11 on bce1
+arp: xxx.xxx.xxx.94 moved from 0c:c4:7a:59:7c:11 to 0c:c4:7a:59:7c:0f on bce1

You can see the thread here.
https://forums.freebsd.org/threads/arp-message-about-an-ip-in-daily-output.86891/

So there is some confusion about this. They are questioning the bridge. Which works. I am questioning why one server out of three gives me this log message. And why I am getting it.

The .94 IP is the router/firewall but the server has a different public IP and the gateway is a separate device upstream with it's own public IP. The router/firewall got changed about 2 weeks ago.

Server info:
FreeBSD nexus.brendhanhorne.com 13.1-RELEASE-p2 FreeBSD 13.1-RELEASE-p2 GENERIC amd64
Upgraded from 12.3 about a week ago.

Router/Firewall
[2.6.0-RELEASE][admin@Ignis.brendhanhorne.com]/root: uname -a
FreeBSD Ignis.brendhanhorne.com 12.3-STABLE FreeBSD 12.3-STABLE RELENG_2_6_0-n226742-1285d6d205f pfSense amd64

bhorne@nexus:~ $ ifconfig -a
bce0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether d4:85:64:7b:80:96
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether d4:85:64:7b:80:98
        inet xxx.xxx.xxx.89 netmask 0xfffffff0 broadcast xxx.xxx.xxx.95
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bce2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether d4:85:64:7b:80:9a
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bce3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
        ether d4:85:64:7b:80:9c
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bhorne@nexus:~ $

I appreciate your help. Thank you.

stephenw10

@understudy said in arp message about an IP in daily output:

From the router firewall (pfsense 2.6)

> 
> [2.6.0-RELEASE][admin@Ignis.brendhanhorne.com]/root: ifconfig
> ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
>         ether 0c:c4:7a:59:7c:0c
>         media: Ethernet autoselect
>         status: no carrier
>         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
>         ether 0c:c4:7a:59:7c:0d
>         media: Ethernet autoselect
>         status: no carrier
>         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> ix2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
>         ether 0c:c4:7a:59:7c:0e
>         media: Ethernet autoselect
>         status: no carrier
>         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> ix3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         description: DMZ
>         options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
>         ether 0c:c4:7a:59:7c:0f
>         inet6 fe80::ec4:7aff:fe59:7c0f%ix3 prefixlen 64 scopeid 0x4
>         media: Ethernet autoselect (100baseTX <full-duplex>)
>         status: active
>         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> ix4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         description: LAN
>         options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
>         ether 0c:c4:7a:59:7c:10
>         inet6 fe80::ec4:7aff:fe59:7c10%ix4 prefixlen 64 scopeid 0x5
>         inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
>         media: Ethernet autoselect (1000baseT <full-duplex>)
>         status: active
>         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> ix5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         description: WAN
>         options=e138bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
>         ether 0c:c4:7a:59:7c:11
>         inet6 fe80::ec4:7aff:fe59:7c11%ix5 prefixlen 64 scopeid 0x6
>         inet xxx.xxx.xxx.94 netmask 0xfffffff8 broadcast xxx.xxx.xxx.95
>         media: Ethernet autoselect (1000baseT <full-duplex,rxpause>)
>         status: active
>         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> enc0: flags=0<> metric 0 mtu 1536
>         groups: enc
>         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>         options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
>         inet6 ::1 prefixlen 128
>         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8
>         inet 127.0.0.1 netmask 0xff000000
>         groups: lo
>         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> pflog0: flags=100<PROMISC> metric 0 mtu 33160
>         groups: pflog
> pfsync0: flags=0<> metric 0 mtu 1500
>         groups: pfsync
> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         description: Bridge0
>         ether 02:9b:09:a6:e4:00
>         id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
>         maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
>         root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
>         member: ix3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>                 ifmaxaddr 0 port 4 priority 128 path cost 2000
>         member: ix5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>                 ifmaxaddr 0 port 6 priority 128 path cost 55
>         groups: bridge
>         nd6 options=1<PERFORMNUD>

I would recommend the same thing. Assign the bridge as WAN and put the IP address there. Spoof the MAC on the bridge so it remains constant.

It's slightly odd since I wouldn't expect the bridge to ever use the ix3 MAC. However since you know it's source you could also just ignore that.

Steve

Understudy

@stephenw10

Okay, got it. I will do that tonight and post the results tomorrow.

Thank you.

Understudy

Did some testing and came up with a solution, possibly a work around.
I was bothered by a couple of things. I was getting this arp report only from one of the servers behind the DMZ.
I did the changing of the Bridge to WAN and spoofed the addresses but I would still get an arp message to show up with dmesg | grep arp about every 30 minutes at the server. I tried swapping the macs and then I would just get the same message from the server with the macs being swapped. So I decided to go after the server. I tried looking into arp and tried arp -ad but it would eventually go back to the settings it had and the problem would return arp -a would show that. I ended up finding something called static arp pairs which can be put into /etc/rc.conf. Basically that pairs a mac with an IP even if it in not on that machine. So I added this to my /etc/rc.conf

## Static Arp Pairs
static_arp_pairs="myarp"
static_arp_myarp="xxx.xxx.xxx.94 0c:c4:7a:59:7c:11"

And then ran

service static_arp start

That appears to have fixed the issue. Which I would say is on the server not the firewall. What I am not aware of is I guess arp runs in the background, maybe?

stephenw10

Static ARP usually causes more problems than it solves IMO. Certainly in pfSense.

Should be OK on the server as long as you remember it's there if the gateway ever changes.

Steve

Understudy

@stephenw10 you are absolutely correct. That is why I consider this a work around and not a solution.