FreeNAS kernel arp attempts to modify permanent arp entry

kevdog · Sep 26, 2019, 2:40 PM

I'm a little bit stumped on this one.

Setup -- pfSense 2.4.4 release p3 -- pfSense with IP address of 10.0.1.1
DNS Ranges 10.0.1.100-10.0.1.150, 10.0.1.165-10.0.1.200

FreeNAS 11.2-U6
Within FreeNAS - FreeBSD jail using dhcp -- IP address of jail 10.0.1.158

10.0.1.158 reserved as static IP address within pfSense with the freeBSD jail setup to obtain IP address through DHCP (however somehow when choosing this option a VNET is created).

Once jail is booted the arp table within the jail appears as such:
? (10.0.1.158) at 02:ff:60:ba:b5:82 on epair0b permanent [ethernet]

Littered within the logs of the jail are the following:
freenas kernel: arp: ce:c0:9f:c5:9a:b1 attempts to modify permanent entry for 10.0.1.1 on epair0b

I get this message repeated every 10-30 seconds

ce:c0:9f:c5:9a:b1 is the MAC address of the pfSense box

As an attempt to suppress the errors I tried adding the pfSense MAC address permanently into the arp cache of the jail.
sudo arp -s 10.0.1.1 ce:c0:9f:c5:9a:b1

This however doesn't change the log message of:
freenas kernel: arp: ce:c0:9f:c5:9a:b1 attempts to modify permanent entry for 10.0.1.1 on
epair0b

Is there anything I can do to prevent this from happening. Logs are filling up super quick.

jimp · Sep 26, 2019, 6:38 PM

I think you have it backward. There is currently a static ARP entry on FreeNAS for 10.0.1.1, pointing to something that isn't pfSense.

kevdog · Sep 26, 2019, 7:11 PM

@jimp

Thanks for getting back to me -- I really appreciate since I'm not sure exactly how to troubleshoot this issue.

OK I troubleshooted a few things and changed my available DHCP range from 10.0.1.100-10.0.1.150, 10.0.1.165-10.0.1.195

I'm no networking expert in creation of virtual LANs and jails. FreeNAS creates a VNET for each jail then associates it with an epair interface that is bridged to the main adapter.

On the main FreeNAS system I have these interfaces:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
	ether 0c:c4:7a:84:a5:94
	hwaddr 0c:c4:7a:84:a5:94
	inet 10.0.1.197 netmask 0xffffff00 broadcast 10.0.1.255
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
	ether 0c:c4:7a:84:a5:95
	hwaddr 0c:c4:7a:84:a5:95
	nd6 options=9<PERFORMNUD,IFDISABLED>
	media: Ethernet autoselect
	status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
	inet 127.0.0.1 netmask 0xff000000
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: iohyve-bridge-igb0
	ether 02:f6:c7:64:02:00
	nd6 options=1<PERFORMNUD>
	groups: bridge
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: vnet0:12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 10 priority 128 path cost 2000
	member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 9 priority 128 path cost 2000000
	member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 8 priority 128 path cost 2000
	member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 7 priority 128 path cost 2000
	member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 6 priority 128 path cost 2000
	member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 5 priority 128 path cost 2000000
	member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 1 priority 128 path cost 20000
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: iohyve-arch-on-FreeNAS--
	options=80000<LINKSTATE>
	ether 00:bd:8d:56:f8:01
	hwaddr 00:bd:8d:56:f8:01
	nd6 options=1<PERFORMNUD>
	media: Ethernet autoselect
	status: active
	groups: tap
	Opened by PID 3915
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:a8:d0:00:06:0a
	hwaddr 02:a8:d0:00:06:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair
epair1a: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:a8:d0:00:07:0a
	hwaddr 02:a8:d0:00:07:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair
epair2a: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 02:a8:d0:00:08:0a
	hwaddr 02:a8:d0:00:08:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: Attached to UbuntuMC
	options=80000<LINKSTATE>
	ether 00:bd:52:cc:f8:00
	hwaddr 00:bd:52:cc:f8:00
	nd6 options=1<PERFORMNUD>
	media: Ethernet autoselect
	status: active
	groups: tap
	Opened by PID 8225
vnet0:12: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: associated with jail: nextcloud as nic: epair0b
	options=8<VLAN_MTU>
	ether 02:ff:60:ba:b5:81
	hwaddr 02:a8:d0:00:0a:0a
	nd6 options=1<PERFORMNUD>
	media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
	status: active
	groups: epair

In this example I'm having issues with the 10.0.1.158 jail or the jail on net0:12 on epair0b -- meaning i get the error in the logs if the jail is running -- however when the jail is turned off, I don't get the error in the logs:

freenas kernel: arp: ce:c0:9f:c5:9a:b1 attempts to modify permanent entry for 10.0.1.1 on epair0b

In terms of arp entries -- (I've got three systems potentially I'm dealing with -- pfSense, FreeNas, Jail within FreeNas)

Arp caches are as follows:
pfSense:

/root: arp -a
? (10.0.1.171) at d0:e7:82:bc:b7:0b on xn1 expires in 1195 seconds [ethernet]
? (10.0.1.10) at 00:e0:67:13:5b:7f on xn1 expires in 985 seconds [ethernet]
? (10.0.1.169) at 40:9f:38:26:e6:af on xn1 expires in 1002 seconds [ethernet]
? (10.0.1.9) at 74:83:c2:1e:ad:c5 on xn1 expires in 977 seconds [ethernet]
? (10.0.1.168) at 38:c9:86:1b:2b:2f on xn1 expires in 1071 seconds [ethernet]
? (10.0.1.174) at 38:8b:59:0e:b8:43 on xn1 expires in 974 seconds [ethernet]
? (10.0.1.173) at 8c:85:90:35:cc:e7 on xn1 expires in 1071 seconds [ethernet]
? (10.0.1.172) at 70:ef:00:b1:5a:92 on xn1 expires in 1186 seconds [ethernet]
? (10.0.1.3) at b4:fb:e4:b0:af:f8 on xn1 expires in 1160 seconds [ethernet]
? (10.0.1.226) at 18:b4:30:9b:b9:56 on xn1 expires in 1006 seconds [ethernet]
? (10.0.1.193) at 94:e9:6a:00:81:67 on xn1 expires in 1111 seconds [ethernet]
pfSense.xxxxx.com (10.0.1.1) at ce:c0:9f:c5:9a:b1 on xn1 permanent [ethernet]
? (10.0.1.160) at 00:a0:98:c6:2b:6d on xn1 expires in 1185 seconds [ethernet]
? (10.0.1.224) at 18:b4:30:30:68:33 on xn1 expires in 977 seconds [ethernet]
? (10.0.1.167) at 5c:ad:cf:c9:dc:6c on xn1 expires in 1071 seconds [ethernet]
? (10.0.1.71) at d4:81:ca:5e:f7:58 on xn1 expires in 976 seconds [ethernet]
? (10.0.1.197) at 0c:c4:7a:84:a5:94 on xn1 permanent [ethernet]
? (10.0.1.228) at 18:b4:30:98:b3:e3 on xn1 expires in 977 seconds [ethernet]
? (10.0.1.132) at 64:52:99:7d:6b:60 on xn1 expires in 972 seconds [ethernet]
? (10.0.1.91) at c8:63:f1:cf:6a:c6 on xn1 expires in 1199 seconds [ethernet]
? (10.0.1.186) at 9c:ae:d3:1e:31:d2 on xn1 expires in 1178 seconds [ethernet]
? (10.0.1.121) at d4:81:ca:5e:de:04 on xn1 expires in 1022 seconds [ethernet]
? (10.0.1.120) at 00:5b:94:d8:c5:0f on xn1 expires in 1158 seconds [ethernet]
? (10.0.1.95) at 00:a0:98:4e:65:cf on xn1 expires in 990 seconds [ethernet]
? (10.0.1.222) at 18:b4:30:36:38:4b on xn1 expires in 1070 seconds [ethernet]
? (10.0.1.158) at 02:ff:60:ba:b5:82 on xn1 expires in 981 seconds [ethernet]
? (10.0.1.221) at 18:b4:30:3a:24:b1 on xn1 expires in 1053 seconds [ethernet]
? (10.0.1.220) at 18:b4:30:99:0f:95 on xn1 expires in 1195 seconds [ethernet]
? (10.0.1.211) at 64:16:66:9d:d6:b8 on xn1 expires in 969 seconds [ethernet]
? (10.0.1.146) at 7a:5b:b5:96:d0:af on xn1 expires in 993 seconds [ethernet]
? (10.0.1.210) at 18:b4:30:15:ab:50 on xn1 expires in 975 seconds [ethernet]
? (10.0.1.178) at 6c:40:08:94:5b:78 on xn1 expires in 971 seconds [ethernet]
? (10.0.1.177) at 00:9d:6b:bc:cc:68 on xn1 expires in 1088 seconds [ethernet]
? (10.0.1.176) at b0:ca:68:c5:06:11 on xn1 expires in 1114 seconds [ethernet]
? (10.0.1.212) at 64:16:66:9e:45:22 on xn1 expires in 1030 seconds [ethernet]
? (10.0.1.116) at 90:dd:5d:cf:ee:04 on xn1 expires in 976 seconds [ethernet]
c-xx.xx.xx.xx.hsd1.il.comcast.net (xx.xx.xx.xx) at 2a:ec:f4:07:7c:72 on xn0 permanent [ethernet]
? (xx.xx.xx.xx) at 00:01:5c:64:f0:46 on xn0 expires in 780 seconds [ethernet

FreeNAS:

# arp -a
? (10.0.1.10) at 00:e0:67:13:5b:7f on igb0 expires in 1139 seconds [ethernet]
? (10.0.1.168) at 38:c9:86:1b:2b:2f on igb0 expires in 1154 seconds [ethernet]
? (10.0.1.173) at 8c:85:90:35:cc:e7 on igb0 expires in 1154 seconds [ethernet]
pfSense.xxxxx.com (10.0.1.1) at ce:c0:9f:c5:9a:b1 on igb0 expires in 1171 seconds [ethernet]
? (10.0.1.6) at 64:a5:c3:5b:bb:90 on igb0 expires in 1182 seconds [ethernet]
? (10.0.1.5) at 34:12:98:03:ca:6a on igb0 expires in 1154 seconds [ethernet]
? (10.0.1.197) at 0c:c4:7a:84:a5:94 on igb0 permanent [ethernet]
? (10.0.1.186) at 9c:ae:d3:1e:31:d2 on igb0 expires in 1161 seconds [ethernet]
? (10.0.1.178) at 6c:40:08:94:5b:78 on igb0 expires in 848 seconds [ethernet]

and finally the jail at 10.0.1.158

? (10.0.1.158) at 02:ff:60:ba:b5:82 on epair0b permanent [ethernet]
? (10.0.1.178) at 6c:40:08:94:5b:78 on epair0b expires in 1198 seconds [ethernet]

Nothing looks amiss to me when I look at the arp cache tables. Hopefully this makes some sense to you. Problem goes away when I stop the jail, however I kind of need it running.

jimp · Sep 26, 2019, 7:49 PM

Curious, I'm not sure why it would only do that with the Jail active, but I'm not familiar enough with FreeNAS to say what it might be doing with the network stack inside the Jail. I haven't seen my plain FreeBSD jails do anything like that in the past.

This is unlikely to be anything related to pfSense, though. You'd probably have better luck asking on a FreeNAS-specific forum/subreddit

kevdog · Sep 26, 2019, 8:37 PM

@jimp Just making sure however it probably isn't relative --- I have pfSense virtualized within xcp-ng (Citrix open source variant). I looked at the arp table for dom0 and it didn't have any other entries that conflicted either. I'm not overlooking a setting (promiscuous mode or some such) when I virtualize pfSense that would lead to this issue? FreeNas is not virtualized and runs on bare metal.

jimp · Sep 27, 2019, 12:27 PM

Doubtful. From what you've posted this would appear to be entirely contained somewhere inside FreeNAS. It's logging what it believes to be an alteration to something else on the FreeNAS system. The fact that the error mentions the actual IP address and MAC address of pfSense, and not something else, would appear to suggest that it's not a problem external to FreeNAS.

Slight possibility there is another device on the network also trying to be 10.0.1.1, but usually in that case you'd see an error logged on pfSense about another device attempting to use its MAC address.