Static Address ARP IP/MAC Flapping - Syslog

[RobbieTT]

I'm getting a weird ARP state change associated with one of my servers. The server has 2 physical interfaces set as an ALB LAG but the issues equally applies when set as a .3ad LAG. The symptoms exist with the server on or 'off'.

Both ports have a fixed IP with a static reservation set on pfSense:

The syslog is flooded with brief ARP state changes where the primary MAC address of the LAG switches to the tertiary, before switching back again:

[snip]
Mar 28 04:33:38	kernel		arp: 10.0.1.42 moved from 00:11:de:ad:b3:66 to 00:11:de:ad:b3:67 on ix1
Mar 28 04:33:36	kernel		arp: 10.0.1.42 moved from 00:11:de:ad:b3:67 to 00:11:de:ad:b3:66 on ix1
Mar 28 04:32:29	kernel		arp: 10.0.1.42 moved from 00:11:de:ad:b3:66 to 00:11:de:ad:b3:67 on ix1
Mar 28 04:32:27	kernel		arp: 10.0.1.42 moved from 00:11:de:ad:b3:67 to 00:11:de:ad:b3:66 on ix1
Mar 28 04:31:21	kernel		arp: 10.0.1.42 moved from 00:11:de:ad:b3:66 to 00:11:de:ad:b3:67 on ix1
Mar 28 04:31:19	kernel		arp: 10.0.1.42 moved from 00:11:de:ad:b3:67 to 00:11:de:ad:b3:66 on ix1
[/snip]

None of my network tools are detecting this apparent switch and a regular ARP command shows nothing amiss. The server can be reached normally and the only observable impact is that WoL takes longer than it should. Beyond that everything performs normally.

The server is a Syno RS819 and its own logs are normal. The server has a very light workload, waking in the early hours to receive a nightly backup from a different server.

I am vexed as to how a static IP can switch its associated MAC and do so repetitively - any ideas to resolve this?

️

[jrey]

@robbiett

I see this as well. Seems to be the Syno end.

In my case the Syno (DS1520+) is Lagged and connected to a managed switch.
That switch is then only connected to the NetGate by a single port.
The NetGate only sees and logs all the flapping.

I notice it more frequently when the lag is load balancing more or saturated > 1000. When the rate between the switch and the syno gets near capacity 2000 the logging is just insane on the netgate.

Works just fine, but yes the logging is annoying.

What the Syno should likely do is provide a spoofed mac that represents both the underlying MAC addresses as one, rather than allowing the underlying ports to provide their MAC directly when balancing. i've had no traction on getting an answer from Synology and have not looked further.

[RobbieTT]

@jrey
Sounds similar but not the same. For me this occurs even when the server is under no load or even in its 'off' state.

It also does not explain why pfSense is allowing a static IP address to associate with a different MAC address. A rather undesirable outcome(!).

️

[jrey]

@robbiett said in Static Address ARP IP/MAC Flapping - Syslog:

is under no load

Yes, just not as much flapping. That would be the Syno checking the ports, under "load balance" but under load the flapping goes way up.

"or even in its 'off' state"
What do you mean by even in its "off" state ? what is in an off state?

"why pfSense is allowing a static IP address to associate with a different MAC address."

Because at that point in time that is the broadcast from the syno.
Looking at the ARP table the static IP is only ever listed once.. but changes frequently I'll give you that (ie every time the syno uses the other MAC addy from the LAG)

I think what you are implying is that you see both entries in the ARP table?

I've never seen the static IP with two MAC addresses in the ARP table, always only one and always the last one it flapped to.

Arp table (IP 0.4 :da)

later it is (IP 0.4 :d9)

The MAC address has changed but still only one IP/MAC combo in APR table

Log of the flapping

I'm not sure what else pfsense could do with the ARP information that is being provided to it by the device at any point in time. It correctly logs the change and is reflected that way in the ARP table.
Added overhead, sure, but how is it broken?

Every device on the network that can display the ARP table will only have one IP/MAC combo at any point in time and will update with every ARP broadcast from the Syno

[johnpoz]

@robbiett if your goal is not to log this.. This setting might work.

net.link.ether.inet.log_arp_movements=0

You could prob set that in the tunables under advanced.. Keep in mind while this should keep the kernel from reporting these movements. It could mask the issue of having dupe IPs on the network.

nothing on my network does this.. So I don't have a easy way to test it.. I don't have anything bonded in my nas.. I use to use smb3 multichannel for better speed, but I just setup each interface with their own IP.. So there was never any moving of or announcement of IP on different interface like what can happen with that adaptive loading setting.

I have since then moved on to just using a single 2.5ge interface to break the 1 gig barrier, etc.

So seems your solution would be either to suppress the notifications from the kernel letting you know hey this IP keeps moving mac address.. this could clean up your logging, but also could prevent you from finding a dupe IP..

Or change your setup so it doesn't do that. Maybe go with active/standby mode so only 1 interface is ever used at a time. Or forget all the bonding stuff and just setup specific IPs for each interface and use smb3 multichannel for better file copy speeds to and from your nas, when your clients also have 2 interfaces. But this would not allow for any sort of aggregation to load balance other than different clients talking to the different IPs.. A dns load balance could be leveraged to get some of your clients to use IPA and others to use IPB.

[jrey]

@johnpoz said in Static Address ARP IP/MAC Flapping - Syslog:

net.link.ether.inet.log_arp_movements=0

Now that you mention it, I recall reading this before, and just electing to let it log for some of the other reasons listed in the doc. (it's just more log traffic, which in my case gets filtered out by the reporting after the syslog is shipped) I've a filter over there that removed the messages for this IP/MAC combos from the detail and just reports that it happening. Other changes that still happen but are rare still report in detail.

Also, some of your proposed solutions although great suggestions, involve degradation of the lag "active/standby mode so only 1 interface is ever used at a time" or throwing more money at the problem,
" IPs for each interface and use smb3 multichannel for better file copy speeds to and from your nas, when your clients also have 2 interfaces."

Not available for everyone / device.

@RobbieTT Here is the doc
https://docs.netgate.com/pfsense/en/latest/troubleshooting/logs-arp-moved.html

[RobbieTT]

@johnpoz said in Static Address ARP IP/MAC Flapping - Syslog:

@robbiett
nothing on my network does this.. So I don't have a easy way to test it.. I don't have anything bonded in my nas.. I use to use smb3 multichannel for better speed, but I just setup each interface with their own IP.. So there was never any moving of or announcement of IP on different interface like what can happen with that adaptive loading setting.

I have since then moved on to just using a single 2.5ge interface to break the 1 gig barrier, etc.

I do have IPs set on both interfaces of the RS819 but the thoughts on using smb3 multichannel may have unlocked part of the mystery. One of my other Syno units (RS217) does have multichannel set and does not display the same behaviour.

One of the side-effects of using smb3 (at least on Syno) is that the ports remain at 1GbE when the unit is 'off' [ports 17 & 18 below] so perhaps that stabilises things. Without multichannel set they drop to 100 MbE when 'off' as shown on ports 19 & 20 - which is the unit that has the odd behaviour:

I agree that having servers running at greater than 1GbE makes things much simpler and I have a couple running on single 10GbE links on the blue ports above. It will be a few years yet before the last 1GbE servers tumble from my network though.

I'll configure the RS819 for multichannel and see if it changes things.

️