SG-3100 switch weird behavior (resolved)

mcury · Oct 18, 2022, 8:43 PM

I recorded, not sure if its going to be useful..
arp_problem.zip

stephenw10 · Oct 18, 2022, 8:46 PM

Hmm, well that seems very precise. Unlikely to be random then. The Marvell 88E6141 has a 2048 address MAC table. I'm going to assume you don't have >2000 devices!
I guess it's feasible something could be generating random MAC continually and filling the table. You would see that in a pcap though.
I can't find a value for a default expiry time. I'm not sure why it would expire at all.
If the table were being reset I might imagine something else would be reset too. Are you seeing any other traffic interrupted at the 3min mark?

Also can you confirm this is just unexpected, it's not actually failing to pass any traffic?

Steve

mcury · Oct 18, 2022, 8:50 PM

@stephenw10 said in SG-3100 switch weird behavior:

Hmm, well that seems very precise. Unlikely to be random then. The Marvell 88E6141 has a 2048 address MAC table. I'm going to assume you don't have >2000 devices!

Not even close.. 25 approximately..

If the table were being reset I might imagine something else would be reset too. Are you seeing any other traffic interrupted at the 3min mark?

No, everything is normal, I noticed it during a packet capture to check something else, otherwise I wouldn't even notice it..

Also can you confirm this is just unexpected, it's not actually failing to pass any traffic?

No, my network is running perfectly, the only issue is this, it seems that the marvell switch spams the packets to everyone since the mac table expired..
I'm thinking here, maybe install something in the Raspberry Pi 4b to force it to use the internet every 3 minutes?
Maybe a cron to run a single ping command?

stephenw10 · Oct 18, 2022, 8:57 PM

Yes, that would correct it. Doesn't have to be to something external, it just needs to hit the switch in the 3100.
You could set the ARP timeout in pfSense to <3mins. That way pfSense will ARP for the RasPi when it times out and the RasPi will respond refilling the switch table.
That's an easy test:

[22.11-DEVELOPMENT][admin@3100.stevew.lan]/root: sysctl net.link.ether.inet.max_age=120
net.link.ether.inet.max_age: 1200 -> 120

Steve

mcury · Oct 18, 2022, 9:02 PM

@stephenw10 said in SG-3100 switch weird behavior:

sysctl net.link.ether.inet.max_age=120

done:

[22.05-RELEASE][root@pfsense.home.arpa]/root: sysctl net.link.ether.inet.max_age=120
net.link.ether.inet.max_age: 1200 -> 120
[22.05-RELEASE][root@pfsense.home.arpa]/root:

mcury · Oct 18, 2022, 9:10 PM

Strange, sometimes it doesn't take 3 minutes for the problem to happen.
I'm not sure if the Marvell mac address is really expiring, or if the problem is something else..

I reverted the change, to 1200 and configured a cron job in rpi4

*/1 * * * * /usr/bin/ping 192.168.255.249

Lets see how that goes..

stephenw10 · Oct 18, 2022, 9:12 PM

I expect that to solve it.

This has been interesting, I've never had to look into it too closely before. I can't find a specific value for the 3100 switch but for the switch in the 7100, which is from the same family of devices, the default MAC address aging time is 300s (5 mins). That has a larger table size so 3mins for the 3100 doesn't seem that unreasonable.

I suspect this might be simply the traffic pattern you have to the RasPi4. The fact it's mostly UDP where it never sends a reply.

Steve

mcury · Oct 18, 2022, 9:16 PM

@stephenw10 I believe that is the issue..
My cron is smashing pfsense with pings, let me change that cron to 1 minute

mcury · Oct 18, 2022, 9:22 PM

This seems to be enough..

* * * * * /usr/bin/ping 192.168.255.249 -c 2

mcury · Oct 18, 2022, 9:36 PM

This solved the problem, 13 minutes of cron job running, no more problems..

Really thanks for the help @stephenw10 and @johnpoz :)

stephenw10 · Oct 18, 2022, 9:55 PM

Cool. I think I prefer reducing the ARP timeout as a solution. You might try setting that to 1 min and see if that also solves it. That's just a system tunable in pfSense, all in the config.
But either will work fine.

Steve

mcury · Oct 18, 2022, 10:01 PM

@stephenw10 said in SG-3100 switch weird behavior (resolved):

Cool. I think I prefer reducing the ARP timeout as a solution.

Wouldn't that change the behavior for everything? Like a global setting?

This ARP timeout would only be triggered in case of a host is not "alive" like the raspberry pi 4b we just observed ?

stephenw10 · Oct 18, 2022, 10:04 PM

It would be global but 1min is not that unusual. I believe Windows uses 30s.
What would happen is that every minute the RasPi4 entry in the pfSense ARP table would time out. So in order to send syslog traffic to it it will ARP for the IP address and the RasPi4 will respond to that refreshing the MAC table in the switch.
It feels like a cleaner solution to me but if the ping is working for you then there then there no need to change it. It would be interesting to know that works if you're able to test it.

Steve

mcury · Oct 18, 2022, 10:05 PM

@stephenw10 said in SG-3100 switch weird behavior (resolved):

It would be interesting to know that works if you're able to test it.

Sure, I'll disable the cron job and test it right now, wireshark is already running, one sec

johnpoz · Oct 18, 2022, 10:21 PM

@stephenw10 said in SG-3100 switch weird behavior (resolved):

I believe Windows uses 30s.

windows uses a weird way of doing it, they use 30 seconds as base and then add a random multiplier on it.. But you can adjust it if you want.

https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/address-resolution-protocol-arp-caching-behavior

Wouldn't setting a static arp for the rpi4 also solve it? Or that is different than the switch arp cache?

I wish arp in windows actually showed you what was left on the cache, like freebsd, linux should do that too. At least in linux you can use ip -statistics neigh

I don't know of any way to actually view how much time is left on mac that is cached.

mcury · Oct 18, 2022, 10:18 PM

@johnpoz said in SG-3100 switch weird behavior (resolved):

Wouldn't setting a static arp for the rpi4 also solve it? Or that is different than the switch arp cache?

I tried that, actually the static ARP is set right now..

login-to-view

mcury · Oct 18, 2022, 10:22 PM

@stephenw10 15 minutes and no problems..

This also works:

sysctl net.link.ether.inet.max_age=60

johnpoz · Oct 18, 2022, 10:24 PM

@mcury so what is better solution? More arps going out for everything ;) Or just pinging pfsense from the rpi4 every minute or 2 minutes..

Weird one for sure..

mcury · Oct 18, 2022, 10:26 PM

@johnpoz said in SG-3100 switch weird behavior (resolved):

so what is better solution? More arps going out for everything ;) Or just pinging pfsense from the rpi4 every minute or 2 minutes..
Weird one for sure..

ehhe, that is weird indeed.
I'm not sure how it works, but it seems that every packet that goes through the switch reset that ARP timer, so the firewall wouldn't need to broadcast it as often.

stephenw10 · Oct 18, 2022, 10:26 PM

@johnpoz said in SG-3100 switch weird behavior (resolved):

Wouldn't setting a static arp for the rpi4 also solve it? Or that is different than the switch arp cache?

Yeah, nothing to do with the switch MAC table. That exists only in the switch IC.

So in fact I would expect setting a static ARP to make this worse because it will never expire, pfSense will never ARP for the IP so no responses will be generated.

So if it was still set static I'm surprised that max_age value made any difference.