SG-3100 switch weird behavior (resolved)

stephenw10

Cool. I think I prefer reducing the ARP timeout as a solution. You might try setting that to 1 min and see if that also solves it. That's just a system tunable in pfSense, all in the config.
But either will work fine.

Steve

mcury

@stephenw10 said in SG-3100 switch weird behavior (resolved):

Cool. I think I prefer reducing the ARP timeout as a solution.

Wouldn't that change the behavior for everything? Like a global setting?

This ARP timeout would only be triggered in case of a host is not "alive" like the raspberry pi 4b we just observed ?

stephenw10

It would be global but 1min is not that unusual. I believe Windows uses 30s.
What would happen is that every minute the RasPi4 entry in the pfSense ARP table would time out. So in order to send syslog traffic to it it will ARP for the IP address and the RasPi4 will respond to that refreshing the MAC table in the switch.
It feels like a cleaner solution to me but if the ping is working for you then there then there no need to change it. It would be interesting to know that works if you're able to test it.

Steve

mcury

@stephenw10 said in SG-3100 switch weird behavior (resolved):

It would be interesting to know that works if you're able to test it.

Sure, I'll disable the cron job and test it right now, wireshark is already running, one sec

johnpoz

@stephenw10 said in SG-3100 switch weird behavior (resolved):

I believe Windows uses 30s.

windows uses a weird way of doing it, they use 30 seconds as base and then add a random multiplier on it.. But you can adjust it if you want.

https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/address-resolution-protocol-arp-caching-behavior

Wouldn't setting a static arp for the rpi4 also solve it? Or that is different than the switch arp cache?

I wish arp in windows actually showed you what was left on the cache, like freebsd, linux should do that too. At least in linux you can use ip -statistics neigh

I don't know of any way to actually view how much time is left on mac that is cached.

mcury

@johnpoz said in SG-3100 switch weird behavior (resolved):

Wouldn't setting a static arp for the rpi4 also solve it? Or that is different than the switch arp cache?

I tried that, actually the static ARP is set right now..

mcury

@stephenw10 15 minutes and no problems..

This also works:

sysctl net.link.ether.inet.max_age=60

johnpoz

@mcury so what is better solution? More arps going out for everything ;) Or just pinging pfsense from the rpi4 every minute or 2 minutes..

Weird one for sure..

mcury

@johnpoz said in SG-3100 switch weird behavior (resolved):

so what is better solution? More arps going out for everything ;) Or just pinging pfsense from the rpi4 every minute or 2 minutes..
Weird one for sure..

ehhe, that is weird indeed.
I'm not sure how it works, but it seems that every packet that goes through the switch reset that ARP timer, so the firewall wouldn't need to broadcast it as often.

stephenw10

@johnpoz said in SG-3100 switch weird behavior (resolved):

Wouldn't setting a static arp for the rpi4 also solve it? Or that is different than the switch arp cache?

Yeah, nothing to do with the switch MAC table. That exists only in the switch IC.

So in fact I would expect setting a static ARP to make this worse because it will never expire, pfSense will never ARP for the IP so no responses will be generated.

So if it was still set static I'm surprised that max_age value made any difference.

mcury

@stephenw10 said in SG-3100 switch weird behavior (resolved):

So if it was still set static I'm surprised that max_age value made any difference.

It is set like that since the beginning, changed yesterday and the problem happened today several times..

stephenw10

Did you set that as a static ARP value to try to solve this or has it always been static?

Because static ARP entries are almost always the wrong choice. That might be the cause here if it was always set static.

Steve

mcury

@stephenw10 said in SG-3100 switch weird behavior (resolved):

Did you set that as a static ARP value to try to solve this or has it always been static?

I always use static, the problem happened, I changed a few IP addresses here and removed the static ARP, it happened a few times after that so I reverted to static ARP.

If you want, I can remove that entry to test too

stephenw10

Yeah, I would test just without the static ARP entry. If you still see the issue test with max_age at 120s again. Anything less than 3mins should prevent it.

Steve

mcury

@stephenw10 Done, disabled static ARP, left static DHCP mapping, reverted to sysctl net.link.ether.inet.max_age=1200 to test
If the problem happens, I'll change it 120 to test.

mcury

Oh, just happened..

Changing to 120 to test.

Edit:

The change didn't take effect, had to ping 192.168.255.253 from pfsense, lets see how it goes now.

Edit2: Problem is happening again, I suppose 120 is too much, let me try 60.

mcury

Static ARP or not didn't change anything.

ping -c 2 cron job with an interval of 60s works.
setting the sysctl net.link.ether.inet.max_age=60 also works.

Now depends on what the user prefer, it seems to me that the cron job would be a better approach since it would only affect a single host.

stephenw10

Hmm, interesting. Thanks for testing.

Yeah entirely up to you really, either should work fine.

Steve

mcury

@stephenw10 said in SG-3100 switch weird behavior (resolved):

Hmm, interesting. Thanks for testing.

Thanks for the help :)

johnpoz

@mcury So this is because the mac cache timeout on the switch on the 3100 is short? So a real switch would fix this, or could allow or account for it.

Short might be the wrong term, its shorter/different than pfsense arp cache. So when pfsense goes to send this traffic to that mac, that it has in its arp table be it static or not. If no arp went out that the switch could see the response then the switch doesn't know where to send it - so it broadcasts to all its ports..

Interesting issue for sure.. I looked on my cisco, and I can alter the mac address table timeout, it defaults to 300 seconds.. You can also set static arp on the switch..

That your rpi is so quiet is strange, most everything is a chatty cathy these days. But I always running something on my pi's, ntp so there is always traffic off the network, so it needs to talk to pfsense. And my other pi, while it runs controller software, and I am almost always ssh'd to it because I use it is my linux shell, so that would have keep alives, and I am from another vlan so again it would always be talking to pfsense (its gateway)..

I should kill my ssh and see how chatty it is..

But this has been a great thread - very interesting root cause of the problem..