SG-3100 switch weird behavior (resolved)
-
@johnpoz said in SG-3100 switch weird behavior (resolved):
so what is better solution? More arps going out for everything ;) Or just pinging pfsense from the rpi4 every minute or 2 minutes..
Weird one for sure..ehhe, that is weird indeed.
I'm not sure how it works, but it seems that every packet that goes through the switch reset that ARP timer, so the firewall wouldn't need to broadcast it as often. -
@johnpoz said in SG-3100 switch weird behavior (resolved):
Wouldn't setting a static arp for the rpi4 also solve it? Or that is different than the switch arp cache?
Yeah, nothing to do with the switch MAC table. That exists only in the switch IC.
So in fact I would expect setting a static ARP to make this worse because it will never expire, pfSense will never ARP for the IP so no responses will be generated.
So if it was still set static I'm surprised that max_age value made any difference.
-
@stephenw10 said in SG-3100 switch weird behavior (resolved):
So if it was still set static I'm surprised that max_age value made any difference.
It is set like that since the beginning, changed yesterday and the problem happened today several times..
-
Did you set that as a static ARP value to try to solve this or has it always been static?
Because static ARP entries are almost always the wrong choice. That might be the cause here if it was always set static.
Steve
-
@stephenw10 said in SG-3100 switch weird behavior (resolved):
Did you set that as a static ARP value to try to solve this or has it always been static?
I always use static, the problem happened, I changed a few IP addresses here and removed the static ARP, it happened a few times after that so I reverted to static ARP.
If you want, I can remove that entry to test too
-
Yeah, I would test just without the static ARP entry. If you still see the issue test with max_age at 120s again. Anything less than 3mins should prevent it.
Steve
-
@stephenw10 Done, disabled static ARP, left static DHCP mapping, reverted to sysctl net.link.ether.inet.max_age=1200 to test
If the problem happens, I'll change it 120 to test.
-
Oh, just happened..
Changing to 120 to test.
Edit:
The change didn't take effect, had to ping 192.168.255.253 from pfsense, lets see how it goes now.
Edit2: Problem is happening again, I suppose 120 is too much, let me try 60.
-
Static ARP or not didn't change anything.
ping -c 2 cron job with an interval of 60s works.
setting the sysctl net.link.ether.inet.max_age=60 also works.Now depends on what the user prefer, it seems to me that the cron job would be a better approach since it would only affect a single host.
-
Hmm, interesting. Thanks for testing.
Yeah entirely up to you really, either should work fine.
Steve
-
@stephenw10 said in SG-3100 switch weird behavior (resolved):
Hmm, interesting. Thanks for testing.
Thanks for the help :)
-
@mcury So this is because the mac cache timeout on the switch on the 3100 is short? So a real switch would fix this, or could allow or account for it.
Short might be the wrong term, its shorter/different than pfsense arp cache. So when pfsense goes to send this traffic to that mac, that it has in its arp table be it static or not. If no arp went out that the switch could see the response then the switch doesn't know where to send it - so it broadcasts to all its ports..
Interesting issue for sure.. I looked on my cisco, and I can alter the mac address table timeout, it defaults to 300 seconds.. You can also set static arp on the switch..
That your rpi is so quiet is strange, most everything is a chatty cathy these days. But I always running something on my pi's, ntp so there is always traffic off the network, so it needs to talk to pfsense. And my other pi, while it runs controller software, and I am almost always ssh'd to it because I use it is my linux shell, so that would have keep alives, and I am from another vlan so again it would always be talking to pfsense (its gateway)..
I should kill my ssh and see how chatty it is..
But this has been a great thread - very interesting root cause of the problem..
-
@johnpoz said in SG-3100 switch weird behavior (resolved):
So this is because the mac cache timeout on the switch on the 3100 is short?
Exactly.
That your rpi is so quiet is strange, most everything is a chatty cathy these days. But I always running something on my pi's, ntp so there is always traffic off the network, so it needs to talk to pfsense. And my other pi, while it runs controller software, and I am almost always ssh'd to it because I use it is my linux shell, so that would have keep alives, and I am from another vlan so again it would always be talking to pfsense (its gateway)..
Yes, I'm running Graylog server in my raspberry pi 4b, everything else is disabled to spare some RAM, so it is actually only receiving data (UDP), in which there is no acks, just a UDP dump. Switch believes that its a dead device.
That ping cron job fixed..But this has been a great thread - very interesting root cause of the problem..
Indeed it was, I'm glad we got it figured out, I was almost doing a clean install in my pfsense or changing the switch but that wouldn't solve anything..
-
@mcury so another work around might be to use TCP for your syslog ;) Then there would be acks and switch would know that IP.. I don't believe pfsense can send tcp syslog though, at least not with the default. Not sure if the syslog-ng package would allow for that?
-
Yeah, I thought that. You can use syslog-ng to 'proxy' it between udp and tcp if you needed to.
-
It's interesting that the Unifi switch apparently doesn't time out it's MAC table like that. I don't think you saw those packets on any other port there.
-
@johnpoz said in SG-3100 switch weird behavior (resolved):
so another work around might be to use TCP for your syslog ;) Then there would be acks and switch would know that IP.. I don't believe pfsense can send tcp syslog though, at least not with the default. Not sure if the syslog-ng package would allow for that?
That would work but I don't think there is TCP for that purpose, at least that I'm aware of.
The raspbery pi 4b is a 4GB model, so its not much for a syslog server in which there is everything indexed for fast search.
It is using elasticsearch for that task.
The ping cron job is pretty light for a network and RAM usage perspective, so I'll stick with this option.But yes, something running TCP would have solved that too..
Graylog is a very powerful tool to have in the network, take a look at my dashboard :)
I can search everything, things such a mac address, an IP, a port, a device who accessed a network folder, unifi devices logs, use a firewall rule Tracking ID to see if some user matched that rule and what time it happened.. There is no limit, really.
dead on arrival, nowhere to be found.
-
@stephenw10 maybe the unifi switch uses just a much longer cache timeout value. I would think switch cache timeout would be of decent length to be honest.
My cisco's have a 5 minute default.
-
@johnpoz said in SG-3100 switch weird behavior (resolved):
maybe the unifi switch uses just a much longer cache timeout value. I would think switch cache timeout would be of decent length to be honest.
It could be, unfortunately the unifi switch mini is pretty basic, there is no SSH access or something like that, that means that everything you can do with it its located in the controller's GUI..
To adopt it, I had to use an option 43 string in the DHCP options because there isn't a way to SSH to it and use the set-inform..
Edit: What I was thinking when I designed this network is to have low power usage and speed..
1 - Synology NAS DS218+
2 - Raspberry Pi 4B
3 - Raspberry Pi 3b
4 - NanoHD
5 - Switch mini
6 - SG-3100 -
@mcury yeah I am well aware of graylogs power ;) hehe
If your bored, or want to give back. There are prob some interested users here that could benefit from a walk thru on how you setup your system..
edit: yeah I picked up one of those flex mini's to play with.. I run unifi APs, etc. I wasn't overly impressed with it. Its tiny, it can be poe powered, its cheap... I could see some very valid use cases for it. I was planning on replacing the little smart switch I have behind my tv with it.. Just haven't gotten around to it as of yet - so for now it sitting on my shelf..
