SG-3100 switch weird behavior (resolved)

mcury · Oct 18, 2022, 10:36 PM

@stephenw10 Done, disabled static ARP, left static DHCP mapping, reverted to sysctl net.link.ether.inet.max_age=1200 to test
If the problem happens, I'll change it 120 to test.

mcury · Oct 18, 2022, 10:43 PM

Oh, just happened..

Changing to 120 to test.

Edit:

The change didn't take effect, had to ping 192.168.255.253 from pfsense, lets see how it goes now.

Edit2: Problem is happening again, I suppose 120 is too much, let me try 60.

mcury · Oct 18, 2022, 10:51 PM

Static ARP or not didn't change anything.

ping -c 2 cron job with an interval of 60s works.
setting the sysctl net.link.ether.inet.max_age=60 also works.

Now depends on what the user prefer, it seems to me that the cron job would be a better approach since it would only affect a single host.

stephenw10 · Oct 18, 2022, 10:54 PM

Hmm, interesting. Thanks for testing.

Yeah entirely up to you really, either should work fine.

Steve

mcury · Oct 19, 2022, 5:56 AM

@stephenw10 said in SG-3100 switch weird behavior (resolved):

Hmm, interesting. Thanks for testing.

Thanks for the help :)

johnpoz · Oct 19, 2022, 5:56 AM

@mcury So this is because the mac cache timeout on the switch on the 3100 is short? So a real switch would fix this, or could allow or account for it.

Short might be the wrong term, its shorter/different than pfsense arp cache. So when pfsense goes to send this traffic to that mac, that it has in its arp table be it static or not. If no arp went out that the switch could see the response then the switch doesn't know where to send it - so it broadcasts to all its ports..

Interesting issue for sure.. I looked on my cisco, and I can alter the mac address table timeout, it defaults to 300 seconds.. You can also set static arp on the switch..

That your rpi is so quiet is strange, most everything is a chatty cathy these days. But I always running something on my pi's, ntp so there is always traffic off the network, so it needs to talk to pfsense. And my other pi, while it runs controller software, and I am almost always ssh'd to it because I use it is my linux shell, so that would have keep alives, and I am from another vlan so again it would always be talking to pfsense (its gateway)..

I should kill my ssh and see how chatty it is..

But this has been a great thread - very interesting root cause of the problem..

mcury · Oct 19, 2022, 3:23 PM

@johnpoz said in SG-3100 switch weird behavior (resolved):

So this is because the mac cache timeout on the switch on the 3100 is short?

Exactly.

That your rpi is so quiet is strange, most everything is a chatty cathy these days. But I always running something on my pi's, ntp so there is always traffic off the network, so it needs to talk to pfsense. And my other pi, while it runs controller software, and I am almost always ssh'd to it because I use it is my linux shell, so that would have keep alives, and I am from another vlan so again it would always be talking to pfsense (its gateway)..

Yes, I'm running Graylog server in my raspberry pi 4b, everything else is disabled to spare some RAM, so it is actually only receiving data (UDP), in which there is no acks, just a UDP dump. Switch believes that its a dead device.
That ping cron job fixed..

But this has been a great thread - very interesting root cause of the problem..

Indeed it was, I'm glad we got it figured out, I was almost doing a clean install in my pfsense or changing the switch but that wouldn't solve anything..

johnpoz · Oct 19, 2022, 3:30 PM

@mcury so another work around might be to use TCP for your syslog ;) Then there would be acks and switch would know that IP.. I don't believe pfsense can send tcp syslog though, at least not with the default. Not sure if the syslog-ng package would allow for that?

stephenw10 · Oct 19, 2022, 3:33 PM

Yeah, I thought that. You can use syslog-ng to 'proxy' it between udp and tcp if you needed to.

stephenw10 · Oct 19, 2022, 3:34 PM

It's interesting that the Unifi switch apparently doesn't time out it's MAC table like that. I don't think you saw those packets on any other port there.

mcury · Oct 19, 2022, 3:40 PM

@johnpoz said in SG-3100 switch weird behavior (resolved):

so another work around might be to use TCP for your syslog ;) Then there would be acks and switch would know that IP.. I don't believe pfsense can send tcp syslog though, at least not with the default. Not sure if the syslog-ng package would allow for that?

That would work but I don't think there is TCP for that purpose, at least that I'm aware of.
The raspbery pi 4b is a 4GB model, so its not much for a syslog server in which there is everything indexed for fast search.
It is using elasticsearch for that task.
The ping cron job is pretty light for a network and RAM usage perspective, so I'll stick with this option.

But yes, something running TCP would have solved that too..

Graylog is a very powerful tool to have in the network, take a look at my dashboard :)

I can search everything, things such a mac address, an IP, a port, a device who accessed a network folder, unifi devices logs, use a firewall rule Tracking ID to see if some user matched that rule and what time it happened.. There is no limit, really.

johnpoz · Oct 19, 2022, 3:43 PM

@stephenw10 maybe the unifi switch uses just a much longer cache timeout value. I would think switch cache timeout would be of decent length to be honest.

My cisco's have a 5 minute default.

mcury · Oct 19, 2022, 3:47 PM

@johnpoz said in SG-3100 switch weird behavior (resolved):

maybe the unifi switch uses just a much longer cache timeout value. I would think switch cache timeout would be of decent length to be honest.

It could be, unfortunately the unifi switch mini is pretty basic, there is no SSH access or something like that, that means that everything you can do with it its located in the controller's GUI..

To adopt it, I had to use an option 43 string in the DHCP options because there isn't a way to SSH to it and use the set-inform..

Edit: What I was thinking when I designed this network is to have low power usage and speed..

1 - Synology NAS DS218+
2 - Raspberry Pi 4B
3 - Raspberry Pi 3b
4 - NanoHD
5 - Switch mini
6 - SG-3100

johnpoz · Oct 19, 2022, 3:49 PM

@mcury yeah I am well aware of graylogs power ;) hehe

If your bored, or want to give back. There are prob some interested users here that could benefit from a walk thru on how you setup your system..

edit: yeah I picked up one of those flex mini's to play with.. I run unifi APs, etc. I wasn't overly impressed with it. Its tiny, it can be poe powered, its cheap... I could see some very valid use cases for it. I was planning on replacing the little smart switch I have behind my tv with it.. Just haven't gotten around to it as of yet - so for now it sitting on my shelf..

mcury · Oct 19, 2022, 3:50 PM

@johnpoz said in SG-3100 switch weird behavior (resolved):

If your bored, or want to give back. There are prob some interested users here that could benefit from a walk thru on how you setup your system..

Sure, I can share everything about it, I'll prepare something to post here on how to configure it and use it in a Raspberry Pi :)

I can see that happening already, a bottle of coffee and keyboard stroke sounds :)

mcury · Oct 19, 2022, 3:52 PM

@johnpoz said in SG-3100 switch weird behavior (resolved):

so for now it sitting on my shelf..

It is a nice little device, it does the job and it is nice for small places, specially if you need PoE.
I'm happy with it to say at least, of course I miss my old Cisco days :)

johnpoz · Oct 19, 2022, 3:56 PM

@mcury once you put it up, I will give it a go via a VM maybe. I don't as of yet have a pi4 to play with.. Been looking for an excuse to get one hehe.. But they have been hard to find as well, I would prob go with the 8GB ram model as well.

My current pi's most likely under powered to run graylog for sure, even if strip down to bare essentials for stuff.

edit: yeah I had some stuff connected to it just in my computer room (home office) hehehe I was seeing lots of disconnects and reconnections.. Trying to recall what was causing it now, have to see if I put that info up here or might of been another forum.. I for sure agree for the price its a very handy little switch to have.

It wasn't actually disconnecting - if I recall it was something to do with it seeing different IPs for pfsense because of the different vlans that were trunked through it - and how pfsense uses the same mac for different vlan interfaces..

mcury · Oct 19, 2022, 3:59 PM

@johnpoz said in SG-3100 switch weird behavior (resolved):

once you put it up, I will give it a go via a VM maybe. I don't as of yet have a pi4 to play with.. Been looking for an excuse to get one hehe.. But they have been hard to find as well, I would prob go with the 8GB ram model as well.

Raspberry Pis are to expensive right now, there is a shortage on the market, go for the VM for now and when the prices are OK, you get one

I'll prepare that, I have some .txt files full of things to share about Graylog.
8GB would be much better than 4GB, you would be able to gather much more data.
Here 4GB is enough for 3 days of logs only.. But that would also depend on how much messages your Graylog server receives per day.

Sure you can optimize the firewall rules to don't log everything too, like connections for the firewall IP port 53 for an example..

johnpoz · Oct 19, 2022, 4:02 PM

@mcury oh found that thread were talking about the disconnects.. If your interested ;)

https://forum.netgate.com/topic/165926/pfsense-keeps-disconnecting-reconnecting-in-unifi-controller

Pretty sure that was thread that got me to get the mini ;) hehe

mcury · Oct 19, 2022, 4:10 PM

@johnpoz said in SG-3100 switch weird behavior (resolved):

Pretty sure that was thread that got me to get the mini ;) hehe

I read that thread, you helped me to decide about getting one for myself. :)
You were drinking a beer hehe