To introduce more things to manage introduces more risk (all is duplicated to avoid having to go to the datacenter in case of one line falls out - both power, network and two network interfaces in team-mode).
When all I need is two ports, it is a bit drastic to buy enterprise switches into this (more rack-space, more cables, more outlets etc). I also have another pfSense SG-2440 (not for use in my network where I have tested the switching speed and found it to be more than good enough - I couldn't really see any delays introduced - I assume we are talking microseconds and not ms?).
I have had the pfSense as transparent fw for two years, with only rules on the WAN-interface. The only change I have done, is to connect our second main switch to the second LAN-interface (called LAN_SW2 or physical port OPT1 of pfSense) instead of just having it serial-connected to just one main switch. So it is in parallell/redundancy-mode. We have RSTP thoughout on all switches.
I do have block rules on the WAN-interface (against known bot-nets), but not against our own IPs and the WAN-interface shouldn't interfere with inter-LAN connections (unless there are something with this bridge-thing I migth have misunderstood).
When it is out of state, could this be due to a network thing like above and that I should not look at pfSense to debug this error?
In the attachment, you will see my interfaces and the bridge. I have just deleted (as you see on the image) the OPT3 virtual interface attached to the bridge in bridge.png, I assume it wasn't needed there in this setup as it didn't change anything.
Update: I have run a commercial tool to detect duplicate IPs, and it found about 10 - but it was because they where in teams/bond/LAG-mode. Just to be sure, I have disabled one interface on every team and now it shows no duplicate ip. But still have problems.
Update 2: Things looks more stable once I disabled the redundancy in teaming/bonding. Maybe an error in software-setup on one server unless you see any errors in my setup. This teaming feature seems to change mac-address often because of the balancing (alb), maybe pfSense or something is getting confused by this?
bridge.png
bridge.png_thumb
ifaces.png_thumb
ifaces.png