Filtering Bridge with Spanning Tree problem?
I've got a PC Engines 3 NIC WRAP board running pfSense RC2 on a 256MB CF card, which I placed in front of my colocated servers in a datacenter. It's set up much like the m0n0wall "Filtered Bridge" setup from http://doc.m0n0.ch/handbook/examples-filtered-bridge.html
LAN = DHCP with NAT for internal admin access only.
WAN = Static IP on same subnet as Colo Provider's port on Switch.
OPT1 / Servers = Bridged to WAN connection, and plugs into my Netgear non-managed switch to distribute connection to my servers which are plugged into the switch.
I'm doing basic firewalling - opening access to essential ports for webhosting/email/ftp etc., providing access to admin ports for ssh/RDP from admin's static IPs, and dropping everything else. No shaping/VPN/NAT/VOIP or anything fancy.
The system was installed for 5 days and was working fine, but then started causing problems with my provider's switch and the vLan that I'm on. "we think is happening and from our switch logs, spanning tree is running on your firewall and become the root node,
causing all vlan10 traffic to go to your bridge instead of our core." While it was still online, pfSense RRD graphs show large volumes of incoming WAN traffic that confirm what was happening with the traffic.
What i don't know is:
what impact a Spanning Tree has on my side of the network or how to disable it;
if I switched back to a non-bridged setup if it would be likely to help;
why it worked fine for 5 days before causing problems;
why this setup (that is replicated in another datacenter and works fine) isn't working!
Sidenote: I am aware that this issue has security implications on the switch into which my WAN port is plugged in, but I can't control that!
If anyone can explain what is happening or, even better, suggest ways for me to resolve the problem I'd be most grateful. I've got access to pfSense via the LAN (non-routable) interface, but it's not connected to the 'net at the moment.
Many thanks for your help, and thanks to the PTB for a great product!
Most likely some kind of ethernetloop problem. First upgrade to the latest snapshot at http://pfsense.com/~sullrich/1.0-SNAPSHOT-09-18-06/ . We introduced some bridge status infos at system>interfaces for bridged interfaces. If you see a red "blocking" there for the red interfaces something is causing a loop. In the past these problems always have proved to be a faulty switchconfiguration and/or wrong patching.
Btw, the m0n0 tutorialdoesn't fully apply to pfSense. The bridge in use with pfSense is completely different and also supports the spanning tree protocol.
Thanks for getting back to me so quickly. I've applied the update, but the Bridge0 status is just set to learning at the moment because sis1 is offline.
I understand what you mean about the m0n0 config - I mainly used it as a network diagram, but I did follow the instructions such as "14.3.4. Enable Filtering Bridge. Go to the System -> Advanced page and check the "Enable filtering bridge" box. Click Save." If I should not have done this please let me know.
Is the spanning tree protocol support in pfSense optional, and if so where can I disable it? I researched and now understand a little more about the protocol, but I don't see any way of configuring it. From what little I understand, I need a way to ensure that the "spanning tree algorithm" worked out between my provider's switch and my pfSense box understands that the Primary bridge is that on the upstream switch, rather than my pfSense box. In fact, nothing on other than the machines connected to OPT1 should be able to use my pfSense bridge …?
Lastly, a question that I should have asked initially; Is my setup the right way of doing it, or would I be better off dropping these bridging + spanning tree complications and moving back to a more traditional external/internal assigned IP address routing solution?
Thanks for your help,
Oh, you should up all your interfaces. A bridge won't pass traffic if the interface it is bridged to is down. This is a difference between m0n0 and pfSense as well. The STP is not optional, it's always enabled for bridges. Also the filtering bridge option works a bit different between m0n0 and pfSense. in pfSense the filtering bridge only affects traffic between the bridged interfaces. If disabled traffic between the two bridged interfaces is passed without filtering. However going to other interfaces/subnets still needs a firewallrule to be allowed. In your case as you want to filter all traffic between the two interfaces you should enable filtering bridge.
I guess your mainproblem is that one of the bridgedmembers is down. Try to uplink it and see if it works.
Thanks Hoba. The interfaces are down because that was the only way to restore service to all the other customers on my upstream's switch/vlan, but I do understand it won't work until it's plugged in!
My concern is that we haven't changed anything in my config, so it will cause problems and need to be taken offline again. It sounds to me as if I need to move away from bridging if it requires STP with no control, as that is what is causing the problems within my current environment. I'll pass on the looping comments to my provider and see if it helps them, but for now my #1 priority is to get firewalled up again ASAP.
Here's the latest update. Earlier I disabled the bridging on the OPT1 interface to bridge with the WAN interface, and unchecked "Enable filtering bridge" in advanced setup. The IPs are configured as follows:
xxx.xxx.113.1 - Port on providers switch. Used as GW for WAN connection below.
xxx.xxx.113.2 - WAN Port on pfSense Box
xxx.xxx.113.14 - OPT1 Port on pfSense Box
xxx.xxx.113.3 - Server1 - Uses xxx.xxx.113.14 as GW
xxx.xxx.113.4 - Server2 - Uses xxx.xxx.113.14 as GW
Unfortunately this config isn't working, and despite having enabled ICMP traffic through the WAN interface from/to all hosts, and temporarily enabling all outbound traffic (wiht logging) on the Servers port, I couldn't connect or ping the servers with the new config.
Is there any chance it has to do with the snapshot I'm now running, and if so can I downgrade?
I'll retest bridge this evening with latest snapshot but I bet it's something else (tested it not too long ago already).