STP and network
-
Thank you for info, I'm probably going for stack-hardware. The cost isn't that big compared to the ones I have, but the benefits looks big.
I can get this for probably 30% lower price than Cisco, plus it has 4x10 Gbit SFP+ stacking ports compared to 2x1 Gbit from Cisco: D-Link SmartPro DGS-1510-52X
Cisco has a stronger name/brand, but I think their UI is a bit targeted against professionals and doesn't give that much info.But my question remains: How may I use LACP-team on the pfSense when I have transparent mode on (since I can't choose any of the LAN-ports).. I will try it later today in a spare pfSense, I have a theory that maybe it works if I remove the bridge, then activate the LACP-ports and after that join the ports into the bridge again. Or maybe it wouldn't work.. If anyone knows if this is possible, you would spare a lot of time if you could say so now…
-
Yeah, I was correct it seems :) Had to deactivate all LAN-interfaces and then I could create the LACP-team (was created as LAN) and then bridge WAN and LAN.
However, I was not able to ping anything on the LAN-interface. I have enable/assigned the LAN-interface and it shows up as connected. But nothing comes through. I was able to ping the gw from the console, but noting on the LAN.
I have a any-any on the LAN in fw rules.
-
"I have a any-any on the LAN in fw rules."
What about your bridge rules - thought you wanted this to be a transparent firewall?
-
"I have a any-any on the LAN in fw rules."
What about your bridge rules - thought you wanted this to be a transparent firewall?
Yes, I do. So you are indicating that I'm missing any fw rules on the virtual interface (like OPT3) I activated with the bridge and need to create an any-rule there as well? I thougth I had, but have to go back to data center ot be sure. Please let me know if that was what you ment or not.
I have had it working as a transparent firewall/bridge for a year or so, that part I know is possible, but maybe there are some details I'm overlooking now…
-
depends on how you setup the bridge..
https://doc.pfsense.org/index.php/Interface_Bridges
Do you have
net.link.bridge.pfil_member and net.link.bridge.pfil_bridge under System > Advanced on the System Tunables tab. With them set at 0 and 1, respectively, then filtering would be performed on the bridge only. -
This was my setting just now (somehow, it has changed since last time - I restored the pfSense backup to a new server and maybe lost some config):
net.link.bridge.pfil_bridge Packet filter on the bridge interface 1
net.link.bridge.pfil_member Packet filter on the member interface 1From documentation, it looks like I'm supposed to only have one of them set to 1. I changed it to be:
net.link.bridge.pfil_bridge Packet filter on the bridge interface 1
net.link.bridge.pfil_member Packet filter on the member interface 0This should control the traffic onto the bridge only and not between the local interfaces. But this last step made all traffic bypass the firewall-rules I have on WAN-side as well… I could connect to computers over the Internet that I did not have opened up for. Is this because the Bridge-interface is controlling traffic both directions? How could I control it only one way?
I was under the impression that when I have a bridge, I can control the traffic from the Internet-side (WAN) and onto the bridge combined (LAN1, OPT1 etc).
How would I set this up so that I can control the traffic from WAN-side and in from the Internet - I do not need to restrict the traffic out from local side and out on the Internet. I have all rules on WAN-side today.
-
So what I want..
1. Create team (LACP) on pfSense (with two physical interfaces, LAN1, OPT1). The new joined local interface will be called LAN and will be connected against two stacked switched with LACP there also. This part is easy to do as far as I can tell and the interface appear as LAN as it should.
2. Create bridge with WAN and LAN, where I will have rules for incoming traffic from the Internet on the WAN-side. My ISPs gw is also on the WAN-side. Seems easy as well.
3. Add Bridge to a virtual interface, like OPT3?
4. Maybe using pfil_member=1, pfil_bridge=0 against the LACP team is the correct choice instead of the normal pfil_bridge setting in this case? So that I can control traffic one direction only.
I'm having public static IPs on my webservers on the LAN-side, that is the only reason why I have transparent fw setup.
Please let me know the correct settings in this scenario or at least an example that should work.
-
The absolute best thing to do is get your upstream to assign a small interface address for your WAN and ROUTE the subnet of addresses to you.
Then you can just put the routed subnet on an inside interface and forget about this transparent bridge stuff.
Have you asked them if they can do that?
-
I thougth I was close to a solution now, so would prefer to make it on my own. No, I though I had the simplest solution already ;p They assign a public range for me and I can just use it on servers. And I don't have to do NAT for every service and so on.
I come from using FortiGate and this was pretty straigthforward without to much technical knowledge of networks. But now I have to actually understand things ;)
Do you mean to assign a local range of IPs instead, that are fewer than I have today on a different subnet?
Do you have a way to make it like I have it now, so I can compare the methods?
-
You get a WAN interface of, say 198.51.100.32/30. Your default gateway is 198.51.100.33 and your interface is 198.51.100.34/30.
They route 203.0.113.64/29 to 198.51.100.34.
You put 203.0.113.65/29 on an inside interface and turn off NAT.
You give hosts on that network 203.0.113.66 - 203.0.113.70.
No bridging mess.
No NAT.
Exactly how it's supposed to be.
-
Ok, I have asked my ISP about this and wait for answer. I do also have some failover system that is not mentioned here, that happens transparent to me.
But, to have it transparent like today with the LACP-trunk, how would I do it? The way I have it working as of today, is apparantly by filtering on the member interface. As soon as I filter only on the bridge, the traffic is loose. In my mind (without thinking about networks), it seems logical that new LAN-team-interface is beeing filtered this way.
-
Just to stress.. Having your public range routed to you is way better than any transparent/bridge nonsense ;) What size public range do you have? /29 is pretty small… But if /28 or bigger I would for sure think it should be routed to you vs just attached to their network.
I personally even if having to work with attached network vs routed would just nat it and use port forwards. Simple enough to use your specific IPs for different servers via vips..
-
Have /24-range, with 256-addresses. It hasn't been stressfull so far the last 10 years, since I don't do a lot of network-stuff or have any special routing/requirements. Think this is the first time I have had problems and that is because I want it to be more redundant by using LACP :)
Most of my servers are web-servers with control-panels that requires a certain IP to bind to (due to licenses). If I was to have local ips on all servers and have mapping to the public-ip for all servers, I suddenly have to mange 256-addresses * 2. And that is before I have to NAT all ports for common services like DirectAdmin and cPanel-servers use. Now I can simply group the servers based on profile.
But I'm sure there are ways to do this simple in NAT as well.
-
It is pretty much insane to have that network on your WAN interface. It should be routed to you instead.
-
You have a /24 and its not routed to you?? Wow.. That is nuts dude.. I would for sure change that.. put pfsense in carp, then get some stack switches between your pfsense carp and your servers and now your cooking with gas.. ;)
-
Have never even been thinking of that, or that there was any disadvantage of running it transparent. When starting this business, I was told that NAT-was slower (performance-wise) and required more setup. The FortiGate I started with supported that easy.
But basically, with your suggestion, I would get assigned additional small network with public static IPs just for my WAN-area. And I could then just remove the bridge on my LAN side and threat the public IPs like I would do on a private network? I don't have any NAT today, so wouldn't have to change there.
Based on this, I shouldn't even have to change the fw rules I think, so that's a good thing. Let's see what my ISP says, maybe there are some kind of setup here that differ from the normal. But I'm still curious to how I would complete the setup in case my ISP says no..
-
Wish I could be more help with bridges on pfsense. But software bridges should be avoided at all costs if you ask me. While your use of it is very valid setup when under the restrictions of having to have a public range directly attached vs routed and wanting to put a firewall in between.
So while your use of transparent is valid, I would suggest if possible migrate away from it. If you had small amount of space like a /29 or even /28 nat with vip and then 1:1 would remove your issues of having to deal with port forwards.. Doesn't remove the issue if you have software licensed to some IP… What if you loose your public space? Do you actually own this /24 in arin or whatever RIR you might be in? If so you should be able to get your own ASN and just route it yourself to wherever you want via your ISP your using, etc.
I manage a /16 from arin.. So never run into these sorts of issues. We just advertise the space we need to use where ever, and be done with it ;) You just need to work with whatever ISP to accept and advertise out your routes, etc.
But if you just got said /24 from some DC network your located in - they really shouldn't have any issues with routing it to you vs directly attaching it to their equipment.
-
What if you loose your public space? Do you actually own this /24 in arin or whatever RIR you might be in?
I don't own it, just renting it as long as I need it. If I was to change ISP, it would be a bit difficult (lot of dns to change..), but not impossible. Can change the IP for the license from control-panels.
OK, I'll hope that my ISP comes with good news and if not, I'll just have to try and fail until it works ;)
-
I have two ports in my datacenter assigned with my current C-net/24-net (an Catalyst owned/administered by my ISP only). Both of the ports work, I currently only use one of them.
I have two pfSense and I consider doing CARP on them (instead of having just a cold-turned-off ready) and have one port to each of my fw.
Would I ask my ISP for a network of two IPs and then assign WAN-IP to each of the pfSense.. or do they need to route it to only one IP/device? I'm beginning to suspect that your suggestion - along with a carp setup - requires me to introduce one additional switch/router. Then I migth as well keep the cold-backup to avoid introducing (more) single-point of failture.
"You get a WAN interface of, say 198.51.100.32/30. Your default gateway is 198.51.100.33 and your interface is 198.51.100.34/30."
BTW: is this reccomendation just because how pfSense works with bridge or would you reccomend this setup no matter what type of firewall?
-
If you need more address on your wan transit network for a carp, then use a /29 vs /30.. Yes carp requires 3 IPs.. So /30 wouldn't work.
-
Doesn't carp communicate over a local IP only? So a dedicated cable on a port not part of LAN/WAN, just a dedicated cable with virtual 10.0.0.1/10.0.0.2 on each?
And then one public WAN-IP for each.
-
My ISP says this:
"This should be possible yes - the challenge is how to be able to route the current network, if you have two firewalls at 2 different WAN-addresses. Without major changes, I can only route /24-network to one address.
-
you could always split the /24 into 2 /25s and route 1 to each.. All comes down to how you want it. Or if you setup carp on your 2 firewalls then you would only be routing to 1 IP, the CARP address on your wan side.
I would have to go back and read the thread if you had laid out how you have your 2 firewalls setup and different networks behind them, etc.
-
"Or if you setup carp on your 2 firewalls then you would only be routing to 1 IP, the CARP address on your wan sid"
This is the prefered method, but I assumed it wasn't an option? If so, it is perfect!
Let's say that they assign a 5 public static IP-transport-network to me, where 80.80.80.81 is the main/assigned interface. The fw1 gets .82 and fw2 gets .83.
I create a local link between a free interface on both, with two static local IPs to maintain the carp… and I put .81 on the cluster.
Is it as simple as that? If so, it would be pefect, but I assume it is more to it ;)
-
Like the drawing attached. I'm using fake static IPs of course, but maybe it is more clear what I want to do?
The 4.4.4.0-network indicate the current /24 network I'm assigned today. I wouldn't need to change the servers from what I have today (I think)
The 8.8.8.0-network indicate the new small transport-network, that will be assigned both WAN and the cluster/CARP on WAN-side.
-
Nope its really that simple ;)
https://doc.pfsense.org/index.php/Configuring_pfSense_Hardware_Redundancy_(CARP)
I have not read thru that doc in awhile - so maybe its a bit dated, maybe something has changed in newer versions. But yeah its pretty simple to setup the carp..
This shows a nat network behind - but you could put your routed network behind there two.. You setup your stack switches and some laggs - and yeah buddy cooking with gas.. And remove all your SOPF issues.
-
I think it was the routed network that made me thing it wasn't possible.
What would my GW be on the inside on each machine, would it be the same as the cluster IP from the transport network like 8.8.8.1 in my drawing? Or can I create additional interface on the cluster (virtual IP or something) so that I can have the same gw as today? (4.4.4.1).
-
sure just use that IP of your routed segment as your carp on the "lan" side of pfsense.. Before you had this
PE (provider equipment) 4.4.4.1 –-- 4.4.4/24 ---- CE (pfsense - BRIDGE) ---- 4.4.4/24 ---- 4.4.4.X Server..
You end up with this
PE x.x.x.1 --- transit x.x.x/29 ---- x.x.x.2,.3,.4 CE (pfsense CARP) 4.4.4.1, .2, .3 ---- 4.4.4/24 ---- 4.4.4.x Server
Does that help?
-
Please read this for a short explanation of the basic elements of building a CARP/HA pair: https://forum.pfsense.org/index.php?topic=136085.msg744802#msg744802
-
1. Get the fw1 to listen on WAN for IP 8.8.8.2, fw2 to 8.8.8.3 and using my ISP provided gateway for the new transport network.
2. Create a LACP-team to create interface called LANTEAM (two ports - same on switch cluster), with LANTEAM-IP 4.4.4.1/24.
3. Log in and set "CARP Shared Virtual IP Addresses" of type "CARP" on interface "WAN" 8.8.8.1 (the main transport IP).
4. Add another "Virtual IP Addresses" of type "CARP", this time on interface LANTEAM to 4.4.4.1 (my current and new gateway).
5. DirectConnect a TP between the fw1/fw2 on local, private IP and setup sync under "HighAvail".Is it like that or do I miss something important? I also need to add fw-rules of course.
-
1. Get the fw1 to listen on WAN for IP 8.8.8.2, fw2 to 8.8.8.3 and using my ISP provided gateway for the new transport network.
3. Log in and set "CARP Shared Virtual IP Addresses" of type "CARP" on interface "WAN" 8.8.8.1 (the main transport IP).Looks good so far…
2. Create a LACP-team to create interface called LANTEAM (two ports - same on switch cluster), with LANTEAM-IP 4.4.4.1/24.
I don't know what an LACP-team is. You might mean an LACP LAG. Team is some microsoft aberration.
If you want to LACP to the inside switches you will need to LACP from BOTH pfSense nodes (4 total ports or more). The first would be interface address 4.4.4.2/24, the second would get interface address 4.4.4.3/24
4. Add another "Virtual IP Addresses" of type "CARP", this time on interface LANTEAM to 4.4.4.1 (my current and new gateway).
Right. Tell all your LAN clients to use the CARP VIP as the default gateway, DNS server (if so required) etc.
5. DirectConnect a TP between the fw1/fw2 on local, private IP and setup sync under "HighAvail".
No idea what a TP is. Many people use a direct patch cable for their sync interfaces. Some use a switch on a "blank" vlan. Both work.
Is it like that or do I miss something important? I also need to add fw-rules of course.
Yes. And you need to adjust Outbound NAT so it NATs to the CARP VIP not to the interface addresses (for networks that might require NAT, that is).
-
With "direct patch cable ", you mean crossed cable? So that only one wire changes position in the other end?
The first would be interface address 4.4.4.2/24, the second would get interface address 4.4.4.3/24
Thank you for clarifying that. In my instruction I wrote 4.4.4.1 for the LAN and that would be wrong/conflict, that's only for the CARP virtual IP since the gw needs to be present for all clients on LAN. Regarding 4 ports must be in LAG, I assume you mean that I haven't drawn the last LACP LAG in my drawing above (but I think I understand that concept now at least).
Yes. And you need to adjust Outbound NAT so it NATs to the CARP VIP not to the interface addresses (for networks that might require NAT, that is).
Here I need to follow up with a question, just to be sure.. I don't think I want NAT in my case, since the server already has the correct IP and port assigned to it (public static IP and the ports is what they are).
Do I need to do any NAT or port-forwarding/translation with this setup? My goal is to avoid both NAT and bridge and hopefully get a fw that acts similar to a transparent gw in the sense that I only need to add the public IP and ports in the firewall-rules for all incoming traffic. Most or all traffic coming from the LAN-side should pass though without problems and with their own server IP as outgoing IP. Please let me know if this is not the case :)
-
Let me know if this drawing is correct. The goal is to have redundancy against one failing switch and one failing pfSense fw (or one cable).
-
In that configuration you are trusting the ISP switch to properly-propagate the CARP traffic on the WAN interfaces which might or might not be the case.
Also, if the WAN link stays up and CARP continues to pass but there is not internet access there will be no failover. But there probably won't be any internet for the secondary either so… It is possible for a strange layer 2 issue that could cause that.
I would rather have a switch under my control connected to WAN and the ISP. Preferably another stack and preferably LACP as in my diagram.
Note that, if you are very careful, you can use a blank VLAN (blank meaning no layer 3 and no other ports configured on it) on the existing stack for the wan traffic. Many (including me) do not particularly like mixing inside and outside traffic on one switch but in practice it can be done safely.
-
"Many (including me) do not particularly like mixing inside and outside traffic on one switch but in practice it can be done safely."
This is very common practice in the enterprise for sure. But if budget, space, power constraints, etc I concur it can be done safely.. Just make sure you know what your doing with vlans and your fine.
-
Many (including me) do not particularly like mixing inside and outside traffic on one switch but in practice it can be done safely.
Do you mean the LAN-side of the configuration since I have public IP-space instead of doing NAT?
I have a seperate network behind this that is not connected to the common network at all (for IPMI, console, NAS, monitoring on dedicated switch and port etc). The traffic on the LAN is mostly https/https and all equipment has their firewalls, so hopefully it is safe enough. It is the setup I usually see when I rent space in other data centers as well. But probably not in an office where you just have a few VPN, AP and maybe no ports incoming. If I was to do NAT for all servers and its services using port-forwarding, we would talk thousands of rules in the fw and I doubt it would handle it very well - at least it would be messy compared to gathering the same type of servers/ports in common groups like "cPanel-serverports" :) Also, the license-validation done by cPanel, DirectAdmin etc. would need to match at all times, reverse.. etc. And the client would be given a private IP so that install of software works.. I see a lot of issues doing it any other way…
-
Do you mean the LAN-side of the configuration since I have public IP-space instead of doing NAT?
No. I mean using a blank VLAN on the same switch stack for the outside. The ISP side.
The firewall does not change just because you have public addresses inside. You just get to skip the NAT step.
-
The point Derelict is taking about is this scenario. That some places will frown on.
Where the public traffic (vlan) flows through the same physical switch and lan side traffic, vs using 2 different physical switches for for traffic outside the firewall and traffic behind the firewall. See attached.
Unless your running some sort of dod facility where this is mandated, its not an issue as long as you vlan the switch correctly so the traffic is isolated from each other at the switch and has to flow over the firewall. If you have it misconfigured, then its possible for vlan bleed through, etc. Which is not possible if you use 2 different physical switch(stacks) for outside and inside traffic.
It is common practice though to use different physical switches.. But it is not a requirement for sure.. You see this most common in smaller setups where the wan runs through the same physical hardware. As a company grows to enterprise size normally they tend to go with different physical hardware for wan and local traffic. We run multiple different physical switch stacks.. There is the customer side switches, admin side, internet side, dmz side.. All of which have multiple vlans on them - but the physical hardware is normally dedicated to a specific zone of traffic..
All of these different zone will have multiple switches in them core, distribution, access, etc.. Depending on the size of the zone - some customers don't mind sharing hardware for cost savings - but some customers may "require" different physical hardware their setups, etc.
But if your limited in hardware its fine to run different zones of traffic types on the same physical switching hardware as long as you properly vlan it.
-
I'm still a but curios to how this traffic can get through :p Purely physical, the traffic HAS to go though the fw and the WAN-side-danger-side in order to get to my LAN. I have no switches before my WAN as you are correctly pointing out. You are saying that traffic still may overcome this and get through in some situations?
In the drawing, the 2nd one is more how I have it. You have the Internet (the sky), the router (my ISPs equipment, still hostile), then my fw WAN-side and then another physical different port (LAN) where my switches is connected. The first drawing would be true if I had both a switch and a fw connected to my ISP.
When I'm testing if a firewall works, I do this by using nmap outside my fw (I do this by automated scripting, just as a precaution due to earlier mistakes in rules), in addition to checking if the service is accessible. Are what you talking about something that would not show up on such a test or are you more saying that if I do something wrong, THEN I could open up the traffic to bypass the physical fw? Are you talking on a logic level and somehow the fw would bypass checking it because it is on another VLAN not detected by fw - since I have a kind of accessible network on my LAN?
-
When you do a transparent, filtering bridge it makes sense to filter on the bridge interfaces, with rules like "WAN" rules on the interface connected to the ISP and rules like "LAN" rules on the interface connected to the inside hosts.
When you try to make a "switch" it makes sense to filter on the bridge interface, with no rules on the interfaces themselves and "LAN"-type rules on the bridge.