STP and network
-
Do you mean the LAN-side of the configuration since I have public IP-space instead of doing NAT?
No. I mean using a blank VLAN on the same switch stack for the outside. The ISP side.
The firewall does not change just because you have public addresses inside. You just get to skip the NAT step.
-
The point Derelict is taking about is this scenario. That some places will frown on.
Where the public traffic (vlan) flows through the same physical switch and lan side traffic, vs using 2 different physical switches for for traffic outside the firewall and traffic behind the firewall. See attached.
Unless your running some sort of dod facility where this is mandated, its not an issue as long as you vlan the switch correctly so the traffic is isolated from each other at the switch and has to flow over the firewall. If you have it misconfigured, then its possible for vlan bleed through, etc. Which is not possible if you use 2 different physical switch(stacks) for outside and inside traffic.
It is common practice though to use different physical switches.. But it is not a requirement for sure.. You see this most common in smaller setups where the wan runs through the same physical hardware. As a company grows to enterprise size normally they tend to go with different physical hardware for wan and local traffic. We run multiple different physical switch stacks.. There is the customer side switches, admin side, internet side, dmz side.. All of which have multiple vlans on them - but the physical hardware is normally dedicated to a specific zone of traffic..
All of these different zone will have multiple switches in them core, distribution, access, etc.. Depending on the size of the zone - some customers don't mind sharing hardware for cost savings - but some customers may "require" different physical hardware their setups, etc.
But if your limited in hardware its fine to run different zones of traffic types on the same physical switching hardware as long as you properly vlan it.
-
I'm still a but curios to how this traffic can get through :p Purely physical, the traffic HAS to go though the fw and the WAN-side-danger-side in order to get to my LAN. I have no switches before my WAN as you are correctly pointing out. You are saying that traffic still may overcome this and get through in some situations?
In the drawing, the 2nd one is more how I have it. You have the Internet (the sky), the router (my ISPs equipment, still hostile), then my fw WAN-side and then another physical different port (LAN) where my switches is connected. The first drawing would be true if I had both a switch and a fw connected to my ISP.
When I'm testing if a firewall works, I do this by using nmap outside my fw (I do this by automated scripting, just as a precaution due to earlier mistakes in rules), in addition to checking if the service is accessible. Are what you talking about something that would not show up on such a test or are you more saying that if I do something wrong, THEN I could open up the traffic to bypass the physical fw? Are you talking on a logic level and somehow the fw would bypass checking it because it is on another VLAN not detected by fw - since I have a kind of accessible network on my LAN?
-
When you do a transparent, filtering bridge it makes sense to filter on the bridge interfaces, with rules like "WAN" rules on the interface connected to the ISP and rules like "LAN" rules on the interface connected to the inside hosts.
When you try to make a "switch" it makes sense to filter on the bridge interface, with no rules on the interfaces themselves and "LAN"-type rules on the bridge.
-
"can get through :p Purely physical"
If you are physically isolated than it can not… I think we are getting a bit off topic to the discussion at hand. But I think it came up with how your actually connected to the ISP be it via CE or PE.. If you were passing the traffic from wan "internet or hostile" through your CE (your equipment in this case Customer Equipment) that the your local traffic also flowed through then there it is possible for barrier incursions.. If the switch is mis configured..
I think this sort of discussion is beyond what the actual question was ;)
I think where we want to get with this is how you connect the Provider network to your network.. And how we mitigate any sort of SPOF issues.. Right.. We want to make the setup as robust as possible and minimize all SPOF (single point of failure)..
So you want lagg to a stacked switching setup both on the Provider side and your Side the Customer.. Do you own/manage the switching infrastructure before pfsense.. Or is the wire all you get from the provider equipment that you are going to plug direct into your pfsense hardware.. And then you manage all the switches behind pfsense?
-
"Do you own/manage the switching infrastructure before pfsense."
No - it is in my rack/building, but I do not have access to its configuration. My ISPs Cisco Catalyst (where I get two ports I can connect to) -> My two pfSense (CARP) -> My two switches (now stacked). This Cisco Catalyst is basically my only single point of failture (but I can call my ISP, get them to go to my location and replace it for me - I can't do that with my stuff).
I think the action-plan is like shown above in my steps. One cable from my ISPs Catalyst to WAN on fw1 and also one to WAN on fw2. Using CARP and sync (dedicated interface with local IP)with one public IP/VIP on WAN-side of the cluster so that my ISP can "put" or route the /24-network to that single IP.
Each pfSense will have a LAG (two ports) that I will name "LAN" (with a .2 and .3 local IP set on each of the two pfSense-servers) that connects to the Switch-stack through LACP (one port from each switch into the LACP-group).
With this setup, I have the hostile/WAN on one side and it is enough for me to add firewallrule on WAN-side against the internal public static IP - along with ports - to let traffic flow. No NAT and no bridge (I think ;).
Any change in this solution? I mean, this looks very similar to the transparent bridge I had, but now it is just a bit more redundant. And all I really wanted was the redundancy on all my own stuff.
-
My ISP says they can fix a /29 linknet this way, I will have it available on both ports. It will be downtime in relation with this they say (not the linknet, but the final routing of the /24 from the old method to the new one).
So far things looks good unless my drawing is in-correct ;) They also say I need to route the traffic back myself, but I assume they are just talking about the individual-hosts on my LAN (setting gw=my local CARP VIP=4.4.4.1). This way, all servers have a route back to the pfSense and pfSense of course have the CARP-VIP=8.8.8.1 on WAN-interface with the transport-IP I get from ISP as GW. No more routing is needed, correct?
Any potential showstoppers in this?
They will start by giving me the /29 linknet, så I can prepare one of the pfSense and test some functionality before live.
-
Anything that shouldn't be possible with the drawing?
-
I would rather control my own WAN switch. As long as that switch handles CARP properly it should work like that.
-
I would rather control my own WAN switch. As long as that switch handles CARP properly it should work like that.
Is there a way my ISPs Cisco Catalyst would not support CARP, so I need to check that? I mean, isn't CARP just to present a single (V)IP? pfSense and its CARP system will choose between themselves what pfSense of the two that should actually be handling the traffic and bind to the (V)IP. So the primary VIP/WAN VIP, is only available at one of the time.
I have presented the drawing to the ISP and they haven't said anything negative, but I assume they are not familiar with how pfSense works.
-
IDK dude. You'll have to ask them. ISP switches and switching gear is always broken. Always.
Instead of using the term CARP use VRRP. Same requirements apply.
-
Or if cisco they might understand HSRP.. Same concept..
The gateway they give you would be a VIP.. etc..
-
Ok, so it's not like using CARP is a switch-independent solution, it needs support also outside the two firewalls. Catalyst is a router from Cisco, so it's definitely Cisco-environment on their end.
Note that they have only one, so I'm not looking at having their equipment redundant (just using two ports).
-
is a switch-independent solution
It is, but uses Multicast Groups and MAC addresses, that are reserved vor HA usage. The MAC address is defined via the VHID Group setting of your CARP style VIP. If it uses the same ID as the VIP on your provider side, that can have NASTY side effects. I had that problem with our ISP that said multiple times, they were NOT using ID 1-10 with their VRRP setup, so I let mine use ID 1. After numerous routing bugs and weird traceroutes I contacted a techy on their end and: yeah we use ID1 - weird, that should have been >100 for you. Switched the ID to 4 and no problems anymore :)
But besides that and multicast, I'm not aware of any deep configurational stuff that the switch would have to support.
Greets
-
Almost all CARP issues must be corrected in the switch. ISPs in particular do stupid stuff.
For instance a co-worker is trying to get CARP working on a cable modem service with a static /29. When the secondary ARPs for the CARP VIP address, it (properly) gets the CARP MAC from the primary AND gets an ARP reply from the ISP device containing the primary's interface MAC address. Thus, broken HA/CARP. We can't see it but I assume it is also proxy-ARPing upstream to the ISP gateway which will break HA too.
That has to be corrected by the ISP and they have been unable to understand the issue much less fix it.
Just an example.
-
It is correct that I don't need any routing? I mean, internally, LAN-side, I will have my LAN CARP VIP .1 (/24) as gw (4.4.4.1) on all computers.
Since the pfSense has WAN CARP VIP on the /29 from my ISP (8.8.8.1) and I have verified that pfSense have internet, the LAN traffic should find the 8.8.8.1 on WAN-side "magically" (as long as the rule is allow for the IP and port/direction).
-
You need to:
Make sure your outbound NAT is set to use the CARP VIP
Make sure your inside clients are set to use the CARP VIP for services on the firewall such as default gateway, DNS services, etc.
-
I'm setting up the CARP VIP (on LAN-side) with the same GW it had/has before when my ISP/Catalyst was managing my GW. So I shouldn't need to change any of the clients.
I will continue to use the providers DNS to avoid changing to much stuff (I assume I can still do that…). So this shouldn't be needed to change either.
But the outbound NAT sounds important: I should choose Outbound NAT, then choose "LAN"-interface I assume. LAN-interface is the joined LAG of two ports. Then, in Address, the VIP for the the WAN VIP should be possible to select in the dropdown? And that's it?
During this setup, I want to allow all outgoing traffic from LAN, so I will let the rest be set to ANY.
-
When you run HA you have to make sure outbound NAT states are created on the CARP VIP not the interface address. Else you will experience dropped connections on failover because WAN address on the primary node is different that WAN address on the secondary node.
https://doc.pfsense.org/index.php/Configuring_pfSense_Hardware_Redundancy_(CARP)
-
LAN interface is different. You have to make sure that all of your LAN clients are given the LAN CARP address as their default gateway, DNS server (if applicable) etc.
Bottom line is you can't expect HA to just work. It does work fine but it requires additional configuration for things that are otherwise automatic. Such as outbound NAT, DHCP server attributes, etc.