Poor Man's HA?
-
First let me say that I am a Pfsense novice and this project is for my own home lab (and keeps me out of the bar). I recently migrated from Pfsense+ on a Netgate SG-3100 that was peridocally flaking out and forcing reboots (yeah, it was WAY overkill for my home network), to running Pfsense CE natively on an HP DL380 G6. Unfortunately the G6 had been mothballed for about 2 years and it too has started randomly rebooting itself 2-3 times/week. I have a 2nd, almost identical G6 (with a little less RAM) that I spun up and installed the same version of Pfsense and was hoping to create an HA configuration so I started by dumping the current config from server1 and imported the rules, NAT, and DHCP leases to server2, but manually installed the same packages (pfblocker-ng-devel, openvpn-client-export, ntopng, apcupsd) so effectively they match.
Then I read the docs about CARP/pfsync and unfortunately they are over my head. I also don't have sufficient and sophisticated enough switches for the HA configuration, so I was wondering if I could just keep server2 up to date and matching server1 periodically backup and FULLY restore server1's config.xml to server2(maybe with an scp command)? Can I import the CA and certificates from server1 to server2 as well? I realize that I would have to move cables around in the event of a failure.
Would this actually work? Thanks for your time and patience and please don't destroy me for my inexperience/ignorance:)
-
Yes you could keep backups of the config and restore to another router. The main advantage of HA/CARP is that if router1 goes down, router2 takes over and anyone using the Internet doesn't know it went down. The states have to sync for that to happen though, so the networking hardware has to basically be identical (the same drivers for the interfaces).
-
Thanks for the confirmation. The servers are literally identical with the same NICs and pfsense bare metal installation but its the switches that exceed my grasp. I'll try and import the full config from router1 to router2 and see how it goes.
-
There's really only two switches needed, one on the WAN side and one for LAN. The sync interface can just be a cable between the two. It does need 3 NICs in each router.
-
@steveits
Ok, now you have me interested in causing myself some trouble ... Apparently I looked at the wrong diagram in the config guide, got intimidated, and backed off. I did not see this one. I do have multiple switches, primarily because I have several POE cameras and ran out of POE ports so I had to pick up another switch. They are all Netgear Prosafe gigabit switches two with POE, one without (JGS524PE, JGS516PE, and JGS524E), and I do have several open ports on each. I also have a few smaller "desktop" gigabit switches that could be put into play as well though they are not smart (and neither am I when it comes to VLANs). I'll take another look at the link you sent. Thanks. -
Sorry, forgot to mention that the DL380 G6 servers each have 5 on-board NICs.
-
@gordon-shumway I set up a dual-node pfSense HA config in my lab and it was pretty smooth if you follow their instructions to the letter. It helps if you start both nodes off with factory defaults and only the interfaces defined, then do the HA config until you get it working, then add your packages and rules.
-
I've been reading the doc and its making sense but there is some thing that still confuses me. I have a relatively new ATT 1G fiber optic modem/router (NOKIA BGW320-505) with 4 x 1G ports. I have one network cable running from a modem port (#2) to the WAN interface on the pfsense server.
The ATT modem device config menu (Firewall>IP Passthrough) is currently set to what ATT calls "IP Passthrough" mode. The "Passthrough Mode" is "DHCPS-fixed" and the "Passthrough Fixed MAC Address" is set to the MAC of the physical WAN port on my "WAN" NIC . Also, in Home Network > Subnets & DHCP, the DHCP server is on and the DHCP range is 192.168.1.64 through 253.
FWIW, there is also an item called "Cascaded Router" (which is currently OFF, that allows me to enter a Cascaded Router Address, Network Address, and Subnet Mask.
My pfsense WAN interface is set to DHCP and it is picking up the DHCP address from the modem (99.7.nn.nn). So that is ONE IP address coming from the modem to my pfsense box.
I apologize for the details above but I don't understand how physically setting the WAN IP addresses on the two pfsense boxes, connected to the WAN switch with the ATT modem, are going to move traffic from the modem to the pfsense network.
Again, please excuse my ignorance of how networks work but I thought when you connect to a router that is serving DHCP you take what range it gives you, not dictate your network by physically assigning its IP.
Thanks again for your time.
-
CARP/HA is not going to work well with a DHCP Internet connection. You want to have at least three public IPs available. One for the main firewall, one for the backup, and at least one as a shared public IP that can float between the two nodes.
-
@dotdash Agreed that is the intent. It can be done with NATted IPs though...
router1 WAN 10.0.0.3
router2 WAN 10.0.0.4
WAN CARP/shared IP 10.0.0.2
WAN gateway (ISP router) 10.0.0.1However to handle incoming connection on WAN the 10.0.0.2 IP would need to be set as a DMZ target. The AT&T passthrough sends the public IP to the router which isn't going to work. Gordon does your AT&T router have a DMZ option that can be set to an IP address? That might be what Cascaded means, if it lets you specify an IP and not a MAC address.
-
@steveits
Speaking theoretically, (I wouldn't want to put anyone through the pain of having to touch one of those terrible ATT modems) CARP virtual IPs do have a unique MAC address. The problem is the virtual IP cannot (AFAIK) be set to request DHCP, and the ATT modem probably won't let you do DMZ host to something set static. -
Its my understanding that the "cable modem", no matter who provides it, is going to give me a single "real" IP address from their block. If I use their cable modem/router's DHCP server then all my stuff is NATted from there on out and I'm screwed out of most of the functionality of pfsense. So I don't want to do that.
If I turn off their DHCP server and set up a "passthrough" configuration then my pfsense router gets the real IP address. Is that not the same effect of setting my router to be the DMZ device?
Their passthrough feature offers me the option to select/input the MAC address of my router (pfsense) anyways, and that is what I have always used. Could I just feed it the MAC of the WAN CARP/VIP to the passthrough screen
? -
In cases where at least a /29 static subnet is not available from the ISP it works OK to have CARP only on the LAN side.
The only downside is that on failover any active Internet connections will be dropped and have to be re-established which usually happens within a few seconds.
This allows each pfSense appliance to be DHCP, etc, and still have HA. -
@gordon-shumway said in Poor Man's HA?:
If I use their cable modem/router's DHCP server then all my stuff is NATted from there on out and I'm screwed out of most of the functionality of pfsense. So I don't want to do that.
First you don't need to use their DHCP server but if their cable modem is switched to router mode and the IP is there - if the device can forward it (something like exposed host, DMZ or anything else how they call it) then you can setup a private network like a transfer network between the modem/router and your pfSense boxes. We have that kind of setup in dozens of customer locations as often ISPs only hand out dynamic IPs or one static IP so to set CARP up the right way we do NAT. It's not the nicest setup, sure, but you aren't "screwed out of most functionality". Who said that and why do you think that? There's only one or two things you have to be aware of but a NATted setup isn't screwed over or the end of the world. It's a necessary evil and we try to avoid it whereever we can. :)
Cheers
-
@gordon-shumway said in Poor Man's HA?:
set up a "passthrough" configuration then my pfsense router gets the real IP address. Is that not the same effect of setting my router to be the DMZ
Yes, but you would want the shared IP to be the passthrough. So something like this, if you can make it work:
router 1 WAN: 192.168.0.100
router 2 WAN: 192.168.0.101
shared IP: (the public IP from passthrough) -
I don't mind having some level of automated failover on LAN side only. This HA idea only came to me after experiencing multiple failures of my SG-3100 and deciding to move to pfsense CE on an Ubuntu KVM as it was recreatable and servicable faster than buying a new Netgate appliance if it died (working from home).
I have multiple desktop PC's running Ubuntu Server with dockers and NAS but I really did not put my firewall into that mix so I resurrected the DL380 G6's. My employer had a "back door sale" and I got them cheap so I was running ESXi on them just to learn about it ESX.
Apparently I have a lot of toys, and sometimes don't know any better, so I tend to over complicate things, but ultimately this doesn't have to be a bullet proof enterprise class solution. I can do sneaker net and walk down to the basement and move a cable if a server dies.
If I can get two pfsense instances running on two identical servers and do some level of automated failover that would be cool. If that doesn't work out because of the ATT modem's restrictions and I have to copy the config from router1 to router2 daily and move cables during a failure, that's ok too.
I'll keep reading the comments until they make sense or my eyes blur, and probably switch back to the pfsense running on the Ubuntu KVM either tonight, or this weekend while I play with HA on the G6's. Thanks everyone for the help and the great ideas.
