Power Failure Bulletproof ::: yes! I have half-ton of UPS.
-
Hello dear friends,
I have a question (please, please, just the question):
:: What tips and tricks can we use in Pfsense in order to make it more reliable to power failure.
This is a common question that I really DON'T SEE answered. People just "oh, put an UPS", here are the facts:
- Not everyone lives in perfect water and power, in Europe or USA, this means third country and lots of dificulties.
- OpenVPN Server in HQ
- Several sites, distance by days, very hard roads (jungle, desert, savanna, mountain)
- 480 Ah capacity UPS
- CANNOT lock away the relays, it would be really unsafe
- Positive side: The solution works great, I've unified all offices in a big and difficult countrie. No random crash fortunately.
BUT sometimes, an iluminated user, shuts it down (sometimes on purpose, sometimes theres no grid power, no generator, and the UPS looks great to the cofee machine)
So eventually (it is rare) I get an office that gets offline, wich means a 4x4 getting out with 2/3 people, 1, 2 maybe 3 days on the road, get repaired, and back (and thank you COVID for making my life even harder),
I've activated RAM disk in hope it does some difference.
All the internet traffic goes encapsulated, so the firewall is fine tunned at the HQ, there are no changes in the sites.
I've tried do make read only in fstab, but it crashes.Thanks in advance.
-
ZFS would be better file system choice if you have issues with sudden power loss.
This is not the default choice, but can be setup on install. It does cost you more in ram then ufs, but should be way more reliable for sudden power loss - no matter the reason.
ZFS has been available since 2.4, which I would sure hope your running current version of pfsense.
Also while it doesn't help if pfsense is unplugged from the ups for the coffee machine. I would make sure the ups your using (if at all possible) is setup in pfsense to shutdown on loss of AC power so proper shutdown is done vs just sucking the ups dry and then having a sudden loss of power to pfsense. There is a package (nut) you can install that should work with most modern upses.
Also prob a good idea to have install media on site, that maybe you could walk someone on site through doing a reinstall. Possible setup out of band access, that could give you shell access to the pfsense box. This could be done with an old laptop, or raspberry pi and the like.. Which could even hotspot off someones phone to get you remote access in a pinch, etc.
And I hear you - many of us are spoiled with good power, and easy access. And worse case smarts hands we can hire in the location to perform recovery operations, who can be on site in 4 hour or less, etc. etc. Trying to run a stable network in isolated parts of the world can be difficult for sure.
-
@tiago-m-azevedo What machines are your routers running on?
I recently posted this question, wondering if the $40 Protectli 12V battery (designed to go between the AC adapter and the router) would work for netgate router devices. Maybe not a fool-proof solution, but perhaps a belt to add to your suspenders.
@johnpoz Thanks for the tip re: ZFS! Another reason why I wish the netgate routers came with more RAM, or would allow me to add more RAM ..
-
Its not like you need a TB of ram or anything.. Its just more ram hungry than ufs is all..
-
@johnpoz thx. I'm small potatoes - just wishing I could get an extra Gig for a comfort margin on an SG-1100
-
Well, if you really need 24/7 power protection, in addition to a UPS you need a backup generator of some sort.
Years ago, when I worked in a major telecom office, most things ran on -48V DC and huge battery banks (about 7,000A total load) with standby diesels. For AC powered devices we had what was called "no break" power, where the incoming AC turned an 8 ton flywheel and alternator. If the power dropped, a clutch would kick in, connecting a diesel to pick up the load. This would happen in a few seconds, with the flywheel carrying the load in the mean time, while also starting the diesel. This may be a bit much for your needs, but that's the sort of thinking you have to do. That is short and long term backup. A UPS handles the short term, but you also need an alternate power source for long term. Also, as Fukushima and Hurricane Sandy showed, you may also want to protect your system from floods.
-
I find the ZFS option an excellent idea to test, I'll read more on the subject.
The UPS I have is exactly that of -48Vdc, and really, it won't drain, I believe it would feed the system for a week. The only problems that we had, were human related.
About the remote access by a middle man... that's topic for hours of debate. Most of these sites are in places were most people didn't knew a computer 5 years ago, others, 5 days ago. The network is the way that we support and make them evolve. Also I install telemetry systems, biometric, VoIP and CCTV. So even employes without user (low user) skills, can be integrated.
About the read-only question as per the old Pfsense embedded?
-
@JKnott We have generators, nowadays they have a "computer" controller that you just define the low limit (46 Vdc) and it kicks in. Comap controllers, from Czech Republic, do amazing things and integration!
-
With generators make sure your transfer switch holds in the "neutral" position for a small period of time. Mine all stay there for five seconds. You do not want the power to come in "out of phase". That alone causes things like tripped circuit breakers and smoked power supplies.
-
@chpalmer Thanks for the heads up. Fortunately our main business is supplying energy to telecom and banking.
-
One other thing to consider is computers that support multiple power supplies. I used to have an IBM Netfinity server, which had 3 power supplies, any 2 of which were sufficient. You'd then have multiple power sources. This sort of thing was also common in telecom, whether AC or DC powered.
-
I have 4 gig of ram, and am using ZFS, the memory usage averages lower than 50%, and that is even with ramdisks.
Firewall's are not heavy i/o, so the memory usage should be fine, but if it is a concern you can cap the memory usage for ZFS in a tunable.
-
@chrcoluk Thank for your tips, I have resources, RAM is not a problem.
About the ZFS, is it possible to make 2 partitions and use them as mirror?
I inclined in this moment on a 2 SSD disk, mirros, ZFS. I think it will attend the power failure needs.When you have ZFS and RAM disk, can you use a USB/SD memory? Does it read/write a lot (with RAM disk)?
Cheers,
-
If you have 2 disks, then the best is to setup a zfs pool in mirror configuration. You can do that in the installer.
I dont know what you mean by USB/sd memory.
-
@chrcoluk I've decided in that way.
ZFS with 2 SSD in mirror, and RAM disk.
Hope it will run ok.
