Another Netgate with storage failure, 6 in total so far

stephenw10

Well you can lose /var and /tmp entirely (if they are ramdisks) and the system will still boot back and replace them.

fireodo

@stephenw10 said in Another Netgate with storage failure, 6 in total so far:

Well you can lose /var and /tmp entirely (if they are ramdisks) and the system will still boot back and replace them.

So, you agree with me, letting /tmp AND /var "out" of sync would do no harm?

Fact is - since 2021 I had reboots but no problems with these settings. Power failure I have not because my pfsense hangs on a UPS.

JonathanLee

Side note: I keep thinking about a smart Optane NVMe
drive that leverages ram for the constant rewrites and a capacitor/battery for power fail to write to SSD. Some Symantec and Veritas systems as well as TinTri systems have such a capacitor. 4200 systems need the NVMe drive. the needs of zfs are burning up onboard components

Put specific items in the ram of the nvme that burn up drives with zfs in this type of environment.

dennypage

@stephenw10 said in Another Netgate with storage failure, 6 in total so far:

Well you can lose /var and /tmp entirely (if they are ramdisks) and the system will still boot back and replace them.

@fireodo said in Another Netgate with storage failure, 6 in total so far:

So, you agree with me, letting /tmp AND /var "out" of sync would do no harm?

That's not the same thing. Use of ramdisks is a configuration driven thing, and the system knows that /var must be recreated at boot. There is a lot of data stored in /var, and if it's corrupted you could encounter any number of problems.

IMO, you're fine letting /tmp go to pot, but not /var.

stephenw10

Yup when ramdisks are enabled it triggers a bunch of things at boot and shutdown. But it might be possible....

mer

@JonathanLee said in Another Netgate with storage failure, 6 in total so far:

needs of zfs are burning up onboard components

I'm not trying to say people aren't having problems, I'm trying to understand "why" they are.

What needs of ZFS are causing this?

Burning up these kinds of components are related to the number of erase cycles, which circles back around to writes.
What parts of pfSense are doing a lot of writes? I can see persistent logging, maybe "check for updates", maybe atime property enabled? Updating persistant store for block lists?

The way I look at it, once the system is configured and running, it should be doing mostly read from the filesystem and running from memory.

dennypage

@stephenw10 said in Another Netgate with storage failure, 6 in total so far:

Be good if you could set it for /tmp only.....

It would be really good if you could simply do ramdisk for /tmp only. No need to save/restore.

SteveITS

@mer said in Another Netgate with storage failure, 6 in total so far:

What parts of pfSense are doing a lot of writes?

Some packages (https://www.netgate.com/supported-pfsense-plus-packages), logging of default block rules, IGMP block logging, logging set in packages, updates of block lists and country lists, nginx access log (dashboard widgets), and similar.

JonathanLee

@SteveITS what files can use a linker file to direct to a usb drive? That model can use a usb drive right ?

w0w

@SteveITS
Hmm, so enabling compression "on the fly" in case of logs can significantly reduce writes, yes?

zfs set compression=lz4 pfsense/var/log

Some log compression options can be enabled via the GUI, but I don’t think they use "on-the-fly" compression.

dennypage

@SteveITS said in Another Netgate with storage failure, 6 in total so far:

logging of default block rules, IGMP block logging

These two can be quite voluminous, but are easy to address:

Add a rule on Local to pass IPv4+IPv6 IGMP with IP options set. I think this should actually be a default rule in pfSense.
Disable logging of packets blocked by the default rule in Firewall Logs. There are often thousands of these every day, and the individual log entries really aren't of much value.

SteveITS

@JonathanLee Oh I have no idea. :)

@w0w said in Another Netgate with storage failure, 6 in total so far:

Some log compression options can be enabled via the GUI, but I don’t think they use "on-the-fly" compression.

Yes it does: https://docs.netgate.com/pfsense/en/latest/monitoring/logs/index.html#log-format

"ZFS already compresses this data"

@dennypage said in Another Netgate with storage failure, 6 in total so far:

easy to address

Yep, mentioned above. In a link maybe, it's been a long thread. We actually don't pass the IGMP, since it's "supposed" to be blocked (always has been) we add a block rule that is set to not log. Otherwise IGMP is logged even if the logging for the default block rule is off.

dennypage

@SteveITS said in Another Netgate with storage failure, 6 in total so far:

We actually don't pass the IGMP, since it's "supposed" to be blocked (always has been) we add a block rule that is set to not log.

I would not say IGMP is supposed to be blocked, and it's rather inefficient to do. Multicast flooding is not desirable, even if it's only mDNS.

Of course, if IGMP is completely disabled in your switches, it doesn't matter. But if it is disabled in your switches, you won't see the IGMP messages to begin with.

SteveITS

@dennypage rephrasing, pfSense blocks them by default.
https://docs.netgate.com/pfsense/en/latest/troubleshooting/log-filter-blocked.html#packets-with-ip-options

dennypage

@SteveITS said in Another Netgate with storage failure, 6 in total so far:

rephrasing, pfSense blocks them by default.

Yea, that's why I called out that pfSense should add a default pass rule for IGMP.

Blocking packets with IP options that are to be forwarded is a good default, however IGMP isn't forwarded. Blocking IGMP by default makes little sense.

andrew_cb

@dennypage @SteveITS I had commented on redmine 15400 but since it was closed I guess that my message went unnoticed.

I have created a new redmine 16068 for adding options to disable logging of packets with IP options.