Another Netgate with storage failure, 6 in total so far

SteveITS

@mer said in Another Netgate with storage failure, 6 in total so far:

What parts of pfSense are doing a lot of writes?

Some packages (https://www.netgate.com/supported-pfsense-plus-packages), logging of default block rules, IGMP block logging, logging set in packages, updates of block lists and country lists, nginx access log (dashboard widgets), and similar.

JonathanLee

@SteveITS what files can use a linker file to direct to a usb drive? That model can use a usb drive right ?

w0w

@SteveITS
Hmm, so enabling compression "on the fly" in case of logs can significantly reduce writes, yes?

zfs set compression=lz4 pfsense/var/log

Some log compression options can be enabled via the GUI, but I don’t think they use "on-the-fly" compression.

dennypage

@SteveITS said in Another Netgate with storage failure, 6 in total so far:

logging of default block rules, IGMP block logging

These two can be quite voluminous, but are easy to address:

Add a rule on Local to pass IPv4+IPv6 IGMP with IP options set. I think this should actually be a default rule in pfSense.
Disable logging of packets blocked by the default rule in Firewall Logs. There are often thousands of these every day, and the individual log entries really aren't of much value.

SteveITS

@JonathanLee Oh I have no idea. :)

@w0w said in Another Netgate with storage failure, 6 in total so far:

Some log compression options can be enabled via the GUI, but I don’t think they use "on-the-fly" compression.

Yes it does: https://docs.netgate.com/pfsense/en/latest/monitoring/logs/index.html#log-format

"ZFS already compresses this data"

@dennypage said in Another Netgate with storage failure, 6 in total so far:

easy to address

Yep, mentioned above. In a link maybe, it's been a long thread. We actually don't pass the IGMP, since it's "supposed" to be blocked (always has been) we add a block rule that is set to not log. Otherwise IGMP is logged even if the logging for the default block rule is off.

dennypage

@SteveITS said in Another Netgate with storage failure, 6 in total so far:

We actually don't pass the IGMP, since it's "supposed" to be blocked (always has been) we add a block rule that is set to not log.

I would not say IGMP is supposed to be blocked, and it's rather inefficient to do. Multicast flooding is not desirable, even if it's only mDNS.

Of course, if IGMP is completely disabled in your switches, it doesn't matter. But if it is disabled in your switches, you won't see the IGMP messages to begin with.

SteveITS

@dennypage rephrasing, pfSense blocks them by default.
https://docs.netgate.com/pfsense/en/latest/troubleshooting/log-filter-blocked.html#packets-with-ip-options

dennypage

@SteveITS said in Another Netgate with storage failure, 6 in total so far:

rephrasing, pfSense blocks them by default.

Yea, that's why I called out that pfSense should add a default pass rule for IGMP.

Blocking packets with IP options that are to be forwarded is a good default, however IGMP isn't forwarded. Blocking IGMP by default makes little sense.

andrew_cb

@dennypage @SteveITS I had commented on redmine 15400 but since it was closed I guess that my message went unnoticed.

I have created a new redmine 16068 for adding options to disable logging of packets with IP options.

kingsleyadam

Thanks to @andrew_cb and others for bringing awareness to this. I had no idea my 6100 has limitations due to the eMMC. I went out of my way to buy a 6100 over my own router build because I just wanted to setup my router and forget about it. As someone who is fully remote the router is the last thing I can have fail.

I saw the threads on Reddit and did a quick check. Just over 1.5 years of having my 6100, it’s already at 70% wear.

I bought 2x16GB Intel Optane Drives which you can get for less than 5 euro a piece and managed to get them installed and set up in a mirror for redundancy (the drives are so cheap, I think it’s silly not too). I also 0’d out my eMMC drive to ensure it does’t cause any conflicts.

I’m not thrilled that I had to do this, I’m thankful there were M.2 ports on the 6100 that I could use. But my biggest take-away is that installing your own SSD is not “supported” and could void your warranty is unacceptable. I think there should be a well documented SSD upgrade for any device that has an available slot, it should not void the warranty, and most definitely shouldn’t be discouraged.

As a side note, I really wish the installer was offline. I was sweating bullets attempting to configure the WAN in the Installer with PPoE and VLAN tagging (don’t get me wrong, it was easy, but if it didn’t work I’d be SOL).