26.07 considerable more writing to disk than 26.03.1

keyser

I upgraded two firewalls to 26.07 yesterday - no issues during the upgrade :-)
My zabbix monitoring has revealed that it brings a substantial added write load to disk/SSD for the EXACT same config.
I cannot see any log entries going bezerk, so what could this be caused by?
I realize it could be some of the packages I’m running (pfBlockerNG, Wireguard, NUT), but they too does not seem to be logging more that before.

EDIT: It’s still a low load as the boxes have been configured to almost not touch the disk (to save the eMMC from wear-out).

johnpoz

@keyser what is the Y axis, and the legend for green and yellow lines? Curious myself - I have not updated to 26.07 yet.. Got a business trip next week, and don't want to risk any issues while away that leaves the wife without internet. It can wait til I get back ;)

I should prob let it do a baseline while gone.

luckman212

@keyser off topic but, are you using Telegraf to ship stats to Grafana to make those charts?

keyser

@johnpoz said in 26.07 considerable more writing to disk than 26.03.1:

@keyser what is the Y axis, and the legend for green and yellow lines? Curious myself - I have not updated to 26.07 yet.. Got a business trip next week, and don't want to risk any issues while away that leaves the wife without internet. It can wait til I get back ;)

I should prob let it do a baseline while gone.

The Y-axis is Kbytes/sec so like I said it's still "nothing" - just something to keep an eye on if the same thing happens on "write intensive" systems

keyser

@luckman212 said in 26.07 considerable more writing to disk than 26.03.1:

@keyser off topic but, are you using Telegraf to ship stats to Grafana to make those charts?

No, I query the box via SNMP (using built-in SNMP in Services), from my Zabbix Raspberry Pi Server. Using Grafana on that same server to make the beautiful dashboards, even though Zabbix can do the same out of the box :-)

marcosm

What services are enabled? Also if using Nexus, any additional services in use there?

keyser

@marcosm pfBlocker, Wireguard and NUT, but like I said they don’t seem to log more than before.
Nexus is enabled for the new UI, but none of the new services in Nexus is enabled.

marcosm

The dtrace tool can help track down the process and files resulting in the increased writes. I don't have any script examples but you can find some by searching around or asking an LLM.

dennypage

Interestingly, I am seeing something a little different. I have a significant increase in the number of write calls, but only a very slight increase in the number of actual bytes.

fw io ops.png

fw io bps.png

I do not have Nexus enabled on that system.

luckman212

@dennypage what's making those pretty graphs?

dennypage

@luckman212 said in 26.07 considerable more writing to disk than 26.03.1:

@dennypage what's making those pretty graphs?

The graphs are from LibreNMS (successor to Observium). The graphs are not quite as nice as Grafana (which I use a lot), but when you need to monitor a large number of systems it's a lot easier to set up and use.

Gertjan

I see ... well, have a look for yourself.
Since week 31, end of July, 2 weeks before I upgrade from 26.03.1 to 26.07.

As I use pfSense for a hotel, and there are a lot of guests connected (captive portal) I considered it as 'normal' and season related.

Btw : some small pfSense packages, and freeradiusd + pfBLockerng. I discovered last week that my freeradius was logging with 'debug' settings. I switches it back to 'normal'.

dennypage

@Gertjan Haven't seen Munin in a while!

Gertjan

@dennypage
Click on the image above to see the entire collection.

Munin : I know there is newer stuff available now.
But I know munin from the inside out.
I know how to edit/write data collectors / plugins /whatever they are called.
It's a simple tool to create RDD data file.
Munin can't be installed as a pfSense package any more, so the FreeBSD version has to be used, and you have to navigate wisely when accepting the dependencies. But again, it's simple as it's just a bunch of mostly python stuff.