Should I be using Unbound Python mode? Is it stable?

[NOCling]

I Reboot after the Ubound change and the iostat was after 8-9h overnight and 12-14h overday.

[keyser]

@nocling Yes, but that is going to be one BADLY skewed statistics because that includes all the writes from a boot, reinitialise logs and reconfiguration sequence. So that number will become a lot less over the coming days/weeks as the boot/initialize proces starts to become less significant in the average calculation.

[keyser]

@keyser said in Should I be using Unbound Python mode? Is it stable?:

You can’t use “iostat -x” as a measure unless you have rebooted your pfsense and waited at least an hour or two. It displays statistics since last boot, and LOADS of factors impact that.
You need to do some monitoring while running with commands like: “iostat -d 5 6” and “top -m io”

Sorry - that should have said “before you waited at least a week or two”

[keyser]

@keyser Hmm, seems the diskfilling issue is not resolved by disabling DNS reply logging after all.

It just fills very slowly now as I’m not really logging anything.

I’ll do a more proper investigation now to see if I can find where the lost diskspace really goes. None of my log files are “oversized” now, and no files have aqquired the currently lost 100Mb space pr. Week in increased filesize.

So there is an issue with some kind of leakage to the filesystem that is not commited as visible/cleared until pfBlockerNG is restarted.

[keyser]

@keyser

A good chunk of the daily lost space happens when the pfBlockerNG CRON job runs.

[mcury]

Mine seems to be fine.. SG-3100 21.05.1 UFS with DNS Reply Logging disabled in DNSBL.. Python mode btw..

[21.05.1-RELEASE][xxx]/root: iostat -x
                        extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
flash/sp       0       0      0.0      0.0     7     0     0     7    0   0
mmcsd0         0       2      0.4     50.1     2     3     0     3    0   0
mmcsd0bo       0       0      0.0      0.0     0     0     0     0    0   0
mmcsd0bo       0       0      0.0      0.0     0     0     0     0    0   0
md0            0       0      0.0      0.0     0     0     0     0    0   0
[21.05.1-RELEASE][xxx]/root: uptime
 5:32PM  up 5 days,  4:41, 2 users, load averages: 0.24, 0.42, 0.41

[vjizzle]

I did some tests on a system with 60+ days uptime and setup unbound with python mode. This is what I see when running the iostat command:

extended device statistics
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b
md0            0       0      0.0      0.0     0     0     0     0    0   0
ada0           0      22      0.0    330.1     0     0     1     4    0   1
pass0          0       0      0.0      0.0     0     0     0     0    0   0

Do I need to worry?

[keyser]

@vjizzle Depends om how large your Sata SSD is.

It’s doing 330Kb writes pr. Second on average. That’s:

0,33Mb * 60sec * 60min * 24hours * 365 days = 10.4TB/year

If your SSD is 128Gb or bigger, it’s probably not an issue. But if it’s 32Gb Id think about it. 16Gb or smaller and I’d be very worried.

[vjizzle]

@keyser It is 120GB m2 ssd so I am safe I hope. I saw on Reddit that bbcan177 released a patch for the growing files on disk. I hope he can tackle the frequent writes as well.

[mcury]

@keyser said in Should I be using Unbound Python mode? Is it stable?:

0,33Mb * 60sec * 60min * 24hours * 365 days = 10.4TB/year

So, based on that formula, I would be using around:

0,05Mb * 60sec * 60min * 24hours * 365 days = 1.58TB/year

With these writes, do you think I should be worried with a 8 GB eMMC Flash ?

[keyser]

@mcury said in Should I be using Unbound Python mode? Is it stable?:

So, based on that formula, I would be using around:

0,05Mb * 60sec * 60min * 24hours * 365 days = 1.58TB/year

With these writes, do you think I should be worried with a 8 GB eMMC Flash ?

I wouldn’t. The 8Gb eMMC is rated at about 11Tb write endurance as far as I have been able to find out.

So you should be good for about 8-9 years - likely longer than your appliance will be in service anyways.

[mcury]

after the last update from pfblockerng, stats are worse than before, at least for me..

top -m io
unbound showing 100%

iostat -x
                        extended device statistics  
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b  
flash/sp       0       0      0.0      0.0     7     0     0     7    0   0 
mmcsd0         0      13     11.8    193.6     1     1     0     1    0   1 
mmcsd0bo       0       0      0.0      0.0     0     0     0     0    0   0 
mmcsd0bo       0       0      0.0      0.0     0     0     0     0    0   0 
md0            0       0      0.0      0.0     0     0     0     0    0   0 

uptime 
11:49AM  up  2:21, 2 users, load averages: 0.30, 0.34, 0.31

Edit: As soon as I changed from python mode to unbound mode, unbound disappeared from top -m io

[keyser]

@mcury said in Should I be using Unbound Python mode? Is it stable?:

after the last update from pfblockerng, stats are worse than before, at least for me..

top -m io
unbound showing 100%

iostat -x
                        extended device statistics  
device       r/s     w/s     kr/s     kw/s  ms/r  ms/w  ms/o  ms/t qlen  %b  
flash/sp       0       0      0.0      0.0     7     0     0     7    0   0 
mmcsd0         0      13     11.8    193.6     1     1     0     1    0   1 
mmcsd0bo       0       0      0.0      0.0     0     0     0     0    0   0 
mmcsd0bo       0       0      0.0      0.0     0     0     0     0    0   0 
md0            0       0      0.0      0.0     0     0     0     0    0   0 

uptime 
11:49AM  up  2:21, 2 users, load averages: 0.30, 0.34, 0.31

Edit: As soon as I changed from python mode to unbound mode, unbound disappeared from top -m io

You can’t use that command since you rebooted only 2:21 hours ago. Is shows average since last boot, and your boot skews those Numbers massively. You need to run realtime i intervals to see Whats going on: “iostat -d 5 6”
Six intervals of 5 seconds

[mcury]

@keyser said in Should I be using Unbound Python mode? Is it stable?:

iostat -d 5 6

unbound mode:
iostat -d 5 6
          flash/           mmcsd0           mmcsd0           mmcsd0 
  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s 
  3.23   0  0.00  14.84  14  0.20   3.23   0  0.00   3.23   0  0.00 
  0.00   0  0.00  20.50   2  0.04   0.00   0  0.00   0.00   0  0.00 
  0.00   0  0.00  13.79   1  0.02   0.00   0  0.00   0.00   0  0.00 
  0.00   0  0.00  25.88   3  0.07   0.00   0  0.00   0.00   0  0.00 
  0.00   0  0.00   0.50   0  0.00   0.00   0  0.00   0.00   0  0.00 
  0.00   0  0.00   4.00   0  0.00   0.00   0  0.00   0.00   0  0.00

changed to python mode again, ran a reload:

python mode:

iostat -d 5 6
          flash/           mmcsd0           mmcsd0           mmcsd0 
  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s 
  3.23   0  0.00  14.97  13  0.20   3.23   0  0.00   3.23   0  0.00 
  0.00   0  0.00  13.97   4  0.05   0.00   0  0.00   0.00   0  0.00 
  0.00   0  0.00   2.83   1  0.00   0.00   0  0.00   0.00   0  0.00 
  0.00   0  0.00  26.67   4  0.11   0.00   0  0.00   0.00   0  0.00 
  0.00   0  0.00  32.00   0  0.01   0.00   0  0.00   0.00   0  0.00 
  0.00   0  0.00  29.67   5  0.16   0.00   0  0.00   0.00   0  0.00

Edit: Something changed.. top -m io is not showing unbound anymore..
So it seems that just changing from python to unbound and then reverting back to python mode fixed it.. at least now it's not showing unbound in top -m io anymore..

[keyser]

@mcury Those numbers also look fine. There’s no massive writing anymore. So you are in the clear :-)