Firewall Rule Counters Max Size?
-
Yes, I would check the output of
pfctl -vvsr. Like for example:[24.03-RELEASE][admin@5100.stevew.lan]/root: pfctl -vvsr | grep -A3 @103 @103 pass in quick on igb0 reply-to (igb0 172.21.16.1) inet all flags S/SA keep state (if-bound) label "USER_RULE: Pass all" label "id:1650668824" ridentifier 1650668824 [ Evaluations: 14386 Packets: 14285 Bytes: 3432719 States: 1 ] [ Inserted: uid 0 pid 0 State Creations: 1 ] [ Last Active Time: Thu May 23 18:31:41 2024 ] -
@GeorgePatches I've long thought it doesn't count up every packet...? For example, the initial connection vs the response.
For example we have an overnight rsync job from our data center to our office which is a minimum of probably 6-10 GB per day and the two routers involved are showing 6.8 GiB and 3.6 GiB for their respective floating rules (which lower priority). The latter matches the 3.6 here:
[ Evaluations: 17515763 Packets: 1246913812 Bytes: 3865990636 States: 0 ]
...and the rsync log for just today shows about 7 GB sent this morning.
Bandwidthd on one end shows 11 GB for "daily" and 308 GB for "weekly."
The last rule change per config history was 4 days ago so unless something reloaded the rules I wouldn't think the counter is reset.
Overall I've always used it more like a relative amount of traffic, like is the rule being hit.
-
@stephenw10 said in Firewall Rule Counters Max Size?:
Yes, I would check the output of pfctl -vvsr. Like for example:
Checked that, the webui and pfctl match at the very least.
@SteveITS said in Firewall Rule Counters Max Size?:
Overall I've always used it more like a relative amount of traffic, like is the rule being hit.
Yeah, but I'm off by a couple orders of magnitude at the moment. Everything is working as far as traffic passing, but this is odd to say the least.
-
@GeorgePatches Right but I thought, in my experience, the rule may match inbound and even if the web server response is larger to download the file, the rule counter doesn't count the outbound.
However I just tested on an existing floating rule for a file download server and if I download a Windows 11 ISO, that rule started at 1.88 MiB, matched the download (qOthersLow) and counted up OK for a couple GB so I dunno...
The 3100 issue was, the number would wrap around and go to -2 GB, counting up to zero.
@stephenw10 Is there a way to check when rules/counters were last reloaded/reset?
edit: ...because if I Filter/Reload it doesn't reset them all to zero.
-
Hmm, I'm not aware of any way to check the time directly.
Though pfInfo shows a value for 'Enabled time'.
So:
pfctl -si -
I'll double check my rules I guess and make sure I'm not doing something silly.
-
OK, things are still not adding up. I updated the rule yesterday afternoon so that it would be more inclusive of traffic, to make sure I'm counting the traffic in this rule and not getting caught in another rule. I know from the backup logs that we uploaded roughly 1TiB, but my rule is showing less than 4GiB. The CLI counters in pfctl agree with the webui, so is something weird in pf?
-
Going to filter through the output of pfctl and try to figure out if this traffic is hitting another rule and getting counted elsewhere. Even if this is wrapping at 2^32 this is just not making sense.
-
Hmm, in a test here I created a new rule to pass iperf traffic on port 5201 then ran an iperf test against it:
[24.03-RELEASE][admin@4200.stevew.lan]/root: pfctl -vvsr | grep -A3 @90 @90 pass in quick on igc3 reply-to (igc3 172.21.16.1) inet proto tcp from <WAN__NETWORK:1> to 172.21.16.11 port = 5201 flags S/SA keep state (if-bound) label "USER_RULE: Allow iperf" label "id:1716563048" ridentifier 1716563048 [ Evaluations: 28 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 0 State Creations: 0 ] [ Last Active Time: N/A ][ ID] Interval Transfer Bitrate [ 5] 0.00-60.00 sec 6.58 GBytes 941 Mbits/sec receiver[24.03-RELEASE][admin@4200.stevew.lan]/root: pfctl -vvsr | grep -A3 @90 @90 pass in quick on igc3 reply-to (igc3 172.21.16.1) inet proto tcp from <WAN__NETWORK:1> to 172.21.16.11 port = 5201 flags S/SA keep state (if-bound) label "USER_RULE: Allow iperf" label "id:1716563048" ridentifier 1716563048 [ Evaluations: 40 Packets: 9752720 Bytes: 7568082888 States: 2 ] [ Inserted: uid 0 pid 0 State Creations: 3 ] [ Last Active Time: Fri May 24 16:10:00 2024 ]So I'd say it's counting traffic in both directions.
However there is an issue here. Still digging....
-
OK, I just did a test upload of 2GiB and now the counter has gone from 3.79GiB before to 1.64GiB now. WTF
-
Yeah, there's definitely an issue. Just trying to define it before I open a bug.
-
@GeorgePatches said in Firewall Rule Counters Max Size?:
OK, I just did a test upload of 2GiB and now the counter has gone from 3.79GiB before to 1.64GiB now. WTF
We have some well over 4 GB (96) so that's not it. I would guess, something's resetting the counter.
Here's some more fun info, I found https://www.reddit.com/r/PFSENSE/comments/eo62f4/how_to_reset_bytes_in_states_on_firewall_rule/, from (I would guess) "our"/Netgate's Jim P which says Filter Reload should zero them. However if I run a reload it reduces some of the numbers but does not zero them out.
I wasn't paying close attention to which was 96 GB above but that's gone, however I have several over 5 GB still with zero states. I'm looking at our data center router so it should have traffic but not 5 GB in a few seconds.
-
Yup doesn't zero them here. Running
pfctl -F allto flush everything does reset them but it also removes all the rules etc until you reload them. So don't do that! -
-
if the goal is to zero all the rules, wouldn't pfctl -z be what you want?
-
Ha, yup. Not sure how I missed that, thanks!
-
If my rules are getting zeroed repeatedly, would this be logged anywhere I could check?
Another thought, could this have something to do with the fact that we're rolling our traffic logs over at a staggering rate? I think we roll all 7 50MB filter log files in 30 minutes.
-
