IP not covered in generated deny alias

[darcey]

I have a GeoIP based deny alias which I then refer to in an inverted nat/firewall allow rule. It consists of several countries, including RU_v4.

I recently saw an unexpected IP pass the rule. The IP does seem to be covered by the maxmind file. Comparing files in /var/db/pfblockerng, the IP concerned is covered by a CIDR in the original GeoIP country file. But not in the generated deny list:

[2.6.0-RELEASE][root@fw]/var/db/pfblockerng: grep '^45.146.1' ./original/RU_v4.orig
45.146.152.0/22
45.146.164.0/22
45.146.168.0/23
45.146.171.0/24
[2.6.0-RELEASE][root@fw]/var/db/pfblockerng: grep '^45.146.1' ./deny/RU_v4.txt
45.146.152.0/22
45.146.168.0/23
45.146.171.0/24

I have the CIDR aggregation option enabled.
What might be causing this to happen?

EDIT: I just tried turning off CIDR aggregation, deleting the generated ./deny/RU_v4.txt and forcing an update. No change in the relevant CIDRs.

[dma_pf]

@darcey I'm not sure this is the issue, but if your alias is set to Alias Deny try to switch it to Alias Native. Alias Deny's are subject to being processed, Alias Native's are not.

login-to-view

[darcey]

@dma_pf said

I'm not sure this is the issue, but if your alias is set to Alias Deny try to switch it to Alias Native. Alias Deny's are subject to being processed, Alias Native's are not.

I'm not sure about de-duplication but I have ip reputation feature disabled.

Thank you for the suggestion. I will try it.

It would be good to understand why the block concerned (45.146.164.0/22) did not make it to the generated deny file, either directly or as part of a larger aggregated subnet.

[darcey]

@dma_pf said in IP not covered in generated deny alias:

@darcey I'm not sure this is the issue, but if your alias is set to Alias Deny try to switch it to Alias Native. Alias Deny's are subject to being processed, Alias Native's are not.

That does indeed resolve the missing subnet. The CIDR appears in the list.
Whether I have wanted autogenerated firewall rules or aliases to use in independent fw rules, I have never used the native alias type in pfBlockerNG.
I may need to revisit forum posts to fully understand what's going on there!

[dma_pf]

@darcey I'm glad my hunch paid off!

It would be good to understand why the block concerned (45.146.164.0/22) did not make it to the generated deny file, either directly or as part of a larger aggregated subnet.

My gut sense is that the 45.146.164.0/22 was in another list and when the de-duplication was done it was removed from your original RU_v4.txt list.

Try running a grep on /var/db/pfblockerng/original/* for the 45.146.164.0/22 network to see if it was in 2 different feeds.

[SteveITS]

@dma_pf Except he's saying the packet was allowed.

If the dedupe crosses lists, that's a huge issue unless one is actually Denying both lists.

We have always used Alias Native, assuming a geo list wouldn't have duplicates within itself.

[darcey]

@dma_pf

The specific subnet only occurs in the one GeoIP feed. However single IPs, in other feeds (used in different aliases), fall within the range.

grep '^45.146.164' ./original/*
./original/Alienvault_v4.orig:45.146.164.110 # Malicious Host
./original/BDS_Ban_v4.orig:45.146.164.28
./original/BDS_Ban_v4.orig:45.146.164.234
./original/CINS_army_v4.orig:45.146.164.152
./original/CINS_army_v4.orig:45.146.164.163
./original/CINS_army_v4.orig:45.146.164.225
./original/CINS_army_v4.orig:45.146.164.234
./original/CINS_army_v4.orig:45.146.164.88
./original/ET_Block_v4.orig:45.146.164.0/24
./original/ISC_Block_v4.orig:45.146.164.0	45.146.164.255	24	3828	SELECTEL	RU	abuse@selectel.ru
./original/RU_v4.orig:45.146.164.0/22

For the alias concerned, the pfBlocker config lists 3 GeoIP countries. Only one of which (RU_v4) contains that range or IPs within that range. Not that I have thought it through or checked, but I envisage dedupe happens for each alias, taking in to account only those feeds specified for that alias. So, feeds used in other aliases, are not considered when de-duping/aggregating an alias?

[SteveITS]

@darcey said in IP not covered in generated deny alias:

feeds used in other aliases, are not considered when de-duping/aggregating an alias?

I would hope not. If those other lists are also set to deny traffic then in the big picture it doesn't matter...? It should be blocked anyway. However if someone uses Alias Deny and then maybe uses the CINS list only blocking certain ports (i.e. not the same kind of blocking), I could conceptually see that as a pitfall.

When you wrote "saw an unexpected IP pass the rule" what did you see specifically? Web server access_log entry?

[darcey]

@steveits said in IP not covered in generated deny alias:

@dma_pf Except he's saying the packet was allowed.

If the dedupe crosses lists, that's a huge issue unless one is actually Denying both lists.

A forward rule (with filter rule association) uses the inverse of this alias as src address. Intention is to allow all, other than those on the list.

We have always used Alias Native, assuming a geo list wouldn't have duplicates within itself.

I have previously only used the actual rule generation Permit, Deny options in pfBlockerNG. When I needed an alias for use in a forward rule I chose next obvious option. However what you say makes sense WRT GeoIP only aliases. I wonder if this is unintended consequences of using a Deny alias as as an inverse src address in an allow rule.

[darcey]

@steveits said in IP not covered in generated deny alias:

@darcey said in IP not covered in generated deny alias:

feeds used in other aliases, are not considered when de-duping/aggregating an alias?

I would hope not.

Yes that's my expectation also. I should not have put the question mark there.

If those other lists are also set to deny traffic then in the big picture it doesn't matter...? It should be blocked anyway. However if someone uses Alias Deny and then maybe uses the CINS list only blocking certain ports (i.e. not the same kind of blocking), I could conceptually see that as a pitfall.
When you wrote "saw an unexpected IP pass the rule" what did you see specifically? Web server access_log entry?

Yes, exactly. I saw some RU IPs in my the web server log and, lazily, put it down to inconsistencies between pfSense's copy of the country lists and the ELK copy of same. But then I checked for the presense of the IP in Maxmind lists in pfBlockers copy.

[dma_pf]

@darcey @SteveITS I found this post by BBcan177 from here:

login-to-view

If I'm reading this right, it would appear that all Alias feeds (regardless of which Alias group they are in) except Native feeds, are evaluated and deduped as a whole. If this is correct it certainly isn't the way I thought it worked.

[SteveITS]

@dma_pf said in IP not covered in generated deny alias:

isn't the way I thought it worked

Nor I!

[darcey]

@dma_pf
Thanks for this. That sheds some light for me too. So if I understand right, when you have both permit and deny alias groups and/or your rules target disparate port sets, it's probably safest to have pfblocker generate native aliases and define independent firewall rules to utilise them. That, or disable deduplication. Does a CIDR aggregation also operate across multiple alias groups?

[SteveITS]

@darcey said in IP not covered in generated deny alias:

if I understand right, when you have both permit and deny alias groups and/or your rules target disparate port sets, it's probably safest to have pfblocker generate native aliases

That's how I would interpret the above.

Offhand I would guess the assumption was made that any Alias Deny lists would be used to block all traffic, not just some ports for some lists, since that's what Deny Inbound/Outbound/Both would do.

I don't know about CIDR aggregation. I think that's off by default though.

[darcey]

Whilst I have a fix and potential explanation, dedupe is proving hard to figure out.

Deduplication is enabled in each case:

Taking the alias group and setting it as 'Alias Native', the CIDR (45.146.164.0/22) from ./original/RU_v4.orig appears unchanged in ./native/RU_v4.txt. As expected.

[2.6.0-RELEASE][root@fw]/var/db/pfblockerng: egrep -r '^45\.146\.16[0-9]\.[0-9]+\/' ./
./original/ET_Block_v4.orig:45.146.164.0/24
./original/RU_v4.orig:45.146.164.0/22
./original/RU_v4.orig:45.146.168.0/23
./native/RU_v4.txt:45.146.164.0/22
./native/RU_v4.txt:45.146.168.0/23
./deny/ET_Block_v4.txt:45.146.164.0/24
./mastercat:45.146.164.0/24

When I change the alias group to 'Alias Deny', I would expect the CIDR to be covered somewhere, though not necessarily in the associated ./deny/RU_v4.txt. However I don't seem to find the CIDR, or a superset of it, in any of the alias output:

[2.6.0-RELEASE][root@fw]/var/db/pfblockerng: egrep -r '^45\.146\.16[0-9]\.[0-9]+\/' ./
./original/ET_Block_v4.orig:45.146.164.0/24
./original/RU_v4.orig:45.146.164.0/22
./original/RU_v4.orig:45.146.168.0/23
./deny/ET_Block_v4.txt:45.146.164.0/24
./deny/RU_v4.txt:45.146.168.0/23
./mastercat:45.146.164.0/24
./mastercat:45.146.168.0/23

I wonder if the dedupe process is handling /22 mask as expected? Hopefully the regex's scope is sufficient to catch the relevant CIDRs.

[dma_pf]

I was re-reading these two items more carefully again this morning.

login-to-view

login-to-view

The top item explicitly indicates that deduping is done for Alias Deny groups. It does not say that for Alias Permit Groups.

The bottom item clearly states that, "When using De-duplication, all Aliases/List are acting as a whole". When I first read that I assumed it meant any Alias/List that was not Native would be deduped. But in looking at it again today I noticed that at the end of the paragraph it explicitly states that, "Match and Permit also do not use any De-duplication".

So it seems to me that the deduping is only happening across lists that are in an Alias Deny group.

@darcey said in IP not covered in generated deny alias:

So if I understand right, when you have both permit and deny alias groups and/or your rules target disparate port sets, it's probably safest to have pfblocker generate native aliases and define independent firewall rules to utilise them.

It seems to me at this point that it's definitely safest to use a Native Alias, especially in lieu of an Alias Deny. It might not really matter with Alias Permit rules unless they have some other processing we don't know about.

@darcey said in IP not covered in generated deny alias:

I wonder if the dedupe process is handling /22 mask as expected? Hopefully the regex's scope is sufficient to catch the relevant CIDRs.

Great work on trying to track what is actually happing! And thanks for the update.

Do you have CIDR Aggregation enabled? I'm asking because I'm wondering if the /22 network got gobbled up into a bigger CIDR. I know you were looking for any 45.146.16x.xxx/xx networks but maybe it is something bigger?

[darcey]

@dma_pf
I have multiple 'Deny Inbound' configurations with diverse destination port aliases (set under 'Advanced Inbound Firewall Rule Settings').
I also had 'Deny Alias' (which I have since changed to 'Alias Native'!). The latter Alias configs, of course, have no port config.
From the info and interpretation you've provided, I now believe it does not, by design, do what I intended when de-duplication is enabled.

Do you have CIDR Aggregation enabled? I'm asking because I'm wondering if the /22 network got gobbled up into a bigger CIDR. I know you were looking for any 45.146.16x.xxx/xx networks but maybe it is something bigger?

In the Deny Alias vs Native Alias above, CIDR aggregation has no effect (on the results of the regex grep at any rate). I checked with it on and off.

I'm going to simplify my rules whilst I have a grasp of pfblocker. It's got disconcerting how quickly I forget stuff these days!
Incidentally, I chose selective port blocking in order to manage firewall logging. I have some GPS trackers for which I allow a limited src range. However I was also interested in logging the dropped traffic that attempts to reach those ports. Same with a webserver. But I did't want to log blocked traffic for unopen ports.

[SteveITS]

Perhaps @BBcan177 will see this and confirm if this is a bug or working as intended.

[shoulders]

I am guessing the deduplication is just using the same mechanism as the normal deny rules.

This makes using the alias function pointless. Each alias should be de-duplicted individually otherwise you cannot tell what aliases is blocking what therefore you cannot create individual blocking rules and even if all rules were enables as recommended by BBcan177 your logging would not be accurate.