Unexpected alias behaviour - two ranges
-
For those who have missed what is actually trying to be addressed in this thread
@Patch said in Unexpected alias behaviour - two ranges:
I discovered this behaviour when setting up a port forward for a PBX. Unfortunately the behaviour was not immediately obvious.
@Patch said in Unexpected alias behaviour - two ranges:
fault detection I suspect was using failure to include specified entries in an alias -> hybrid NAT rule failed -> after firewall restart failure to register of 1 of 4 VoIP suppliers
Important features of the bug
-
the fault results in failure of pfsense packet filtering not just a display error in debugging tools
-
the error is only revealed when pfsense restarting not after editing and applying an alias change. So not a nice bug to have in a live system.
-
how it presents in my live system is too complex for anyone else to reproduce or Netgate to fix. As a result substitute test end points and a simplified bug reproduction have been searched for (a process which risk masking the bug root cause or miss appropriating blame).
About the testing
-
Lock up of alias table generation has been used as a substitute marker of packet filter failure of rules which use these aliases.
-
Increasing the entries in each set or increasing the number of sets combined changes the fault behaviour. At least 1 FQDN is required in each IP_set to trigger the error.
-
I have not observed an obvious bug effect in having many FQDN in a set but have not directly tested this. No clear difference between ISC DHCP or Kea DHCP. Doubling the VM ram does not make any difference. Entering the alias via import, manually 1 entry at a time, many host in 1 entry, or network expansion all make minimal difference. A double space between items entered in a host type alias is expanded to a blank entry (which can be manually deleted) but otherwise makes no difference I could detect.
-
Diagnostics -> Tables are useful when the system is working well. It's less clear during fault conditions or as a marker for the bug being investigated in this thread. Double entry in the DNS resolver logs may corresponds to entries missing from these tables. After the primary alias tables stop updating, other aliases table entries is also blocked.
-
If the alias tables are just a diagnostic aid, which are not used in actual filter creation, so as a result at times not representative. Then it would be useful to support more direct alias content display perhaps, through keactrl or directly displaying the database content used by Kea
To state the obvious
-
I don't like having a production system which stops working for reasons I don't understand so can not reliably avoid. I can configure my systems to keep hierarchical aliases small (combine less than 4 sets with <50 entries) and revert to a higher ram VM allocation, so can avoid triggering this bug in my live systems. Netgate and other users may be less happy to discover it themselves in the future, but I can't speak for them, and my debugging time to support them is finite.
-
The bug can be triggered by sequential or random sets of IP addresses. So blocking easy creation of sequential IP addresses is irrelevant to this bug.
-
Summarising many hours of testing results in information dense posts. While these post are not easy to read, doing the underlying testing is more painful. Useful testing results new understanding of system behaviour, reflected in thread history.
@stephenw10 said in Unexpected alias behaviour - two ranges:
that is going to hurt some users. And save some others. Potentially.
We are off topic but blocking entering IP ranges in an alias is a bad idea.
- It is sensible to preserve range definition where that optimises resultant filter performance and configuration clarity. As such when a host line is entered which contains a range best left as a range, pfsense could:
- Change the alias type to Network or
- Leave the alias type as Host but also retain that line(s) subnet prefix length (it appears when a host type alias is displayed pfsense initially displays all host with a subnet prefix length then hides it).
-
-
It does sound like a bad bug. A deny rule with a partially filled alias, for example.
I am curious, does it matter where the FQDN is, in your alias? Does it stop updating the alias after the FQDN, if it is listed first or last?
In my linked thread above it's a rarely used allow rule and I notice it only when I can't connect.
Diagnostics -> Tables are useful when the system is working well. It's less clear during fault conditions or as a marker for the bug
The tables are an output of what pf is holding in memory so they should always match. "pfctl -T show -t aliasname" will show the table's contents at a command line, if that helps.
Per this doc "An alias becomes a table once the firewall loads it into the ruleset."
Kea is the DHCP server...? (not clear how that's involved...)
Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
When upgrading, allow 10-15 minutes to reboot, or more depending on packages, CPU, and/or disk speed.
Upvote đź‘Ť helpful posts! -
@Patch said in Unexpected alias behaviour - two ranges:
At least 1 FQDN is required in each IP_set to trigger the error.
This is definitely something. It's regarding this exact functionality, from which anything I've attempted to offer this thread stems.
@Patch said in Unexpected alias behaviour - two ranges:
I can configure my systems to keep hierarchical aliases small
[Emphasis added.]
I believe what you mean to reference here (and I may be wrong but I think I follow what you're saying)—is more precisely technically referred to as nested aliases. Nested anything—unless prescribed as an absolutely necessary means to accomplish a very particular and limited end—makes anti-kludge warriors and veterans everywhere shudder at the mere idea.
Would you be willing to share more about your specific use case of nested aliases?
-
@SteveITS What, pray tell, is a "partially filled alias"?
-
@tinfoilmatt said in Unexpected alias behaviour - two ranges:
What, pray tell, is a "partially filled alias"?
OP says they are not filling completely:
@Patch said in Unexpected alias behaviour - two ranges:
IP_set1 : host type, 50 IPv4 hosts (/32) and at least 1 FQDN
IP_set2 : host type, 512 IPv4 hosts (/32) and at least 1 FQDN
IP_set3 : host type, 50 IPv4 hosts (/32) and at least 1 FQDNDiagnostic -> tables -> records: Combined_IP = 256, IP_set1=50, IP_set2=206, IP_set3=0
...should have
612615, has 256. Which seems like a suspiciously specific number, tbh.Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
When upgrading, allow 10-15 minutes to reboot, or more depending on packages, CPU, and/or disk speed.
Upvote đź‘Ť helpful posts! -
@SteveITS said in Unexpected alias behaviour - two ranges:
OP says they are not filling completely:
An alias either is, or is not. When alias creation fails it's either due to parser logic error and/or a user input error. I thought we'd establised the latter as an errant space.
-
@SteveITS Doing that math also, it would seem that there's an error—not necessarily a software bug—somewhere in there, sure.
Parsing of "IP_set1" has most likely invalidated the FQDN for some reason (is the conclusion requring the least number of assumpions at least). And now all bets are off with the remaining two, and then of course with the "Combined_IP" nested alias too.
If I was being paid to resolve this on behalf of a client, I would establish the end goal intent—and then blow everything out and start over.
-
A partially populate table is probably a better description here. That's what I saw when I hit it temporarily.
Entering an range of IP addresses as a single line in a host alias is a valid entry. For example:
192.168.1.26-192.168.1.58What's not expected is multiple subnets as a single line in a host alias like:
192.168.10.0/24 192.168.48.0/24. That's not a range.But it looks to be possible to hit this using only valid host aliases. There is a bug here IMO it just needs better defining to fix. Working on it....
-
@stephenw10 said in Unexpected alias behaviour - two ranges:
That's what I saw when I hit it temporarily.
Were/are these the relevant steps?
- Create an Alias (host type).
- Add a FQDN and two /24 networks one of which includes [one of] the FQDN IPv4 address.
- Save and apply.
- Look at the filter reload screen,
- When complete look at the created table for the Alias.
-
@tinfoilmatt There were steps above...https://forum.netgate.com/topic/199152/unexpected-alias-behaviour-two-ranges/26
As I understood it (if I followed) some additional key details were:
- all child_aliases to be put in the parent_alias needed 1 FQDN, to trigger this
- then restore the new configuration (or reboot?)
The restore of course reboots. As I understand the report, it is a latent problem until the reboot when the alias was no longer fully populated.
A general usage example (not tested here but used in my other thread), we have an alias that has aliases containing IPs of our clients as well as various dynamic DNS IPs. Obviously we don't want to set up the same rules for each so a nested alias makes sense.
-
@SteveITS This thread is lacking in coherent, reproducible steps which demonstrate anything. Not picking on you as you're not OP. But Stephen most recently reaffirms that, at one point, he was able to do—something. In this post he referred to it as "case 2".
That's what I'm wanting to try to recreate for myself.
-
@SteveITS said in Unexpected alias behaviour - two ranges:
I am curious, does it matter where the FQDN is, in your alias? Does it stop updating the alias after the FQDN, if it is listed first or last?
Starting from https://forum.netgate.com/post/1229337
In practice I have entered a FQDN then many actual IPv4 addresses. I have mostly used a fixed prefix such as 201 or 202 or 203 (using a different number for each IP_set alias consists almost) followed by random numbers (0-255). It is far easier to add sequential IPv4 addresses but I was unsure what optimisation pfsense does so avoided that.These IP_sets are then combined in Combined_IP alias (nested / hierarchical)
The bug is revealed on full alias rebuild. In testing I used a configuration restore to ensure repeatability and clear starting point. Restarting pfsense has trigger it in my active systems.
Using 3 IP_sets containing 50, 512, 50 IP addresses,
for me it happens every time within a 30 sec of pfsense starting up. Smaller IP_set sizes can fail less cleanly. Tested with a clean install pfsense v2.81 and v2.72.@SteveITS said in Unexpected alias behaviour - two ranges:
...should have 612 615, has 256. Which seems like a suspiciously specific number, tbh
I agree that's a suspicious number but if I use 2 IP_sets the number is larger and with other IP set sizes the Combined_IP varies slightly.
@stephenw10 said in Unexpected alias behaviour - two ranges:
A partially populate table is probably a better description here.
That's a reasonable term.
Looking at the Resolver logs the missing alias table entries appear to correspond to
said in Unexpected alias behaviour - two ranges:
IP_set3 table is empty however the log shows the actual 50 IP addresses are added but duplicates of "Adding Action: pf table: IP_set3 host:" but I think all 50 appear.
Similarly "Adding Action: pf table: IP_set2 host: " shows some duplicates. Not all actual IP addresses appear in the 2000 log entires. I was not able to readily tell if all 512 appear at least once in Adding Action: pf table: IP_set2 host:
-
@Patch said in Unexpected alias behaviour - two ranges:
These IP_sets are then combined in Combined_IP alias (nested / hierarchical)
-
If anybody can distill two posts above this one, I'm happy to test.
-
Seems like I've replicated it.
I created a VM with 2.8.1.
I used easyrule to allow access on WAN.
I bypassed the GUI setup wizard.I created 4 aliases. IPs were created in Excel, enter the first and drag down 255 cells, then copy/paste into the Import in pfSense to create the alias.
alias_50_1: Host(s) 10.1.1.1, 10.1.1.2, 10.1.1.3, 10.1.1.4, 10.1.1.5, 10.1.1.6, 10.1.1.7, 10.1.1.8, 10.1.1.9, 10.1.1.10… (through .50) alias_50_2: Host(s) 10.2.2.1, 10.2.2.2, 10.2.2.3, 10.2.2.4, 10.2.2.5, 10.2.2.6, 10.2.2.7, 10.2.2.8, 10.2.2.9, 10.2.2.10… (through .50) alias_512: Host(s) 10.0.0.1, 10.0.0.2, 10.0.0.3, 10.0.0.4, 10.0.0.5, 10.0.0.6, 10.0.0.7, 10.0.0.8, 10.0.0.9, 10.0.0.10… (through .255 and 10.10.0.1-.255)Each of the first 3 ended with a hostname, e.g. pfsense.org.
alias_all: Host(s) alias_512, alias_50_1, alias_50_2If I click Apply and look at Diag > Tables, alias_512 shows:
Date of last update of table is unknown. 172 records....and ends at 10.0.0.170 but has the hostname for alias_512:
forum.netgate.com. 3 IN A 208.123.73.77Matching that, alias_all has 278 records.
The two "50" aliases do include the FQDN IPs (the third was netgate.com). So they seem correct. The alias_512 is missing most of its list. Which does show correctly if I edit the alias.
I did not need to reboot.
I'll make a follow on post momentarily. I can send or upload the config if desired but it seems easy enough.
-
If you run filter_reload does it fully populate?
Mixed mode aliases have been a problem in the past. I've long recommended not mixing IPs and FQDNs but I had thought those issues were resolved. Looks like we have a regression.
-
The first time I tried this I had an error in my list for alias_512. I accidentally scrolled two extra rows, leaving this in the import copy/paste:
10.10.0.256
10.10.0.257Obviously an error. Filterdns (DNS Resolver log) threw an error trying to resolve the "hostnames" since they are not IPs. The result was still the problem above, however, the results/numbers were slightly different and not off by two.
Deleting those and clicking Apply again reproduced the issue, hence my above post.
I'm not sure what that means but it seems odd that removing the two invalid IPs resulted in 1) several more (more than 2) additional IPs made it into the alias_512 table, and 2) the FQDN forum.netgate.com at the bottom of that list was resolved and its IPs also in that table. Even though 10.0.0.171-.255 and 10.10.0.1-.255 are not. Possibly an out of memory error and the "hostnames" take a bit more RAM than the last few IPs? I did try adding them back in again and the tables did not shrink as I expected.
Note also that as @Patch reported, the log shows the missing IPs being added:
Nov 7 01:34:37 filterdns 55828 Adding Action: pf table: alias_all host: 10.10.0.253 Nov 7 01:34:37 filterdns 55828 Adding Action: pf table: alias_all host: 10.10.0.254 Nov 7 01:34:37 filterdns 55828 Adding Action: pf table: alias_all host: 10.10.0.255 -
@stephenw10 said in Unexpected alias behaviour - two ranges:
If you run filter_reload does it fully populate?
no.
Initializing Creating aliases Creating gateway group item... Generating Limiter rules Generating NAT rules Creating 1:1 rules... Creating outbound NAT rules Creating automatic outbound rules Setting up TFTP helper Generating filter rules Creating default rules Pre-caching Default allow LAN to any rule... Creating filter rule Default allow LAN to any rule ... Creating filter rules Default allow LAN to any rule ... Setting up pass/block rules Setting up pass/block rules Default allow LAN to any rule Creating rule Default allow LAN to any rule Pre-caching Default allow LAN IPv6 to any rule... Creating filter rule Default allow LAN IPv6 to any rule ... Creating filter rules Default allow LAN IPv6 to any rule ... Setting up pass/block rules Setting up pass/block rules Default allow LAN IPv6 to any rule Creating rule Default allow LAN IPv6 to any rule Pre-caching Passed via EasyRule... Creating filter rule Passed via EasyRule ... Creating filter rules Passed via EasyRule ... Setting up pass/block rules Setting up pass/block rules Passed via EasyRule Creating rule Passed via EasyRule Creating IPsec rules... Generating ALTQ queues Loading filter rules Setting up logging information Setting up SCRUB information Processing down interface states Running plugins Done -
@stephenw10 said in Unexpected alias behaviour - two ranges:
not mixing IPs and FQDNs
If I remove forum.netgate.com from alias_512 then I get 618 records so I think it fully populates.
-
OK, weirder, I set the tunable for https://docs.netgate.com/pfsense/en/latest/troubleshooting/filterdns-thread-errors.html to 4096, just to see. I applied it, and did a Filter Reload.
alias_512 now contains only 12 records, scattered through the list plus the last 7+FQDN:
10.0.0.138 10.10.0.1 10.10.0.58 10.10.0.249 10.10.0.250 10.10.0.251 10.10.0.252 10.10.0.253 10.10.0.254 10.10.0.255 208.123.73.77 2610:160:11:11::6However alias_all still contains 618 entries. Which makes me think it either was created successfully, or not updated at all.
