Unexpected alias behaviour - two ranges
-
It doesn't validate them separately either. Probably assumes "anytext" is a hostname even without dots, though the docs say "fully qualified domain names."
A quick solution (to break @Patch's "feature"
) is to reject spaces in the form field.@Patch The pfB aliases pull from a URL like https://127.0.0.1:443/pfblockerng/pfblockerng.php?pfb=pfB_PRI1_6_v6.
-
Yup. However I also can't replicate the original issue where an empty line is created. Adding multiple subnets there creates a valid alias with all subnets expanded.
So I suspect there was a rogue character in @Patch's imported list as you suggested.
As such I'm inclined to do nothing here. Anything we did do would likely break the workflow for anyone using the undocumented behaviour.
-
Now that I know about this “feature” I can see where it’s going to save me some time. :)
-
Sorry about the delay in responding.
pfsense was behaving very strangely which was complicated by- testing on my home active router
- pfsense configuration evolved over many years and includes a 3 packages (eg pfBlockerNG)
- frequent restarts required
- fault detection I suspect was using failure to include specified entries in an alias -> hybrid NAT rule failed -> after firewall restart failure to register of 1 of 4 VoIP suppliers
Solution, simplify the test set up
- Create a new VM on another appliance, with at least 2 NIC (actually 3 passed through)
- Clean install pfsense v2.8.1 via installer v1.1. Allow DHCP over ride of DNS & network time.
- Enable Kea, enable pfsense GUI from Wan (wan pass rule, un-check Block private network and loopback addresses)
- Slowly rebuild aliases of interest, regularly saving configuration after each error free entry
- Error detection based on comparing the Alias definition shown in Firewall -> Aliases -> IP -> <Alias> -> edit. Compared to Diagnostics -> Tables -> <Alias>
The result of which is:
@stephenw10 said in Unexpected alias behaviour - two ranges:
I suspect there was a rogue character in @Patch's imported list
You are correct. The networks were separated by a double space. Sorry the forum removes double spaces so you could not copy from my post. Using a single space prevents creation of the blank host. But doing so does not fix the problem with creating the desired Alias.
I believe the task which creates the table from the alias definition is too resource limited and the performance optimisation some times means the desired table is never created.
For example
-
Create an Alias (network type). Add a couple of /24 networks and a couple of FQDN or /32 addresses. Save and apply changes. They show up in the Alias table as desired. Edit the Alias definition to delete the /24 networks and on of the /32 addresses. Save and apply the changes. The /32 address will be removed but the /24 networks will stay. Reboot pfsense and the /32 will also be removed leaving only the valid /32 entry
-
Create an Alias (host type). Add a FQDN and two /24 networks one of which includes the FQDN IPv4 address. Save and apply. Look at the filter reload screen, when complete look at the created table for the Alias. For me it shows about 470 entries initially. Manually triggering Status -> Filter Reload -> adds about a couple more entries to the table for the alias. Wait 5 minutes for the Aliases Hostnames Resolve Interval to time out -> Alias table shows all 512 entries (duplicate removed)
-
Delete all the /24 network entries from the above alias definition, save and apply. Look at the Alias table, in now contains no entries. The single table entry for the duplicate address was deleted on removal of the explicit IP address. Filter reload and waiting >5min makes no change. Restart pfsense and the entry is repopulated.
-
Repeat test 2. but use with a single line entry of both /24 networks in one line. For me about 150 entries appear initially in the table, a few more with each Filter reload, all on waiting 5 minutes
-
A similar difference is seen in loading the 50 Crazytel IP. Initially pasting the 50 IP in on line sorted in reverse order loaded 7 entries. Repeating the test after the VM had been running for several hours loaded 25 entries. Pasting them in sorted in ascending order may have resulted in a couple more being initially loaded. As before manually triggering a filter reload added a few more. Wait 5 minutes and they are all loaded. Manually entering each IP into a separate line in the Alias and all 50 are loaded initially.
-
Running similar tests on configuration with many other Aliases (and pfblocker) and less is done initially. The 5 minute alias reload appears not to have the resources / time to ever complete some aliases
Summary
- pasting multiple single space separated entries into as single line is very efficient for data entry but appears to reduce initial processing done on alias entry by about a halve to a third.
- pfsense spreads alias processing over multiple Aliases Hostnames Resolve Intervals if required. More processing is done on reboot. Less processing is done on Filter reload.
- More complex configurations slow the Alias processing (perhaps to a stand still at times)
- Alias processing optimisation maybe over zealous at times requiring pfsense reboot to resolve.
- a force alias full regeneration option would be nice during configuration updates. Or perhaps more time devoted to it on manual filter reload
-
The alias processing does to some cool things which I like such as:
- enter range IP xx.xx.xx.xx-yy.yy.yy.yy in Host or network aliases
- in Network alias paste xx.xx.xx.xx/yy -> sets mask without needing to manually select via menu
- Change Host alias to Network alias to enable adding networks to an existing alias (The reverse can be done however that expanded networks which are not automatically re-compressed if you change your mind).
- Alias can included an alias simplifying maintenance and documentation.
- Multiple entries can be added in a single line which pfsense automatically expands. Doing so reduces initial table processing which is not trivial to accelerate.
- Duplicate host entries are automatically removed. However there is some risk of no entry being left there if one of the entries is delete (or maybe also changed). Duplicate covered by a network range appear not to be removed (depending on the filter design this may have minimal performance impact and maintaining a single entry for these duplicates maybe difficult). Similarly a relatively long consecutive sequence of host IP appears not to be converted to a range (but maybe else where in actual filter design)
-
@Patch said in Unexpected alias behaviour - two ranges:
Using a single space prevents creation of the blank host. But doing so does not fix the problem with creating the desired Alias.
Yes it does. The rest is a super longwinded way of saying, 'my bad.'
-
@Patch said in Unexpected alias behaviour - two ranges:
5 minute alias reload appears not to have the resources / time to ever complete some aliases
Are you talking about FQDNs or the /24 subnets here? I thought the 5 minute timer was for resolving FQDNs via DNS. We have a scenario where some either stop resolving or are maybe never added to the table...hard to tell since 99% of the time they're not used. However per your description, some may overlap (laptop goes to an allowed public IP) so I'm wondering if one fails it is removing "both" IPs/entries?
-
Hmm OK I can replicate case 2 here. Digging....
Do you see the full alias set shown in the Resolver logs when you add it?
For me I see that and it does load all 512 entries after a delay.
-
@stephenw10
I was testing last night and I think I have found the case which actually started my searching.- create an alias type network containing two /24 network and a couple of. /32 I have been using FQDN here which I think triggers alias table creation.
- create an alias type host containing two expanded /24 network and a couple of single hosts
- create an alias containing the above two alias.
- I also created a firewall rule using the last alias but don’t think that’s essential
- for test repeatability I have been clearing the alias tables, saving the configuration, then restoring the configuration
For me alias table generation locks up completely at about 300 entries. I think it also blocks other alias calculations. Tested in v2.8.1 and v2.7.2
Yet to test if using 512 random rather than sequential IPv4 addresses prevents the lock upPs
I have not looked for or at the Resolver logs. I will look when I’m able to test further -
@Patch said in Unexpected alias behaviour - two ranges:
For me alias table generation locks up completely at about 300 entries.
That's until you run filter-reload?
And it definitely still does it without any FQDNs present? Because otherwise it looks like it hangs filterdns but that can't be the case with FQDNs.
-
@stephenw10 said in Unexpected alias behaviour - two ranges:
That's until you run filter-reload?
No.
Makes no difference to me. Similarly leaving it running for 24 hours makes no difference.After some experimentation I can lock up the system immediately if I
Create the following alias (I have been using random IPv4 addresses but only minor differences occur if sequential IP addresses are used. Creating the Aliases via the bulk import option also makes no difference.)- IP_set1 : host type, 50 IPv4 hosts (/32) and at least 1 FQDN
- IP_set2 : host type, 512 IPv4 hosts (/32) and at least 1 FQDN
- IP_set3 : host type, 50 IPv4 hosts (/32) and at least 1 FQDN
- Combined_IP : Host type consisting of the above 3 aliases (IP_set1 IP_set2 IP_set3)
Then
- Create a firewall rule which uses the alias Combined_IP
- Diagnostic -> Tables -> select each of the above aliases and "Empty table"
- Save the configuration
To test, Restore the above configuration. My results
- Diagnostic -> tables -> records: Combined_IP = 256, IP_set1=50, IP_set2=206, IP_set3=0
- Waiting longer makes no difference, Filter reload makes no difference.
- Create a new Alias with hosts forum.netgate.com & redmine.pfsense.org -> empty table only generated
Testing with pfsense v2.7.2 results in similar results
- Combined_IP = 256, IP_set1=50, IP_set2=156, IP_set3=50
It appears pfsense alias capacity is way less than 5000 entries if an alias contains other aliases.
Not sure if this helps localise the issue. The similar Combine_IP size across software versions is interesting but is higher if only two aliasses are combined. Processor load remains trivial.@SteveITS said in Unexpected alias behaviour - two ranges:
some may overlap (laptop goes to an allowed public IP) so I'm wondering if one fails it is removing "both" IPs/entries?
I'm yet to test this as I have been first focussing on why wide spread alias problems have been occurring.It is on my list of things to do as I have a white list alias (containing my home IP and laptops current IP). Losing home access when the laptop leaves home is not desirable but happened recently. I was thinking of using a host over ride to try and simulate this. But the fault could have been caused by something unrelated.
Edit
Added the requirement for each IP_set to include at least 1 FQDN -
@Patch If you run "killall filterdns" and Status>Filter Reload do the tables populate?
-
@SteveITS is killall filterdns run from the command line or GUI menu option?
You do realise the IP_set alias consists almost exclusively of actual IPv4 addresses. I used a spreadsheet random number generator to construct addresses in the format
222.<random 1-255>. .<random 1-255> .<random 1-255>
The leading number changed for different aliases
However I don’t think the actual IP addresses make any difference.The issue being the failure varies with number of hosts and number of aliases. The aliases now contain less than 2 FQDN in total now.
But will try later today
-
@Patch The command should run both places. It just ends the processes. I have run it in the GUI.
Yeah I'm aware of the difference, I'm just trying to connect dots. Yours may be a totally different issue than mine, but it started to sound similar.
-
Yes you should be able to run that in either place. Though I would run it on the real command line if possible in case it does something unexpected.
-
@SteveITS said in Unexpected alias behaviour - two ranges:
If you run "killall filterdns" and Status>Filter Reload do the tables populate?
No but I guess this is not the expected response
btw @stephenw10 what happens when you try to replicate the behaviour?
While I assume it makes no difference, I'm using a Proxmox VM with 2 GB ram (GUI shows 18% memory usage), a Host type processor (i5-1235U with 2 cores), Hard disk: 8GB SSD, Bios OVMS (UEFI), Machine q35.
-
@SteveITS @stephenw10
Oops.
My test description was wrong.
Each IP_set alias needs at least one FQDN for the fault to be shown.- Adding the FQDN results in the table for each IP_set alias being created / viewable
- Removing all FQDN results in the Combined_IP being rapidly calculated.
Above post edited to include this requirement https://forum.netgate.com/post/1229337
-
@Patch filterdns processes are left running to monitor for updates in hostnames for Aliases/IPsec/etc, one thread per hostname. So, maybe unrelated to my observed problem.
But I’d expect some if you had FQDNs to resolve…?
-
@stephenw10 said in Unexpected alias behaviour - two ranges:
Do you see the full alias set shown in the Resolver logs when you add it?
More than showing in the alias tables
-
I can't be sure all entries are shown as display is limited to 2000 entries
-
IP_set3 table is empty however the log shows the actual 50 IP addresses are added but duplicates of "Adding Action: pf table: IP_set3 host:" but I think all 50 appear.
-
Similarly "Adding Action: pf table: IP_set2 host: " shows some duplicates. Not all actual IP addresses appear in the 2000 log entires. I was not able to readily tell if all 512 appear at least once in Adding Action: pf table: IP_set2 host:
As I have not looked at these logs in the past, I'm not sure what is normal
-
-
I think I agree at this point in some of the most incoherent SQA-masquerading-as-troubleshooting I've ever witnessed that—
It's true. The ability to add IP addresses and/or IP ranges to "Host" type aliases should be removed completely (and vice versa) via validation. That this makes no sense whatsoever on its face notwithstanding, it clearly has more than mere potential to lead to all of the above confusion.
