Aliases stopped working

[steve.scotter]

Hi,

I have two pfsense firewalls in a high availability configuration using CARP failover. Both are running 2.6.0-RELEASE (amd64). All package are up to date. A list of the installed packages will be provided at the bottom of this post for completeness, but I don't think it's relevant to my issue.

On Monday evening I reboot the primary firewall (the one that holds CARP Master) as a diagnostic step when trying to resolve an odd issue regarding DNS resolution (DNS clients where unable to resolve DNS against using the CARP Master, but could via the CARP Backup; restarting DNS Resolver service didn't help, but rebooting the whole firewall did. The firewall had been up 462 days until then... probably when it was patched to 2.6.0).

After the reboot things seemed to be back to normal; DNS clients could resolve DNS via both CARP Master and CARP Backup.

However, today I noticed a server which communicates to a endpoint on the internet was reporting as offline from the internet console perspective. When I checked, the server was up and running, but was unable to communicate with it's internet based console.

A rule exists which uses an alias of type host(s) which contains a list of FQDN that server needs to access.

As you can see this table was setup in 2018 and has been working well for us until the reboot.

When I checked the firewall logs I could see the traffic is being blocked. This lead me to check the contents of the Table via Diagnostic > Tables and discovered that it was empty.

On closer inspection ALL tables of type Host(s) which have a FQDN are empty on the CARP Master, but correctly populated on the CARP Backup.

I've tried the following things to try and cajole the firewall to repopulate the alias tables.

• Edit an existing / Create a new firewall rule
• Edit an existing / Create a new alias of type Host(s)
• Status -> Filter reload
• Set Aliases Hostnames Resolve Interval to 60 within System > Advanced > Firewall & NAT

None of those actions have prompted the tables to be refresh on the CARP Master. The CARP Backup seems unaffected by those actions.

I've also just discovered that emptying (via Diagnostics > Tables > %tablename% and pressing Empty Table) on CARP Backup, that table does not seem to be repopulated either, suggesting a larger issue.

Any help gratefully received!

Regards

Steve

Installed packages

arpwatch	        0.2.0_6
Filer	                0.60.6_5
mailreport              3.6.3_3
nmap	                1.4.4_5
Notes	                0.2.9_3
openvpn-client-export	1.6_9
zabbix-agent6	        1.0.5
zabbix-proxy6	        1.0.5	

[steve.scotter]

I forgot to mention that DNS seems fine on the firewall.

If I try and resolve any of the FQDN's via Diagnostics > DNS Lookup or via SSH > 8. Shell and using dig or host commands they resolve fine.

[steve.scotter]

As a sanity check I have just created a alias of type Host(s) on a completely seperate and fully function pfsense firewall. It worked as expected and I saw the following in the logs

When I repeated my experiment on the CARP Master I got the following in the logs...

As you can see the /firewall_aliases_edit.php: Configuration Change: admin@a.b.c.d (Local Database): Edited a firewall alias. is missing from the CARP Master logs.

Very strange... chasing that down now!

[steve.scotter]

After several days of tinkering, looking at source code via https://github.com/pfsense/pfsense and comparing this broken instances with a working one I discovered that the broken instances didn't have /usr/local/sbin/filterdns running.

The broken instances

ps aux | grep filter
root    80996    0.3  0.0   12240   3432  -  Ss   Fri20      11:12.36 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
root    44845    0.0  0.0   11240   2516  4  S+   14:38       0:00.00 grep filter

A working instances elsewhere within my company

ps aux | grep filter
root    34312   0.0  0.0  23924   3472  -  Is   08:42        0:00.28 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 60 -c /var/etc/filterdns.conf -d 1
root    50337   0.0  0.0  12244   3432  -  Ss   14Mar23    293:36.93 /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
root    75276   0.0  0.0  11244   2516  0  S+   14:10        0:00.00 grep filter

Running filterdns manually didn't seem to generate any useful information (even when changing -d from 1 to 3) either on the console or the /var/log/resolver.log log file.

When I looked in /var/etc/filterdns.conf there were 2002 lines. 1778 of them where part of seven /24s which had been expanded out into individual IP addresses.

Changing that entry to have a type of Network(s), removing all the expanded individual IPs and adding the seven /24s back in has brought the number of lines in /var/etc/filterdns.conf to 53 and running /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1 now works (still silently, but hey its working).

Interestingly that alias with the seven /24's had been in there years. I suspect one or more another aliases had grown over some unknown threshold and that's why it broke.

Would be VERY helpful if filterdns would generate some logging when it fails to start!