Firewall rules need reload after two LANs go up

cmcqueen

The specific symptom I first noticed was, IPv6 connectivity would sometimes go down after I turned on my work gear in the morning at home (working from home this year). The fix is, to reload the firewall rules.

I have my work gear connected to a separate LAN subnet from the rest of the house (this is on hardware with 4 NICs). The Ethernet connection for this 2nd LAN subnet connects to one switch which gets powered on/off with all my work gear.

But, I also have one other device connected to a 3rd LAN subnet, which I use with traffic shaping to test its handling of packet drops and longer delays (eg testing 100ms ping time).

What I suspect is happening is, when I power everything up together, pfSense sees two LAN subnets coming link-up nearly simultaneously. In the logs I see it is reloading firewall rules twice. So I wonder if pfSense trying to reload firewall rules twice in quick succession is messing up the rules. For some reason, the specific failure I notice is that IPv6 connectivity breaks. Anyway, manually reloading the filter rules fixes it.

That's what I think is going on, but perhaps I'm misdiagnosing it. I'm happy to do any tests or collect any logs that might help in further diagnosing it.

cmcqueen

Here is the system log from such an occasion:

Jan 25 09:47:03 kernel igb1: link state changed to UP
Jan 25 09:47:03 check_reload_status Linkup starting igb1
Jan 25 09:47:04 php-fpm 68492 /rc.linkup: DEVD Ethernet attached event for opt2
Jan 25 09:47:04 php-fpm 68492 /rc.linkup: HOTPLUG: Configuring interface opt2
Jan 25 09:47:04 check_reload_status Restarting ipsec tunnels
Jan 25 09:47:06 kernel igb3: link state changed to UP
Jan 25 09:47:06 check_reload_status Linkup starting igb3
Jan 25 09:47:07 dhcpleases /etc/hosts changed size from original!
Jan 25 09:47:07 dhcpleases Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process.
Jan 25 09:47:07 php-fpm 4500 /rc.linkup: DEVD Ethernet attached event for opt1
Jan 25 09:47:07 php-fpm 4500 /rc.linkup: HOTPLUG: Configuring interface opt1
Jan 25 09:47:07 dhcpleases Could not deliver signal HUP to process because its pidfile (/var/run/unbound.pid) does not exist, No such process.
Jan 25 09:47:07 check_reload_status Restarting ipsec tunnels
Jan 25 09:47:07 dhcpleases kqueue error: unknown
Jan 25 09:47:08 dhcpleases /etc/hosts changed size from original!
Jan 25 09:47:08 php-fpm 4500 /rc.linkup: The command '/usr/local/sbin/unbound -c /var/unbound/unbound.conf' returned exit code '1', the output was '[1611528428] unbound[34415:0] error: bind: address already in use [1611528428] unbound[34415:0] fatal error: could not open ports'
Jan 25 09:47:08 dhcpleases kqueue error: unknown
Jan 25 09:47:09 check_reload_status updating dyndns opt2
Jan 25 09:47:09 check_reload_status Reloading filter
Jan 25 09:47:11 check_reload_status updating dyndns opt1
Jan 25 09:47:11 check_reload_status Reloading filter
Jan 25 09:47:23 kernel cannot forward src fe80:3::96de:80ff:fe18:531b, dst 2403:xxxx:xxxx:xxxx:20e:c4ff:xxxx:xxxx, nxt 58, rcvif igb2, outif igb0
Jan 25 09:55:26 php-fpm 345 /index.php: Session timed out for user 'admin' from: 172.17.20.32 (Local Database)
Jan 25 09:55:28 php-fpm 345 /index.php: Successful login for user 'admin' from: 172.17.20.32 (Local Database)

stephenw10

It shouldn't really make much difference.

You can see it trying to restart Unbound faster than it can start though.

What symptoms do you see after this happens? What's not working that makes you reload the ruleset?

Steve

cmcqueen

@stephenw10 The symptom is that IPv6 data doesn't get through from LAN clients. Eg IPv6 pings, connections to IPv6 addresses of web sites—no data gets through. But, IPv6 ping works fine from the pfSense diagnostic page.

stephenw10

pfSense itself can both ping out to external sites and to internal hosts?

There mist be something missing, a route or IP somewhere. Or potentially an actual pass rule. You can check the rules that are actually loaded using pfctl -sr

Steve

cmcqueen

@stephenw10 Another symptom I'm seeing is that a client on the 2nd LAN interface (opt1) sometimes can't access the pfSense router web interface.

Again, the solution is to manually reload the filter rules (by connecting to the web interface via the 1st LAN interface).

stephenw10

Try looking at the ruleset in /tmp/rules.debug when it's failed.

Then do a filter reload so it's working and check again.

Compare the files, what changes? (if anything)

Steve