Connections/states DROP when changing web configurator COLOR!!

netblues

@stephenw10 And my ssh sessions use tcp, but they get dropped.
(and clearing states always leads to web gui access lost for several seconds)

Couldn't find anything strange on the logs, but then I might not be logging what is needed to catch this. (Internal to external destinations traffic)
I just noticed that it only affects natted traffic, NOT routed.
I set up an experiment with iperf to a remote host. over typical ipv4 outbound nat (for Internet access)
Output follows
Its obvious when I changed colors in webgui
At the same time remote ssh sessions freeze (and disconnect with timeout_)
Routed sessions (and iperf) via pf remain intact.

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 35.8 MBytes 300 Mbits/sec 0 551 KBytes
[ 5] 1.00-2.00 sec 35.8 MBytes 300 Mbits/sec 0 543 KBytes
[ 5] 2.00-3.00 sec 35.9 MBytes 301 Mbits/sec 0 523 KBytes
[ 5] 3.00-4.00 sec 35.8 MBytes 300 Mbits/sec 0 520 KBytes
[ 5] 4.00-5.00 sec 35.8 MBytes 300 Mbits/sec 0 526 KBytes
[ 5] 5.00-6.00 sec 35.8 MBytes 300 Mbits/sec 0 540 KBytes
[ 5] 6.00-7.00 sec 35.8 MBytes 300 Mbits/sec 0 526 KBytes
[ 5] 7.00-8.00 sec 35.8 MBytes 300 Mbits/sec 0 523 KBytes
[ 5] 8.00-9.00 sec 35.8 MBytes 300 Mbits/sec 0 518 KBytes
[ 5] 9.00-10.00 sec 35.8 MBytes 300 Mbits/sec 0 523 KBytes
[ 5] 10.00-11.00 sec 35.8 MBytes 300 Mbits/sec 0 523 KBytes
[ 5] 11.00-12.00 sec 35.8 MBytes 300 Mbits/sec 0 526 KBytes
[ 5] 12.00-13.00 sec 35.9 MBytes 301 Mbits/sec 7 627 KBytes
[ 5] 13.00-14.00 sec 35.8 MBytes 300 Mbits/sec 0 579 KBytes
[ 5] 14.00-15.00 sec 35.8 MBytes 300 Mbits/sec 0 579 KBytes
[ 5] 15.00-16.00 sec 17.0 MBytes 143 Mbits/sec 2 1.41 KBytes
[ 5] 16.00-17.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 17.00-18.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 18.00-19.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 19.00-20.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 20.00-21.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 21.00-22.00 sec 0.00 Bytes 0.00 bits/sec 1 1.00 KBytes
[ 5] 22.00-23.00 sec 0.00 Bytes 0.00 bits/sec 0 1.00 KBytes
[ 5] 23.00-24.00 sec 0.00 Bytes 0.00 bits/sec 0 1.00 KBytes
[ 5] 24.00-25.00 sec 0.00 Bytes 0.00 bits/sec 0 1.00 KBytes
[ 5] 25.00-26.00 sec 0.00 Bytes 0.00 bits/sec 0 1.00 KBytes
[ 5] 26.00-26.81 sec 0.00 Bytes 0.00 bits/sec 0 1.00 KBytes

Happens always. !

Any ideas where should I look for logs more than welcome

stephenw10

When I make that change I see logged:

Jul 24 15:06:24 	check_reload_status 	687 	Syncing firewall
Jul 24 15:06:24 	php-fpm 	57549 	/system.php: Configuration Change: admin@172.21.16.8 (Local Database): System:
Jul 24 15:06:25 	php-fpm 	57549 	/system.php: Staging AutoConfigBackup encrypted configuration backup for deferred upload to https://acb.netgate.com
Jul 24 15:06:27 	root 	45627 	/etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
Jul 24 15:06:34 	php-fpm 	57549 	/system.php: NTPD is starting up.
Jul 24 15:06:34 	check_reload_status 	687 	Reloading filter

So you can see it reloads the firewall ruleset. For some reason. It's not immediately obvious what setting on that page might require it....

netblues

@stephenw10 And it restarts ntp too?

Doesn't look like a feature to me, but is it a bug?

And why reloading filter kills nat sessions to begin with.. I don't remember that being the case in the past (but I might be wrong)

stephenw10

It restarts ntpd because that page has the external ntp server settings on it. But nothing there looks like it would affect the firewall...

netblues

@stephenw10 Did some more fiddling since this made no sense at all.
Went back to 24.11, via boot environments reloaded the newer configuration, and then migrated again to 25.07rc

The problem went away. Almost impossible to reproduce again, strange as it is.

However, going back to 24.11 i saw that sticky connections did work.
Migrating to latest 25.07 rc, sticky connections are gone.

stephenw10

Like in a load-balanced gateway group?

netblues

@stephenw10 Exactly on that.

stephenw10

Hmm, I'm not aware of that. How are you testing? What are you seeing?

netblues

@stephenw10
I have two 1Gbit ftth lines(with 500Mbits upload)
One is dhcp, the other is pppoe
I have a gateway group called Loadb with the two gateways as Tier1 and another backup tier
as tier3 via 5g
On the lan side, I have a typical rule that sends the traffic there

This works fine. With stickiness enabled I get this

Which is expected, and doing a speedtest utilizes randomly one of the two lines

This is with stickiness enabled

Unchecking "use sticky connections", the source tracking tab disappears (as expected)
and a speedtest (multi connection by default) gets the aggregate as it should

Upgrading to 25.07 rc has no entries in the source tracking states and speedtests are always get the aggregate traffic.
This is a problem when visiting https sites , especially web banking and other session sensitive, since you get disconnected due to different source ip. It can be mitigated by excluding https traffic with policy routing, but then the issue persists on non standard ports.

This is with the new kernel mode pppoe driver but I have tried disabling it with no difference too.

stephenw10

Hmm, OK yup I'm seeing that. Digging....

Edit: Looks like this: https://redmine.pfsense.org/issues/16282

netblues

@stephenw10 Yes, this is exactly the case.
And while we are there, it would be nice to have stickiness being enabled per interface and not on as a whole. Even better if it could be done at the source ip level too.

stephenw10

Ok, that is fixed in 25.07 and will be in the next 2.8.1-beta.

The stickiness is per source IP. So each internal client will be stuck to a gateway/WAN.

netblues

@stephenw10 You mean 25.07 release? Since we are on rc.

Yes, I know that stickiness is per source ip, but there is no control over it.
its either all of them or none.

stephenw10

Mmm, so prevent source tracking for specific IPs or subnets?

I did wonder if sticky connections could be per gateway group. That seems like it should be possible. You could then use rules to route specific clients or subnets to a non-sticky group.