Connections/states DROP when changing web configurator COLOR!!

stephenw10

Hmm, unexpected indeed! The webgui uses TCP so a new state would just be opened if it was dropped.

Do you see hits in the firewall log when that happens?

netblues

@stephenw10 And my ssh sessions use tcp, but they get dropped.
(and clearing states always leads to web gui access lost for several seconds)

Couldn't find anything strange on the logs, but then I might not be logging what is needed to catch this. (Internal to external destinations traffic)
I just noticed that it only affects natted traffic, NOT routed.
I set up an experiment with iperf to a remote host. over typical ipv4 outbound nat (for Internet access)
Output follows
Its obvious when I changed colors in webgui
At the same time remote ssh sessions freeze (and disconnect with timeout_)
Routed sessions (and iperf) via pf remain intact.

[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 35.8 MBytes 300 Mbits/sec 0 551 KBytes
[ 5] 1.00-2.00 sec 35.8 MBytes 300 Mbits/sec 0 543 KBytes
[ 5] 2.00-3.00 sec 35.9 MBytes 301 Mbits/sec 0 523 KBytes
[ 5] 3.00-4.00 sec 35.8 MBytes 300 Mbits/sec 0 520 KBytes
[ 5] 4.00-5.00 sec 35.8 MBytes 300 Mbits/sec 0 526 KBytes
[ 5] 5.00-6.00 sec 35.8 MBytes 300 Mbits/sec 0 540 KBytes
[ 5] 6.00-7.00 sec 35.8 MBytes 300 Mbits/sec 0 526 KBytes
[ 5] 7.00-8.00 sec 35.8 MBytes 300 Mbits/sec 0 523 KBytes
[ 5] 8.00-9.00 sec 35.8 MBytes 300 Mbits/sec 0 518 KBytes
[ 5] 9.00-10.00 sec 35.8 MBytes 300 Mbits/sec 0 523 KBytes
[ 5] 10.00-11.00 sec 35.8 MBytes 300 Mbits/sec 0 523 KBytes
[ 5] 11.00-12.00 sec 35.8 MBytes 300 Mbits/sec 0 526 KBytes
[ 5] 12.00-13.00 sec 35.9 MBytes 301 Mbits/sec 7 627 KBytes
[ 5] 13.00-14.00 sec 35.8 MBytes 300 Mbits/sec 0 579 KBytes
[ 5] 14.00-15.00 sec 35.8 MBytes 300 Mbits/sec 0 579 KBytes
[ 5] 15.00-16.00 sec 17.0 MBytes 143 Mbits/sec 2 1.41 KBytes
[ 5] 16.00-17.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 17.00-18.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 18.00-19.00 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes
[ 5] 19.00-20.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 20.00-21.00 sec 0.00 Bytes 0.00 bits/sec 0 1.41 KBytes
[ 5] 21.00-22.00 sec 0.00 Bytes 0.00 bits/sec 1 1.00 KBytes
[ 5] 22.00-23.00 sec 0.00 Bytes 0.00 bits/sec 0 1.00 KBytes
[ 5] 23.00-24.00 sec 0.00 Bytes 0.00 bits/sec 0 1.00 KBytes
[ 5] 24.00-25.00 sec 0.00 Bytes 0.00 bits/sec 0 1.00 KBytes
[ 5] 25.00-26.00 sec 0.00 Bytes 0.00 bits/sec 0 1.00 KBytes
[ 5] 26.00-26.81 sec 0.00 Bytes 0.00 bits/sec 0 1.00 KBytes

Happens always. !

Any ideas where should I look for logs more than welcome

stephenw10

When I make that change I see logged:

Jul 24 15:06:24 	check_reload_status 	687 	Syncing firewall
Jul 24 15:06:24 	php-fpm 	57549 	/system.php: Configuration Change: admin@172.21.16.8 (Local Database): System:
Jul 24 15:06:25 	php-fpm 	57549 	/system.php: Staging AutoConfigBackup encrypted configuration backup for deferred upload to https://acb.netgate.com
Jul 24 15:06:27 	root 	45627 	/etc/rc.d/hostid: WARNING: hostid: unable to figure out a UUID from DMI data, generating a new one
Jul 24 15:06:34 	php-fpm 	57549 	/system.php: NTPD is starting up.
Jul 24 15:06:34 	check_reload_status 	687 	Reloading filter

So you can see it reloads the firewall ruleset. For some reason. It's not immediately obvious what setting on that page might require it....

netblues

@stephenw10 And it restarts ntp too?

Doesn't look like a feature to me, but is it a bug?

And why reloading filter kills nat sessions to begin with.. I don't remember that being the case in the past (but I might be wrong)

stephenw10

It restarts ntpd because that page has the external ntp server settings on it. But nothing there looks like it would affect the firewall...

netblues

@stephenw10 Did some more fiddling since this made no sense at all.
Went back to 24.11, via boot environments reloaded the newer configuration, and then migrated again to 25.07rc

The problem went away. Almost impossible to reproduce again, strange as it is.

However, going back to 24.11 i saw that sticky connections did work.
Migrating to latest 25.07 rc, sticky connections are gone.

stephenw10

Like in a load-balanced gateway group?

netblues

@stephenw10 Exactly on that.

stephenw10

Hmm, I'm not aware of that. How are you testing? What are you seeing?

netblues

@stephenw10
I have two 1Gbit ftth lines(with 500Mbits upload)
One is dhcp, the other is pppoe
I have a gateway group called Loadb with the two gateways as Tier1 and another backup tier
as tier3 via 5g
On the lan side, I have a typical rule that sends the traffic there

This works fine. With stickiness enabled I get this

Which is expected, and doing a speedtest utilizes randomly one of the two lines

This is with stickiness enabled

Unchecking "use sticky connections", the source tracking tab disappears (as expected)
and a speedtest (multi connection by default) gets the aggregate as it should

Upgrading to 25.07 rc has no entries in the source tracking states and speedtests are always get the aggregate traffic.
This is a problem when visiting https sites , especially web banking and other session sensitive, since you get disconnected due to different source ip. It can be mitigated by excluding https traffic with policy routing, but then the issue persists on non standard ports.

This is with the new kernel mode pppoe driver but I have tried disabling it with no difference too.