23.09.1 to 24.03 - Makes bridge unstable

oddbear

I upgraded my Netgate 6100 Max from 23.09.1 to 24.03, and after that my network bridge is very unstable.
I tried restarting, reconfigure and all things I could think about, but it could be working for some minutes, and then drop all out.

Booted back to 23.09.1 through Boot Environments, and there it works perfectly.

Any one had any issues like this?
Or could help me figuring it out?

Or come with an alternative for bridge (if any). Reason for it is that I have a LAN port with less firewall rules than a port going to a access point. But because some devices on the wifi should be able to receive broadcast signals from the lan, I would think bridge is the way.

SteveITS

@oddbear Possibly, the default state policy change:
https://docs.netgate.com/pfsense/en/latest/releases/24-03.html#general
https://www.netgate.com/blog/state-policy-default-change

At least, it's easy to toggle it and see.

oddbear

@SteveITS Thanks for the advice, I had totally missed this one as a possible issue. The bridge seems to work as expected after I changed the default state policy.

stephenw10

Interesting. How did you have the bridge filtering configured? What traffic was being blocked?

oddbear

@stephenw10 I have a dual WAN failover that might have impact from what I read in the release and blog post.

I had not changed any of the "System Tunables", so default there. I guess that is what you meant?
But did tried changing the net.link.bridge.pfil_member and net.link.bridge.pfil_bridge, and move my rules from the interface to the bridge for a short time. But did not get any difference there.

The firewall rules on the WIFI interface, was to allow DHCP, and all IPv4 *, nothing more at that time.

From what I remember (as I just tried everything I could think about and find on the internet), the ICMP traffic was fine. But TCP worked for some time, seconds to minutes. And then nothing came through.

stephenw10

@oddbear said in 23.09.1 to 24.03 - Makes bridge unstable:

But TCP worked for some time, seconds to minutes. And then nothing came through.

Mmm, that's pretty much exactly what I would expect from an asymmetric route of some kind. What traffic was failing? Connections from a client on the bridge to an external server? Via some policy routing for the dual WAN?

SteveITS

@stephenw10 said in 23.09.1 to 24.03 - Makes bridge unstable:

exactly what I would expect from a symmetric route

I suspect you meant "an asymmetric route" and that was autocorrect...?

oddbear

@stephenw10 all clients connected to the igc1 (connected to a WIFI AP) had issues. igc1 is bridged to igc0 (LAN connected to a switches) through brigde0.

DHCP (running on pfSense on igc0) then was not picked up on igc1, and not possible to connect to anything... except the AP (ASUS ZenWiFi AX XT8).

Default gateway IPv4 set to a failover gateway group.

stephenw10

@SteveITS said in 23.09.1 to 24.03 - Makes bridge unstable:

I suspect you meant "an asymmetric route" and that was autocorrect...?

Ooops. Yup, can't even blame autocorrect there. Just missed out some letters entirely.

stephenw10

@oddbear said in 23.09.1 to 24.03 - Makes bridge unstable:

DHCP (running on pfSense on igc0) then was not picked up on igc1, and not possible to connect to anything... except the AP (ASUS ZenWiFi AX XT8).

Default gateway IPv4 set to a failover gateway group.

Hmm so igc0 assigned a LAN. igc1 assigned as a OPT interface without an IP address set. The bridge contains igc0 and igc1 but isn't assigned itself?

Are the bridge filtering values set at the defaults?
https://docs.netgate.com/pfsense/en/latest/bridges/firewall.html

oddbear

@stephenw10

Yes.

They are the default, as this:
net.link.bridge.pfil_member = 1
net.link.bridge.pfil_bridge = 0

There are 5 interface assignments
ix3 (WAN 1, DHCP)
ix2 (WAN 2, Static IPv4)
igc0 (LAN switch connected to port, Static IPv4)
igc1 (Access Point connected to port, None)
bridge0 (contains igc0 + igc1, None)

Default gateway is set as failover with ix3 as Tier 1, and ix2 as Tier 2.

DHCP server gives IPs to LAN, and with the bridge to the Access Point.
Firewall rules is set on igc0 and igc1. Not the bridge interface.

stephenw10

Hmm, interesting. And you saw connections failing on both wired and wireless clients?

oddbear

@stephenw10 Just the wireless ones on igc1 failing. The wired ones on igc0 worked fine.

stephenw10

Hmm, interesting did it show blocked traffic? On igc1?

oddbear

@stephenw10 Needed to wait until the WIFI was not in use, before I could test again.

Has set the policy back, and the "WIFI " (bridge) is now down again...

Here are some examples from system logs, firewall, normal view:
x May 3 11:41:03 >LAN Default deny rule IPv4 (1000000104) ...:443 192.168.100.55:52372 TCP:SA
x May 3 11:41:03 LAN_WIFI_BRIDGE Default deny rule IPv4 (1000000103) 192.168.100.2:53240 192.168.100.255:7788 UDP
x May 3 11:41:04 >LAN Default deny rule IPv4 (1000000104) ...:80 192.168.100.55:52343 TCP:SA
x May 3 11:41:04 >LAN Default deny rule IPv4 (1000000104) ...:443 192.168.100.55:52340 TCP:SA

Ping works. 192.168.100.1 is the pfSense box, .2 is the Acccess Point.
Online IPs masked with *

oddbear

Update after looking at the blocking traffic.

On the LAN igc0, the allow IPv4 Rule was set to source "LAN subnets", I changed this to Any, and it seems to work as before with the floating rule, also with the interface state policy.
I did have a dropout in a Voice Call over WIFI, but not sure if that was related, so I will be testing more.

I am learning a lot new things with this issue.

Edit:
Seems like the issues are back after some without any other changes, so that was probably not it.

stephenw10

Hmm interesting. What rules do you have on igc1?

The default rule blocking that traffic on LAN is the outbound rule. It's doing so because the state opened by the SYN from 192.168.100.55 is on igc1 and is now bind to that so fails to match reply traffic on igc0.

So you could just set the rule(s) on igc1 to floating binding instead of the global option.

If you're not filtering between LAN and WIFI you could just move the bridge filtering onto the bridge itself so all states are created there.

oddbear

@stephenw10 I ended up moving the static IP, and the DHCP server to the bridge itself. Now everything seems to work correctly with the new policy (has run fine for 2 days).

Old:
ix3 (WAN 1, DHCP)
ix2 (WAN 2, Static IPv4)
igc0 (LAN switch connected to port, Static IPv4) + DHCP Server
igc1 (Access Point connected to port, None)
bridge0 (contains igc0 + igc1, None)

New:
ix3 (WAN 1, DHCP)
ix2 (WAN 2, Static IPv4)
igc0 (LAN switch connected to port, None)
igc1 (Access Point connected to port, None)
bridge0 (contains igc0 + igc1, Static IPv4) + DHCP Server

stephenw10

Yup if you don't need to filter between the bridge member segments that's what I would have done.