Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0

ColdBrew

@stephenw10 Slow to every page, sometimes I can at least log into the dashboard, but can't go anywhere else after that point, and sometimes it just hangs while logging in.

There was no firewall rule blocking it initially, I have a rule on my LAN interface that allows IT workstations to connect to the MANAGEMENT subnet with HTTP, HTTPS, etc, so traffic coming out of LAN (where my computer is) is PASS to anywhere (including my secondary pfSense instance). Before, my secondary pfSense wouldn't block it, but now it's blocking it using a rule I don't even see in my ruleset, it's bizarre.

I added a rule in the MANAGEMENT interface on my secondary pfSense instance that's a complete copy of the original rule in LAN.

The original rule above was letting traffic pass for IT.

johnpoz

@sweber said in Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0:

(including my secondary pfSense instance)

How about you draw out what you have actually setup... Because its not just a single pfsense, sounds like you might have asymmetrical routing, etc.

How exactly is this connected together?

ColdBrew

@johnpoz PF01 is currently 2.5.2, I'm testing 2.6.0 on PF02:
(When I wrote "FINE" or "Laggy", I'm referring to accessing the webconfigurator through that interface)

stephenw10

Yup, I'd guess asymmetric routing.

You try to access PF02 on the management interface from a client on LAN. The client uses it's default gateway to reach it, which is the PF01. PF01 routes it to PF02 via the MGMT interface. But then PF02 tries to reply directly to the client because it has an interface in the LAN subnet.
That is blocked outbound on LAN in PF02 because it doesn't have a state for that. Hence the permission denied error from nginx.
So I would expect that to have shown blocked outbound in LAN in the firewall logs on PF02?

Nothing should have changed since 2.5.2 though.

Steve

ColdBrew

@stephenw10 Interesting, although nothing changed since 2.5.2 we've never had this issue and my rules didn't change, so what happened? That's the question...

Both firewalls are using pfSync, and I didn't find any outbound connections being blocked on PF02 going back to my client PC.

I checked all the logs for services n such on PF02 after I updated it, but just the fact that I'm having this issue is making me nervous on relying on it for failover, haha! I didn't change anything on 2.5.2, and after PF02 upgraded to 2.6.0, I started having this webconfigurator issue.

ColdBrew

I just double-checked and all my settings, firewall configurations, everything, is as it was before on 2.5.2 and also identical to PF01's configuration (where they sync).

Something must've happened with this update, either a change/fix in the update or an error while it was updating, that's causing this.

ColdBrew

I temporarily allowed ICMP to PF02 and perform another tracert from my workstation to PF02, and it reached successfully as it should through PF01:

I keep doing multiple tracerts and every time it routes the same, so this seems like an issue on the pfSense side on PF02.

This issue

stephenw10

@sweber said in Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0:

I checked my firewall and noticed it was blocking the connection from my workstation to the firewall through the MANAGEMENT interface.

Do you have those block logs still or can you recreate them? Is that the only blocked traffic you saw on either node?

Am I correct that you are testing this from a client in the LAN subnet? Only there?

Steve

johnpoz

@sweber said in Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0:

PF02, and it reached successfully as it should through PF01

But what about the answer? Since pf02 has an interface in 10.30.0..

ColdBrew

@stephenw10 The only blocking I saw was on PF02 on the MANAGEMENT interface after it came from PF01. I'll also try accessing from a client on MANAGEMENT out of curiosity, give me a few minutes.

@johnpoz I'll packet capture and look at the answer

johnpoz

@sweber yeah such a setup would scream asymmetrical to me

Why are you not just accessing the gui via the Lan IP of pf2?

ColdBrew

@johnpoz I just tested directly on the MANAGEMENT interface and it's working fine, so it's some routing issue, but again nothing changed so it's so weird it's suddenly not working anymore with our design we've had for years

Gonna capture some packets

ColdBrew

I can confirm that PF02 is talking back to my workstation on the SECURE interface on PF02, bu it seems to only work when I'm on the login page, enter credentials, and log in to the dashboard; that's as far as I can get. If I try going into any other menu, it just spins and loads forever, the URL doesn't even populate in the address bar, and I don't see any firewall logs showing my workstation talking to PF02 asking for the page and PF02 responding on SECURE.

ColdBrew

Ohhh, interesting thing!

Once i load into the dashboard, when I try to access other pages, those following connections to PF02 are getting blocked on PF01. So, the initial connection is using TCP:S to log in, then access the dashboard and everything else is TCP:A? I've never seen this before...

Here's the rule PF01 is saying blocked TCP:A:

That's the weird thing, we've always had a rule allow TCP/UDP from IT Workstations to Management net and management ports (HTTPS, SSH, all that jazz):

So why in the heck is it blocking it if I have an allow rule for TCP? (That should apply to all subs of TCP, including TCP:S, TCP:A, etc, right?)

johnpoz

@sweber said in Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0:

So why in the heck is it blocking it if I have an allow rule for TCP?

When you see a block on non Syn, you either have asymmetrical traffic or the state went away.. The state is opened by the syn.. If the state goes away and the client continues to send traffic it would be blocked because there is no state.. This would be blocked by the default deny.

Are you doing outbound filtering? those little arrows mean outbound filtering..

ColdBrew

@johnpoz Outbound filtering? Not that I know of, I don't recall configuring that.

So, this must mean the connection state between my workstation and PF02 is getting dropped, and that's why it's breaking? I wonder what would be causing that..

netermin

@sweber Do you have the zabbix agent installed?

johnpoz

@sweber said in Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0:

Outbound filtering? Not that I know of

Do you have rules in floating?

ColdBrew

@netermin No Zabbix.

@johnpoz No rules in floating.

johnpoz

@sweber said in Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0:

No rules in floating.

Well have no idea how your blocking on the outbound direction then? Lack of state? The default deny rules are in/out

if you look in the full rule set

#---------------------------------------------------------------------------
# default deny rules
#---------------------------------------------------------------------------
block in  inet all ridentifier 1000000103 label "Default deny rule IPv4"
block out  inet all ridentifier 1000000104 label "Default deny rule IPv4"
block in  inet6 all ridentifier 1000000105 label "Default deny rule IPv6"
block out  inet6 all ridentifier 1000000106 label "Default deny rule IPv6"

talking back to my workstation on the SECURE interface on PF02,

What interface is that? lan or magement? Still not understanding why you would not just access the gui from the lan interface? Your having to setup a rule to allow it to talk to management from lan anyway? So why not just allow it to the lan IP? Gui going to listen on all IPs anyway.