Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0
-
@johnpoz PF01 is currently 2.5.2, I'm testing 2.6.0 on PF02:
(When I wrote "FINE" or "Laggy", I'm referring to accessing the webconfigurator through that interface)
-
Yup, I'd guess asymmetric routing.
You try to access PF02 on the management interface from a client on LAN. The client uses it's default gateway to reach it, which is the PF01. PF01 routes it to PF02 via the MGMT interface. But then PF02 tries to reply directly to the client because it has an interface in the LAN subnet.
That is blocked outbound on LAN in PF02 because it doesn't have a state for that. Hence the permission denied error from nginx.
So I would expect that to have shown blocked outbound in LAN in the firewall logs on PF02?Nothing should have changed since 2.5.2 though.
Steve
-
@stephenw10 Interesting, although nothing changed since 2.5.2 we've never had this issue and my rules didn't change, so what happened? That's the question...
Both firewalls are using pfSync, and I didn't find any outbound connections being blocked on PF02 going back to my client PC.
I checked all the logs for services n such on PF02 after I updated it, but just the fact that I'm having this issue is making me nervous on relying on it for failover, haha! I didn't change anything on 2.5.2, and after PF02 upgraded to 2.6.0, I started having this webconfigurator issue.
-
I just double-checked and all my settings, firewall configurations, everything, is as it was before on 2.5.2 and also identical to PF01's configuration (where they sync).
Something must've happened with this update, either a change/fix in the update or an error while it was updating, that's causing this.
-
I temporarily allowed ICMP to PF02 and perform another tracert from my workstation to PF02, and it reached successfully as it should through PF01:
I keep doing multiple tracerts and every time it routes the same, so this seems like an issue on the pfSense side on PF02.
This issue
-
@sweber said in Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0:
I checked my firewall and noticed it was blocking the connection from my workstation to the firewall through the MANAGEMENT interface.
Do you have those block logs still or can you recreate them? Is that the only blocked traffic you saw on either node?
Am I correct that you are testing this from a client in the LAN subnet? Only there?
Steve
-
@sweber said in Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0:
PF02, and it reached successfully as it should through PF01
But what about the answer? Since pf02 has an interface in 10.30.0..
-
@stephenw10 The only blocking I saw was on PF02 on the MANAGEMENT interface after it came from PF01. I'll also try accessing from a client on MANAGEMENT out of curiosity, give me a few minutes.
@johnpoz I'll packet capture and look at the answer
-
@sweber yeah such a setup would scream asymmetrical to me
Why are you not just accessing the gui via the Lan IP of pf2?
-
@johnpoz I just tested directly on the MANAGEMENT interface and it's working fine, so it's some routing issue, but again nothing changed so it's so weird it's suddenly not working anymore with our design we've had for years
Gonna capture some packets
-
I can confirm that PF02 is talking back to my workstation on the SECURE interface on PF02, bu it seems to only work when I'm on the login page, enter credentials, and log in to the dashboard; that's as far as I can get. If I try going into any other menu, it just spins and loads forever, the URL doesn't even populate in the address bar, and I don't see any firewall logs showing my workstation talking to PF02 asking for the page and PF02 responding on SECURE.
-
Ohhh, interesting thing!
Once i load into the dashboard, when I try to access other pages, those following connections to PF02 are getting blocked on PF01. So, the initial connection is using TCP:S to log in, then access the dashboard and everything else is TCP:A? I've never seen this before...
Here's the rule PF01 is saying blocked TCP:A:
That's the weird thing, we've always had a rule allow TCP/UDP from IT Workstations to Management net and management ports (HTTPS, SSH, all that jazz):
So why in the heck is it blocking it if I have an allow rule for TCP? (That should apply to all subs of TCP, including TCP:S, TCP:A, etc, right?) -
@sweber said in Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0:
So why in the heck is it blocking it if I have an allow rule for TCP?
When you see a block on non Syn, you either have asymmetrical traffic or the state went away.. The state is opened by the syn.. If the state goes away and the client continues to send traffic it would be blocked because there is no state.. This would be blocked by the default deny.
Are you doing outbound filtering? those little arrows mean outbound filtering..
-
@johnpoz Outbound filtering? Not that I know of, I don't recall configuring that.
So, this must mean the connection state between my workstation and PF02 is getting dropped, and that's why it's breaking? I wonder what would be causing that..
-
@sweber Do you have the zabbix agent installed?
-
@sweber said in Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0:
Outbound filtering? Not that I know of
Do you have rules in floating?
-
-
@sweber said in Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0:
No rules in floating.
Well have no idea how your blocking on the outbound direction then? Lack of state? The default deny rules are in/out
if you look in the full rule set
#--------------------------------------------------------------------------- # default deny rules #--------------------------------------------------------------------------- block in inet all ridentifier 1000000103 label "Default deny rule IPv4" block out inet all ridentifier 1000000104 label "Default deny rule IPv4" block in inet6 all ridentifier 1000000105 label "Default deny rule IPv6" block out inet6 all ridentifier 1000000106 label "Default deny rule IPv6"talking back to my workstation on the SECURE interface on PF02,
What interface is that? lan or magement? Still not understanding why you would not just access the gui from the lan interface? Your having to setup a rule to allow it to talk to management from lan anyway? So why not just allow it to the lan IP? Gui going to listen on all IPs anyway.
An intelligent man is sometimes forced to be drunk to spend time with his fools
If you get confused: Listen to the Music Play
Please don't Chat/PM me for help, unless mod related
SG-4860 24.11 | Lab VMs 2.8, 24.11 -
@johnpoz LAN/SECURE are the same thing.
I mean, I can just access it through SECURE, the previous admin just set it up with the management IP cause it made more sense I guess.
-
@sweber said in Webconfigurator slow through MANAGEMENT interface after upgrade from 2.5.2 to 2.6.0:
with the management IP cause it made more sense I guess.
Yeah if you were on the management network... But accessing it the way you are from lan is going to be asymmetrical that is for sure.. And you showed that with your sniff..
To be honest I don't know how it would have actually worked correctly.. Where you source natting before? If you source natted the traffic going to management IP from the lan, so it looked like it came from the other pfsense management IP then your return traffic wouldn't be asymmetrical
Or if you host routed on your client to use either pf01 or pf02 lan IP to get to the specific management IP.. then again the return traffic wouldn't be asymmetrical.
If the carp vip was owned by the specific pf you were going through for your gateway where the management IP was located it should work as well. But if you access the other pf managment IP from the lan its going to be asymmetrical return traffic - unless you could set reply-to on the traffic. Which I don't think you can do unless its a wan, this get set on a wan interface, unless you disable it. But I am not sure you could set specific reply-to on just normal interfaces?
The easy solution is just access the lan IP of the pf you want to access the gui of when your on the lan network.
