Trouble accessing SG-1100 web UI via IPsec
-
I want to report a weird behavior that kept me busy for a few days to figure out what's wrong. Maybe this helps someone in a similar situation.
I've been using bigger pfSense appliances like the XG-7100 in several office sites so I was confident in choosing a SG-1100 for a tiny branch office that needed a site-2-site connection.
It was quite easy to setup the IPsec tunnel via direct access to the LAN port of the SG-1100 but then when I unplugged myself and wanted to access the web UI like:
my local workstation's OpenVPN -> my remote office pfSense -> IPsec site-2-site tunnel to the SG-1100 standing next to my local workstation
I encountered connectivity problems. While the port scan (22, 80, 443) succeeded and I was also able to access port 80 with curl, resulting in a redirect to 443, I was unable to work with port 443. The behavior was an randomly slow (it often felt like it would never complete, so I cancelled) TLS handshake and even when I deactivated the https endpoint of the web UI I usually only received a partial html response body.
I fiddled with a lot of settings that you find in the troubleshooting guides like deactivating checksum stuff, adjusting MTU/MSS, I downgraded the configured tunnel ciphers/protocols but nothing seemed to help. After a while I was lucky to figure out that the AES128/256-GCM caused this behavior. Switching the P2s to "AES (Auto)" suddenly made everything work. Fun fact - the related P1 still uses AES256-GCM (128 bit) without issues.
After all I'm a bit sad that I don't understand the underlying issue. But as long as this setting can be considered secure, I guess I'll be fine. If anyone would be interested in more information about the used configuration, I'll be happy to provide them.
-
@tumble And the IPsec tunnel otherwise works as expected with different kinds of traffic?
My initial thought was that you are suffering an MTU size/packet fragmentation issue. -
Was it only the SG-1100 webgui you were seeing problems with? Were you able to access hosts on the SG-1100 LAN OK?
My first thought here is this: https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/access-firewall-over-ipsec.html
Steve
-
@keyser said in Trouble accessing SG-1100 web UI via IPsec:
initial thought was that you are suffering an MTU size/packet fragmentation issue
Yeah, was my thought as well, because this kind of behavior I experienced when the MTU didn't match along the path. Wasn't the culprit here though.
@keyser said in Trouble accessing SG-1100 web UI via IPsec:
And the IPsec tunnel otherwise works as expected with different kinds of traffic
That's how I remember it, but my memory might be flawed.
@stephenw10 said in Trouble accessing SG-1100 web UI via IPsec:
Was it only the SG-1100 webgui you were seeing problems with?
That's how I remember it, yeah.
@stephenw10 said in Trouble accessing SG-1100 web UI via IPsec:
Were you able to access hosts on the SG-1100 LAN OK
Yeah, that worked without any issues which made me look at the IPsec configuration.
So, I tried to go back to my old P2 settings (with AES-GCM) to reproduce the situation and was actually able to do so. Then I tried to access intranet websites to rule out that it's a general problem and happens only when accessing the pfSense web UI. Unfortunately, I noticed I had a DNS issue then. My intranet domain was resolved by the wrong DNS server (a public one). In our case this means you land on a public website that tells you that you got to activate your VPN to access internal services.
So, I was not able to access my intranet and the behavior towards the SG-1100 web UI was exactly as broken as I described in my first posting. However, when I fixed my local workstation's DNS settings (to only use the DNS resolver on the SG-1100) and in addition corrected my search domain in the /etc/resolv.conf I was suddenly able to make it through.
After all, I think I don't have it completely figured out but obviously calling the https port requires name resolution to be working and this resulted in this weird behavior. My P2s are now running with the desired AES-GCM and I guess this can be seen as solved. Sorry for spreading doubt here. But who knows, maybe some other poor soul runs into this someday. :)
Edit: oops, looks like I'm not out of the woods. I'm back in the same broken situation, but unless I figure out something essential I'll stick with not spreading more doubt, because I think it could have something to do with my local setup and plugging in and out etc.… Thank you for your time guys. :)
-
I would try the work around in the link I posted there. It's less of an issue with TCP traffic but can still cause problems.
-
@stephenw10 just tried that and it didn't change anything about the problem. Only switching back to AES (Auto) does the trick.
Another strange side-effect worth mentioning - after unplugging myself from the LAN port (i.e., now accessing the SG-1100 via WAN through IPsec) results in not reaching the SG-1100 due to the firewall blocking me (I see filtered ports with nmap). So I plug back in to the LAN port and check the firewall log and I can see that the firewall doesn't accept me via IPsec although the rules in the IPsec firewall tab explicitly allow it. When I reboot the SG-1100 it suddenly allows me to pass again (without me modfiying the rules or doing anything else). I guess that's caused by some states piling up and my exotic setup, so probably don't need to worry about that.
-
Can we see a screenshot of the blocks that happen when you do that?
Steve
-
The 10.42.42.42 is my IPsec IP. The 192.168.178.24 was my LAN IP.
And the IPsec tab firewall rules are:
with the relevant alias "BER_PRIVILEGED_VPN" looking like this:
and the port scan before and after retry (this one from an earlier case like that. I apparently didn't even port scan anymore on the last one, just triggered the reboot):
-
The blocked TCP:S is odd given those rules.
What rule does it show is blocking that traffic if you mouse-over the X?
-
-
Ok, so it's not matching the pass rule somehow. Either the alias is not populating for some reason or the ruleset is not loading. Try going to Status > Filter Reload and reloading it. Check for errors.
Steve
-
@stephenw10 the alias seems fine:
After a filter reload
Initializing Creating aliases Creating gateway group item... Generating Limiter rules Generating NAT rules Creating 1:1 rules... Creating outbound NAT rules Creating automatic outbound rules Setting up TFTP helper Generating filter rules Creating default rules Pre-caching Block all... Creating filter rule Block all ... Creating filter rules Block all ... Setting up pass/block rules Setting up pass/block rules Block all Creating rule Block all Creating filter rule Block all ... Creating filter rules Block all ... Setting up pass/block rules Setting up pass/block rules Block all Creating rule Block all Pre-caching Default allow LAN to any rule... Creating filter rule Default allow LAN to any rule ... Creating filter rules Default allow LAN to any rule ... Setting up pass/block rules Setting up pass/block rules Default allow LAN to any rule Creating rule Default allow LAN to any rule Pre-caching Default allow LAN IPv6 to any rule... Creating filter rule Default allow LAN IPv6 to any rule ... Creating filter rules Default allow LAN IPv6 to any rule ... Setting up pass/block rules Setting up pass/block rules Default allow LAN IPv6 to any rule Creating rule Default allow LAN IPv6 to any rule Pre-caching allow pings from BER... Creating filter rule allow pings from BER ... Creating filter rules allow pings from BER ... Setting up pass/block rules Setting up pass/block rules allow pings from BER Creating rule allow pings from BER Creating filter rule allow pings from BER ... Creating filter rules allow pings from BER ... Setting up pass/block rules Setting up pass/block rules allow pings from BER Creating rule allow pings from BER Pre-caching allow operators to access this firewall... Creating filter rule allow operators to access this firewall ... Creating filter rules allow operators to access this firewall ... Setting up pass/block rules Setting up pass/block rules allow operators to access this firewall Creating rule allow operators to access this firewall Creating filter rule allow operators to access this firewall ... Creating filter rules allow operators to access this firewall ... Setting up pass/block rules Setting up pass/block rules allow operators to access this firewall Creating rule allow operators to access this firewall Pre-caching allow operators to access this firewall... Creating filter rule allow operators to access this firewall ... Creating filter rules allow operators to access this firewall ... Setting up pass/block rules Setting up pass/block rules allow operators to access this firewall Creating rule allow operators to access this firewall Creating filter rule allow operators to access this firewall ... Creating filter rules allow operators to access this firewall ... Setting up pass/block rules Setting up pass/block rules allow operators to access this firewall Creating rule allow operators to access this firewall Pre-caching Block all... Creating filter rule Block all ... Creating filter rules Block all ... Setting up pass/block rules Setting up pass/block rules Block all Creating rule Block all Creating filter rule Block all ... Creating filter rules Block all ... Setting up pass/block rules Setting up pass/block rules Block all Creating rule Block all Pre-caching allow all OpenVPN... Creating filter rule allow all OpenVPN ... Creating filter rules allow all OpenVPN ... Creating IPsec rules... Creating uPNP rules... Generating ALTQ queues Loading filter rules Setting up logging information Setting up SCRUB information Processing down interface states Running plugins Doneand unplugging from LAN again, then going in via IPsec, the result of a port scan is filtered and the log shows it again:
Gonna reboot the SG-1110 now and expect to be fine again.
-
@tumble
can you please try apfctl -vvsTables?
is the table loaded?you should see something like
--a-r-- BER_PRIVILEGED_VPN Addresses:etc etc
you can also test it withpfctl -t BER_PRIVILEGED_VPN -T test 10.42.42.42do you have floating rules ? extra packages installed ?
it's strange because I have a lot of tables, some with 1000/2000 ips, and I never encountered such a problem, there must be some misconfiguration somewhere
but i have a guy on the italian section of the forum with a similar problem
tcp:s blocked by a deny rule even if the ip is inside an alias and a pass rule is present for it -
@kiokoman said in Trouble accessing SG-1100 web UI via IPsec:
do you have floating rules ? extra packages installed ?
Nope, the SG-1100 is pretty much as it comes out of the box. All I did was setting the domain, configuring the DNS resolver and configure the IPsec tunnel + the rules you can see above.
@kiokoman said in Trouble accessing SG-1100 web UI via IPsec:
it's strange because I have a lot of tables, some with 1000/2000 ips, and I never encountered such a problem
Same here on my bigger appliances (XG-7100).
Unfortunately, I had to ship the SG-1100 to our branch office so I lack physical access to it for now and unless it comes back to me for some reason, I won't get physical access again too soon/ever.
-
@tumble
the guy on the italian forum found out that the problem was associated to a wrong value on MTUwrong MTU -> blocked TCP:Sproblem reappeared, it was not mtu for him but maybe related to
https://redmine.pfsense.org/issues/9296 -
Hard to see how that could be. The packet is arriving over the IPSec. TCP Syn packets are tiny anyway. But if you've seen something similar before I guess....
But that pass rule should match and clearly isn't. IP Options on it or something odd?
Steve
