Hosting connectivity oddities behind pfSense
-
So, this seemed like a really weird issue to me.
pfSense - Firewall/router
Plesk - Hosting server
cPanel - Hosting server
WHMCS - Site is on the Plesk server
1:1 NAT
All ports needed for both servers added in Firewall rules.So this was a rebuild (as was needed) and I decided to put it behind a better Firewall. Everything seemed like it was working before I took it in to DC. Websites were working fine. Just some minor issues to work through, but connectivity seemed fine.
Come to today, a client (known them personally for years, and he's in IT too) said he was getting an error from his account where the "Login to Plesk Control Panel" button should be. This was just an account issue, so applied the password in Plesk and then it showed the error below:
"Error code: 0. Error message: Curl error: [7] Failed to connect to example.com port 8443: Connection refused."This to me is odd, as if it couldn't connect to Plesk, how was it able to know the password was incorrect?
All the attempts to connect to the Plesk panel port (8443) from outside the network connected through to the server without. Connections from WHMCS to cPanel wouldn't go through. Changing settings in the Firewall on the cPanel server and Plesk server managed to get this working. But then I have removed these settings, as the Plesk Firewall is meant to allow outbound traffic by default and cPanel appears to have all the ports needed for outbound traffic.
Then spent hours trying to work out why Plesk refused connections from WHMCS. Connections from both the Plesk and cPanel servers (telnet) to the hostname of the Plesk server and port 8443 would not work. I could connect on the LAN IP, but if I set this in WHMCS and tried to click the control panel button it would load the LAN IP, so this was not an option.
I then tried adding an outbound NAT mapping for both WAN and LAN with the LAN Network as the source. This then somehow got the connections to Plesk working and WHMCS is now able to interact again with Plesk. I have disabled the mappings and made sure it was applied and connections are still working.
Nothing was in the Firewall logs in regards to these connections being blocked.
So how is it that these connections appeared to be blocked when trying telnet hostname 8443 from both hosting servers, but weren't from external connections, and now they appear to be working?
Nothing seems to make sense. Any ideas or experiences that would help figure this out would be greatly appreciated.
-
guess it's a nat reflection issue.
a good solution is to setup split-dns.
a bad solution is to fuck around with nat-reflection settings.enjoy
https://docs.netgate.com/pfsense/en/latest/nat/reflection.html#nat-splitdns
https://docs.netgate.com/pfsense/en/latest/nat/reflection.html -
Thanks heaps for the information. Any idea what the default settings for Nat Reflection are in Advanced -> Firewall & NAT?
Things seem to be working with it turned on, so will now leave it for now and revisit it if things go pear shaped.
-
@easy-hostingnz It defaults to disabled. Enabling it there enables reflection for all rules. Alternately you can edit a NAT rule and change NAT Reflection from "system default" to enable it.
Reflection sends that connection/traffic through the router, while split DNS doesn't use the router because the devices uses a LAN IP. If the NAT doesn't translate ports then either will work.