Routing on secondary WAN not working
-
At an office location we have the primary WAN with a statically assigned address on a /30 (xn0).
We have a LAN (xn1) which is provided Internet access via NAT over xn0. This works perfectly fine.
We also have a VPN to another office (OpenVPN client via xn0). This also works perfectly fine.The tricky part is that we have a need to host some services on a public IP address. Our ISP has given us an additional /30 which they have routed over the original /30.
So far, so good. We simply added another interface (xn2), assigned it one of the IPs on the new /30, connected the machine with the public services to this interface and gave it the other IP address on the new /30.
I've got wide open firewall rules for testing (allow ipv4 *).
Remote hosts on the Internet can ping the public service machine no problem, however when I try to SSH to it, the connection won't establish.
SSH from pfSense to the public service machine works fine.I've run tcpdump on pfsense and the remote host whilst trying to establish the SSH session.
On the remote host I only ever see the initial SYN packet, never the SYN ACK.
On pfSense, I see the SYN and SYN ACK on both the xn2 and xn0 interfaces.What could possibly be happening to the SYN ACK packet such that it never arrives back at the remote host? My understanding is that since it appears in tcpdump on on xn0, it has passed through the firewall OK.
Routing must be fine as ping works OK.We're looking to replace an existing Linux box running shorewall with pfSense. We are running an identical network configuration on the Linux box (sans the OpenVPN client) and if I switch back to that, the remote host can SSH into the public services machine OK.
A crude diagram:
remote host ----- Internet -----> (xn0 orig /30) pfSense -+- (xn2 new /30) -----> (new /30) public services machine | +- (xn1 priv /24) -----> local LAN -
Did you configure the Outbound NAT on WAN / xn0 accordingly to the doc?
See here: https://docs.netgate.com/pfsense/en/latest/book/routing/routing-public-ip-addresses.html -
Thanks for the reply @viragomann .
I did not see that page, however had set NAT to 'Manual Outbound NAT rule generation' and deleted the relevant rules.Just to make sure, I changed to 'Hybrid Outbound NAT rule generation' as per the docs and added the manual exception rule.
Unfortunately the results are the same - ping works from the public Internet through to the 'internal' WAN, but nothing else.Note that everything, including pfSense is running under XCP-ng, however I don't believe the issue is there as the old Linux router VM with the same three interfaces works fine.
-
@KentW
Do outound connections work from the internal WAN? Can't see the outbound NAT rule you've added, so ensure that there is no NAT on WAN on traffic from the new subnet.Did you get an additional gateway within the new subnet from your ISP?
@KentW said in Routing on secondary WAN not working:
I've run tcpdump on pfsense and the remote host whilst trying to establish the SSH session.
On the remote host I only ever see the initial SYN packet, never the SYN ACK.
On pfSense, I see the SYN and SYN ACK on both the xn2 and xn0 interfaces.On the WAN interface verify if the packets have the origin IP of the internal device and check the MAC where the packets are sent to.
-
OK, I finally found some time to dig into this. Writing this followup in case anyone runs into a similar issue in the future.
Some investigation with tcpdump showed that the remote host was seeing checksum errors on the TCP packets (yet ICMP was working fine).
Disabling hardware checksum offloading in pfSense resolved my issue (System -> Advanced -> Networking -> Hardware Checksum Offloading).
pfSense is running under XCPng 8.1.0 on an unknown motherboard using onboard Intel 82576 Gigabit NIC.
Not sure if the checksum error is creeping in due to XCP or NIC drivers, but in my case the network traffic is quite low, so I'll run with the hardware checksum disabled.