VPS as public IP provider



  • Good day:

    I've been reading a while, but this is my first time posting. I'm beating my head against a problem and I hope I'm missing something obvious. This will be fairly long, but I'm hoping to answer all the right questions up front.

    My premise:
    Today, I have a business-class Internet pipe which grants me a static IP and no caps. I host some stuff (Owncloud, mail server, etc.) which require the public IP. The cost of this service has grown and grown, and I'm looking at eliminating the business class service, which will put me back on a dynamic IP. For various reasons, I want to keep my equipment hosted locally, so I came up with the idea of using a cheap VPS at a very close data center (RTT ~2ms beyond the ISP gateway) to provide multiple public IPs. Going forward, I'm also interested in a second Internet circuit at home (possibly wireless), but that's out of the scope right now.

    What I've got:
    On premise: Virtualized pfSense 2.3.2, currently on a static IP, with about 6 different VLANs. The firewall itself works perfect.
    In the cloud: Virtualized pfSense 2.3.2, "one arm" configuration with only a static WAN IP.

    What I've done:
    I've configured an OpenVPN tunnel in routed mode, with the VPS as the server and the home as the client. I'm using the tunnel network 192.168.254.0/29, so the server is .1 and the client is .2.
    I've configured a firewall rule on-premise to policy based route all traffic from a specific inside host (just a testing server in an isolated VLAN) through the remote gateway.
    I've configured a NAT rule on the VPS side to route traffic to a specific port on the VPS' external IP to the internal system (SSH for now).

    What works and what doesn't:
    Traffic from the testing server outbound is properly routed. It appears to the web as originating from the VPS' external IP.
    SSHing from the VPS to the  internal server (using its internal IP address) works.
    SSHing from the world to the VPS' external IP, on the port configured in the NAT rule, doesn't work.

    Troubleshooting:
    I see the traffic being routed properly all the way to the internal server. It responds, to the originating external IP. However, this traffic is routed to the home network's WAN interface - an asymmetric route.

    So, I've considered a few options. One, change from routing mode to bridged mode, then configure the server to have its default route with the VPS and static routes to the on-premise network. This would seem to be the simplest answer, although I don't have any real need for bridged mode, and the general consensus around these parts is "don't do it!" Two, run haproxy on the VPS… this works, but I see only the VPS's internal IP in my log files... a workable solution, although not the most ideal.

    I've played with NAT and firewall rules until I'm blue in the face. Even disabling all of the firewall and NAT rules, the traffic still takes the same asymmetric route.

    I think I'm missing something incredibly obvious here... were this an ASA or a Palo Alto, I would have solved this long ago - but ASAs don't run on VPSs and I can't afford a virtual Palo Alto in AWS! Any help would be greatly appreciated!



  • I did find a workaround, by adding a second copy of PFSense that sees the VPN as its default gateway. Then, creating static routes on the hosting servers to have them talk back into the inside network via the existing gateway, but default gateway to the new PFSense. While this works, and I've got plenty of resources to run it, it still feels klugey and will be more administrative overhead. If anyone has suggestions for making all of this work within one instance of PFSense, I'd love to hear them.



  • Something to try :).

    Assign the openvpn interfaces to OPT1 OPT2 network names on pfSense interfaces\assing, then also make sure to configure a gateway on that interface's edit page. This should result in the rules on this OPT1 openvpn interface to have 'reply-to' in its firewall rules check this on /tmp/rules.debug. That should allow for reply traffic to pass back out the proper interface.

    Regards,
    PiBa-NL


  • Rebel Alliance Global Moderator

    What exactly is the problem with your dynamic IP, does your isp force this to change all the time?  I am on a dynamic IP from comcast, just home connection.  And this IP has been the same for years..  Only reason this should change is if your going offline for long enough for your lease to expire, and then someone else grabs it before you come back online.

    Unless of course your isp one of those asshat ones that force ip change all the time?

    So if you wanted to leverage a vps as a remote site and route traffic that comes in that site you would have to source nat to make it looks like comes from the IP(s) in that remote network.  Where your problem is while you might route traffic that comes into your vps, if its not natting that traffic its source would still be the public IP.  In that case then sure when client on your network wants to answer it would just go out its normal routing and pfsense would send it out its normal default routing.  Why would it route it back out the vpn if the source is a public IP.

    The solution of reply-to might work a well.  I have not played with that sort of solution much, I tend to be more kiss if want traffic to return to the vps via that tunnel just make sure the traffic looks like it came from that remote site ip range.



  • Any updates on this? I have a similar setup working fine but I'd like to do this with multiple VPS IPs and multiple hosts behind the pfSense router.

    If possible, I'd also like to rewrite the source IP to the actual client IP and not the VPN gateway.