Seems like a NAT issue with static outbound…



  • Hello all.

    Some background info:

    We're a PC repair shop, 5 full-time techs.  We've been using pfSense as our edge router with great success for over a year now in our office/shop.  We run something like 11 subnets total if you count the two site-to-site vpn tunnels, and the 5 workbench subnets behind another non-NAT router.

    I recently switched the pfSense box over to static outbound NAT, in order to get our Hamachi clients working properly.  This is documented elsewhere, and the Hamachi clients work great now.  No more relayed (meaning slow) connections.

    I carefully set up a static outbound NAT rule for each subnet, and all work just fine, except…..

    We have a box running the UltraVNC Repeater software.  Combined with UltraVNC-SC, this gives us quick and simple remote support to our customers.  They d/l the file from our website and run it, and we can connect to their desktop instantly.  Nice.  We use it in what their docs call "Mode II".  Our website forwards that file d/l request into this little box, which served the EXE file.  That part works just fine.  The VNC repeater setup uses two port forwards, 5500 and 5901.  I assume one is for auth or control, and one for I/O data

    Ever since changing over to static outbound NAT, the VNC portion has stopped working.  The VNC connection goes live, and confirms the connection like it should, indicating at least some level of end-to-end connectivity.  The viewer windows comes up, but we never get a screen image on our end.  Just stays black.

    I've been over and through the NAT setup and firewall rules.  I can't seem to figure this out.  Any insight at all would be fantastic.

    Curious side-note:  I have a site-2-site tunnel to my home network (another pfSense box).  If I pull up a system with MS RDP, then use it to start the VNC connection, it works, and I can see the same screen in both windows side-by-side.  That is, until I close the RDP session, and the VNC connection then drops out as well.  This seems to show that there's some kind of problem getting a path through the router??  Not sure.

    This behavior seems very odd, and I don't really know if I'm having a problem with TCP states, firewall rules, or NAT rules.  If anybody can get me pointed in the right direction, I'll be forever grateful.

    -Justin



  • Have you confirmed that this is as simple as checking the "static port" box and it breaks?  If so, it might be simpler to just add a rule for the VNC ports before the others, and have them NOT use "static port".



  • First off, I appreciate the response.  It's broken without specifying static port (assuming you are referring to the manual outbound NAT rule).  To be clear, currently the rule sits as follows:

    Interface: WAN
    Source:
    Type: Network
    Address: 10.10.10.1/24
    Source port: (blank)
    Destination:
    Type: any
    Address: (blank)
    Dest. port: (blank)
    Translation:
    Address: Interface Address
    Port: (blank)
    Static Port: (unchecked)



  • I guess I am confused, then.  Rereading your OP, I got the impression VNC always worked, but didn't after you changed to static outbound to allow hamachi?



  • Yes.  VNC repeater worked fine, but Hamachi could only produce relayed (slow) connections.  Solution for better Hamachi performance was to change to manual outbound NAT, which helps with Hamachi but seems to break the VNC repeater.  Essentially I think I need to confirm and/or correct my outbound NAT and port forwards for this box.  I can supply any additional config details if needed.

    Thanks again.

    -Justin



  • You said this was documented elsewhere (how to fix hamachi).  Did the page in question explain WHY auto-nat was hosing hamachi?  Try this (getting more info here).

    Go back to auto-nat, and in a shell, 'pfctl -s rules; pfctl -s nat'.  Save output.  Switch back to manual nat with no static port and repeat the exercise.  Compare the two sets of output (hopefully not many differences.)  Also, have you tried setting static port on?



  • OK.  Now I'm confused.  Switched back to automatic outbound NAT.  Hamachi immediately complains that all the tunnels are relayed, as expected.  No big deal for now.  However, even back on automatic outbound NAT, the VNC connections still fail.  Looks like I was mistaken as to the cause of the problem.

    Compared the two outputs after piping into text files, and the differences are enormous.  Probably due to the NAT handling of 11 subnets and 6 VLANs.  Seems like a ton of stuff to compare visually.  I'll go through it and look for clues, but at this point I think need to confirm the VNC box isn't the problem.

    I appreciate the input.  I'll keep chasing this and make certain there's not an issue with the VNC box or configuration before we continue here.

    Thanks again.  Will post back with results sometime soon.

    -Justin



  • okay, let us know…



  • OK, here's an update.

    It seems like NO port forwarding was actually working, except for the one port I had forwarded on the LAN interface.  Due to some other nagging issues (relating to the system flat-out crashing anytime I deleted my unused VLANs) I decided to rebuild the configuration from scratch.

    I've reset the box to factory defaults.  Reconfigured the interfaces, IP addresses, and firewall rules.  All subnets now have internet access again.  No VLANs are configured (we're doing routing instead).

    Since one of my pfSense interfaces feeds into another router (vyatta vc5), it seems that I have to do manual outbound NAT in order for those subnets behind the second router to have access to the internet.  So that's what I have done.  Seems to work properly.

    My current issue now is that, once again, port forwards seem to be ineffective other than on the LAN interface, the interface which pfSense sets up by default.  Forwarding ports into any of my OPT interfaces seems to be ineffective.  For instance, forwarding port 80 into a host on OPT1 does not pass the traffic.  It seems this behavior mimmicks the problems I had getting the previous VNC repeater box working as well, as it was also on an OPT interface.

    Is port forwarding handled differently with manual outbound NAT?  Do I need to also create some kind of reverse rule?  I'm stumped.



  • I'd be curious to see the output of 'pfctl -s all | grep 80' with the port forward to a LAN host, and then an OPT1 host (more usefully, a diff of the rules - hopefully not too huge?)



  • Oh its huge.  Several pages worth.  Help me out w/ a command to pare it down a bit?



  • It's huge even grepping for port 80?  Sigh.  Try grepping for 80 and pipe that into another grep for the host you are forwarding that to.  Maybe also pick a silly port no-one uses (like 12345?)



  • ok, here it is as it sits right now…  call me paranoid but I've masked my public IP with letters...  This is with Manual outbound NAT

    2 ports forwarded on the OPT1 interface:

    pfctl -s all | grep 192.168.5.99

    rdr on sis0 inet proto tcp from any to A.B.C.D port = hosts2-ns -> 192.168.5.99 port 80
    rdr on sis0 inet proto tcp from any to A.B.C.D port = https -> 192.168.5.99

    ...and here's one on the LAN interface...

    pfctl -s all | grep 192.168.0.6

    rdr on sis0 inet proto tcp from any to A.B.C.D port = smtp -> 192.168.0.6
    pass in quick on sis0 reply-to (sis0 A.B.C.1) inet proto tcp from any to 192.168.0.6 port = smtp flags S/SA keep state label "USER_RULE: NAT Mail to Exchange"

    Not knowing much of the nuts-n-bolts of BSD, I'll need a bit of help to decipher this.



  • I'm confused.  Maybe I wasn't clear, but what I wanted was the output for both cases, with LAN and OPT1 as only differences.  Is that what you posted?



  • Well, yes..

    But the examples are for two different hosts, which I already had set up.  Can I replicate the same port forward to two different destination hosts?  I didn't expect that would be allowed.

    What I've posted is one example of ports 80 & 443 forwarded to a host on the OPT1 interface, with destination IP 192.168.5.99.

    Second example is port 25, forwarded to a host on the LAN interface, 192.168.0.6.

    The output is clearly different between the two.

    If you'd rather I make a more precise comparison, I can certainly do that, but I expected this set of examples might display the differences you were looking for.

    I use the destination IP's as the grep variable, which seemed to produce good results.



  • Ah, okay, this makes sense then.  What I see that is suspicious: there is no 'pass in quick' rule for the two OPT1 port forwards.  If you go to the firewall => rules section, do you not see any auto-generated rules for the OPT1 port forwards?  if not, that is the issue, IMO.  I don't know why it would not be generating those (maybe a bug?)  If so, you can probably get around that by adding them yourself?



  • OK, sorry for the confusion, but this makes more sense.  I had an error in there, which I've now corrected.  Funny how scrutiny turns up those things… Here's the (correct) output per the previous request.  Port 80 on Opt1, port 25 on LAN.  I've confirmed the port forward actually

    pfctl -s all | grep 192.168.5.99

    rdr on sis0 inet proto tcp from any to A.B.C.D port = http -> 192.168.5.99
    pass in quick on sis0 reply-to (sis0 A.B.C.1) inet proto tcp from any to 192.168.5.99 port = http flags S/SA keep state label "USER_RULE: NAT "

    pfctl -s all | grep 192.168.0.6

    rdr on sis0 inet proto tcp from any to A.B.C.D port = smtp -> 192.168.0.6
    pass in quick on sis0 reply-to (sis0 A.B.C.1) inet proto tcp from any to 192.168.0.6 port = smtp flags S/SA keep state label "USER_RULE: NAT Mail to Exchange"

    So now the outputs match, as I would expect, but the port forward doesn't actually "work" on the OPT interface.  I still wonder if this has something to do with the outbound NAT, and the packets not getting back out?  Just a thought.



  • what (if any) are your OPT1 rules?



  • Also, test out this theory (if possible) by running some kind of sniffer on a host on the OPT1 subnet, and see if it sees anything coming from the pfsense box when you try to connect from outside.



  • Good  Morning.  Opt FW rules look like this.  This interface IP is 192.168.5.1/24

    Proto  Source  Port  Destination  Port  Gateway  Schedule  Description 
    X    * OPT3 * 192.168.0.0/24 * *

    * OPT3 * *                 * *

    Wan rules look like this:

    TCP  *  *  192.168.0.6  25 (SMTP)  *      NAT Mail to Exchange

    TCP * * 192.168.5.99 80 (HTTP) *   NAT  Web server

    Nat rules next:

    WAN  TCP  25 (SMTP)  192.168.0.6(ext.: A.B.C.D) 25 (SMTP) Mail to Exchange

    WAN  TCP  80 (HTTP)  192.168.5.99(ext.: A.B.C.D) 80 (HTTP)

    To review, the port 25 forwards OK, the 80 does not.  Mail host is on the LAN interface, web is on the OPT interface.

    Outbound manual NAT rule is the standard default, with static port enabled (for both networks).  Static port enabled/disabled seems to make no difference.

    I'll try to capture some packets next and see what happens.



  • Well, the packet capture was very enlightening.  It showed, well, no packets at all.  A little detective work with alternate ports indicates that my lovely ISP is blocking port 80, despite allowing port 25 and being a commercial connection.  Changed to another port number and everything works as it should.  As for the original issue with the VNC ports, I suspect there was a separate issue there as well.  I'll consider this mystery solved.  Thanks for all your time and effort, it certainly did lead me to the solution.  Port forwarding works just fine, as long as the packets actually get there!


Log in to reply