NAT reflection on multi-WAN and multi-LAN



  • I am running Pfsense 2.1 beta on an Intel Atom DCC2500 with 2 NICs
    I have 4 VLANs for my 4 WAN-connections on interface em1
    and some 20 VLANs on interface em0

    On 2 WAN-connections we were receiving a WAN-address from our ISP for which we are also a reseller.
    The latter is relevant for this topic because it means we have clients that are using that same ISP.
    The 2 other WAN-connections are with a different ISP and they are behind a NAT-router.

    One of the "real" WAN-connections is configured with DHCP and the other one is static.
    We were receiving a subnet /24 with the gateway on x.x.x.1

    In the past I turned on NAT-reflection and noticed that it didn't work well (I don't know exactly what the problem was, then).
    In version 2.1 there's a new form of NAT-reflection that uses a proxy as a helper.
    A few days ago I turned that option on and was pleasantly surprised that we now were able to reach a server on vlan100 using its exposed WAN-address from a client on vlan200.
    This was not possible before with NAT-reflection turned off.

    Yesterday, however, I discovered that when I tried to reach a client's portal using https I went through our own server on vlan100.
    This client also has a WAN-IP in the same subnet as one of our own WAN-connection.
    We have 68.230.240.164/24 and they have 68.230.240.79/24
    I knew it would be fixed if I would have scope of /32

    I then asked our ISP to hand out 68.230.240.164/32 instead of 68.230.240.164/24
    I knew many routers have a problem with such a subnet although this is absolutely unnecessary (I don't want to start a discussion about that!! Did that several times already many years back)
    It turned out it's that way for pfsense (FreeBSD) as well.
    Because the gateway is outside the scope it now can't be reached.

    I'm somewhat uneasy with FreeBSD although I do a lot with Linux.
    I quickly found the command to get my gateway reachable again

    route add -net 68.230.240.1/32 -iface em1_vlan13 
    

    I haven't seen the NAT-reflection rules, but I assume they are wrong as they are probably reflecting all traffic of the WAN's subnet when in fact it should only reflect that 1 IP.
    Can someone confirm this and maybe get a bug fix for this…..

    My workaround works, but it's ugly.
    I now need to generate that command and to make certain it is also stored in the config.xml I'm using cron for this.
    On a debian system I would use an executable script in /etc/network/if-up.d/ but FreeBSD doesn't work that way.

    Such a cronjob is already in my pfsense because of another bug involving the broken support of hardware vlan tagging of my Intel Ethernet card.
    I have this 1-minute cronjob for this:

    ifconfig em0 | grep -q VLAN_HWTAG && ifconfig em0 -vlanhwtag 
    

    Maybe it's unnecessary to re-check this status all the time, but I can't afford to take the risk.
    If the interface is restarted it might turn on vlanhwtag which will render my pfsense unreachable.

    For checking the route and adding it if it's not there I ended up with this command:

    netstat -rn | grep -q '^68\.230\.240\.1/32' || route add -net 68.230.240.1/32 -iface em1_vlan13 
    

    Of course I extensively tested this before I put it in a cronjob.
    It turned out it didn't work properly if I use 'grep -q' instead of merely 'grep'
    quite strange indeed…..
    somehow the netstat command ruins the errorlevel of grep...

    This is correct:

    netstat -rn | grep '^68\.230\.240\.1/32'
    68.230.240.1/32    00:22:4d:7b:7f:0e  US          0   591725 em1_vl
    echo $?
    0
    
    

    This is NOT correct:

    netstat -rn | grep -q '^68\.230\.240\.1/32'
    echo $?
    141
    
    

    Strange isn't it (and only because I'm using '-q')?

    I solved it by doing grep twice:

    netstat -rn | grep '^68\.230' | grep -q '^68\.230\.240\.1/32'
    echo $?
    0
    
    

    Summary:

    • A solution for gateways outside the scope. When ADSL PPP-connections are converted to RFC1483 this is not uncommon

    • Changing the NAT-reflection rules to tamper only with the WAN-IP and not the complete subnet

    • How to execute rules if an interface comes up and make sure it gets into the config.xml

    • What is the reason of this strange grep behaviour in combination with netstat?



  • For the NAT reflection, are you sure that your port forward is not too permissive?  Overly permissive port forwards are one of the most common causes of loss of connectivity related to NAT reflection.  2.0 detects the most common one, using "any" for the destination, and changes it for the reflection rules, but does nothing for other address selections because anything else could be valid.

    Unless I'm misunderstanding the setup, what you are seeing should not happen if your port forward only uses a single IP or alias of multiple IPs for the destination field rather than "any" or a subnet (unless maybe it is a subnet where you own every IP).



  • It was a configuration error.

    I only specified the interface and left the destination address unspecified…
    I should of course selected that interface's address.

    I just reverted all changes and it is working.
    I killed that cronjob that makes sure my gateway is manually added.

    Thanks for your input.

    • The "gateway outside scope issue":
      I still think pfsense should take care of a situation that has a gateway outside of the scope.
      The reason "It's not according to RFC" just isn't enough reason…
      Things should be about "working" vs "not working"

      In fact the ISP's should even give a /32 address to their clients as those clients within the /24 network can only be reached through the gateway and not directly....
      I've seen routers that couldn't cope with that and they had te be given a /32 address

    • The issue with grep:
      I don't know, but I'm not giving it too much thought. If someone has an idea, you're welcome


Locked