MUTLI WAN & NAT ??? [SOLVED - Reply#10]



  • Hi , So I think I may be missing something simple.. but right now I am not sure..

    I have 2 WAN Connections, Setup with PFSense 2.0.2 , MultiGW Works great, Multi Lan LB Works great.. Everything .. Just Great.. until it comes to NAT

    I can get main WAN connection to port fwd on any port and I can reach from the cloud no problem..

    I have NAT on my second WAN connection setup and it for the life of me wont work.. I cant hit it from the cloud.. I have tried using different ports

    So I have tried say to open port 80 for WAN 1 and WAN 2 , Only can reach WAN1.

    Open Port 9090 on WAN1 and port 80 on WAN 2 , Only can reach WAN1.

    I have tried putting the WAN2 Rules before WAN1 also , nothing….

    If I have WAN1 and WAN 2 open on port 80 and then pull WAN1 modem power ( Causing a fail-over ) it works great, all traffic leaves WAN2 no timeouts during the fail-over...BUT Still absolutely unable to open any port... some help please ! :)  ???



  • ? Anyone have any thoughts? Anything would be of help..



  • You are not alone, same here :(
    I can tell you what the problem is, but not how to solve it ;/
    You try to connect through WAN2 on Port 80 (exampel), the traffic goes to your Lan-PC listen on Port 80. The PC answers to PFsense and PFsense send the answer on WAN1 ;/
    Your Device dont recognize that, because it comes from another IP;/
    There is a problem (or iam braindead) with the reply-to rules. Pfsense dont reminder itself where Traffic actually comes from. ;/
    Dirty Fix around is to make a rule on the LAN interface which redirects the Port80 Traffic to WAN2. The Problem here is, that u cant reach Port 80 over WAN1 now.

    !IAM SEARCHING FOR AN ANSWER TOO! :'( :'( :'( :'( :'(



  • Some screenshots would be nice. ( NAT / LAN / WAN / Outbound NAT )



  • Ok will get you some screenshots ,np.

    Ok so I have just tried

    (WAN)
    SERVER 1 (Port80) –------------->                                              --------------> WAN1
                                                      SWITCH  ----------->  PFSENSE
    SERVER 2 (Port22) --------------->                                              --------------> WAN2
                                                                                                  (OPT1)

    what I tried in my first post was

    (WAN)
    SERVER 1 (Port80) --------------->                                              --------------> WAN1
                                                      SWITCH  ----------->  PFSENSE
    SERVER 1 (Port80) --------------->                                              --------------> WAN2
                                                                                                  (OPT1)

    ALSO

    (WAN)
    SERVER 1 (Port80) --------------->                                              --------------> WAN1
                                                      SWITCH  ----------->  PFSENSE
    SERVER 1 (Port22) --------------->                                              --------------> WAN2
                                                                                                  (OPT1)

    What I have noticed , I think it's a problem wit NAT on the OPT1 interface... It doesn't matter if I use the same LAN IP or a differnet one, the port wont open on the OPT1 interface...and yes I have the services running ;)



  • right now I have my Firewall: NAT: Outbound to:

    Automatic outbound NAT rule generation
             (IPsec passthrough included)


    and there are 4 DNS servers there, I just removed 3 of them for the screenshot.



  • So I have gone out and purchased another NIC so now I have tried
                    LAN2 .167 (port22)            .167
                    –------------->                -------->            ------------->WAN
    SERVER 1                          SWITCH            PFSENSE
                    --------------->                -------->            ------------->WAN2 (OPT1)
                    LAN1 .168 (port80)            .168

    So I have done that above , 2 complete separate NIC cards for 2 different LAN segments and have even set the gateways separate LAN1 gets WAN GW , LAN2 gets WAN2 GW

    The port still wont open?!?

    here is my firewall logs when I try to check the port from say http://www.yougetsignal.com/tools/open-ports/



  • Ok so I have done the exact same setup as above . I am not sure why I didn't come to be before.. I clicked the Act of the blocked rule of port 22, there was a 1 scrub on all fragment reassemble block and a block all.. I used the easy rule pass and i disabled scrub and I am not getting firewall block logs for port 22 now when I check to see if it is open.. ( the port still shows closed when I check it ) when I check say port 9191 that is open on WAN1 and not WAN2 , I get the block rule and thats perfect, if i check the port against WAN1 , its open and all is fine.

    On my default route page how can I change the default route gateway to the MultiWAN group I have made, Right now it it set for the WAN gateway only…even thou LAN traffic is LB via multiWAN gw rule on LAN... I'm a little confused now..



  • OK! So I have " Fixed " it.. The issue was just as I figured in my last post.. on the Diagnostics: Routing tables page, The Default route was the " WAN PORT" IP.. I changed my default gateway to my OPT1 Interface and now the default route ip shows the OPT1 IP. Now I am able to reach any port I open from both WAN and OPT1 Ip's …. This is clearly a bug in PFSense, There should be a way to change the default route to a gateway group..this would solve the problem I think.. :) Well my issue is solved correct workaround or not... it's a fix for now..



  • When i change the default gateway to OPT1, then everything i opend on WAN1 dont work anymore…



  • Hey Marv, Here is my full config. I even have ICMP setup on this just so you can check from an external source if ur even able to ping the ip. It's all working very well for me now. nslookup on the domain resolves both ip's and load balancing is working great.

    a link to the image for full size
    http://m37offroading.ca/fullconfig.png



  • Thanks for ur Time.
    But i doesnt work for me ;/
    I have no floating rules…

    The Rule is linked?? Yours not, what is this?

    My Pfsense Version is 2.0.2-Release i386

    I just want to have access to 8059 from the outside, over the FritzboxWAN ;(
    Without Failover and only Fritzbox connectet it works fine ;(



  • If i Set the Fritzbox Wan to default it works, but then every Portforwarding on the Unitymedia WAN is unable to reach :/



  • If I do this it works, with default on WAN…(pic)

    I cant forward ports on WAN, i can do verthing, it never works...

    Iam freaking out  :D




  • Well I see a couple things wrong,

    your Gateway group, Set both to Tier 1,

    For the Firewall: NAT: Port Forward Page, On the configuration of the port 8059 go the the Filter rule association and make that None, Also NAT Refection , Enabled ( if you want to be able to access it from internal also)

    Firewall: Rules, See mine above, See how I have rules in ever NIC and a Floating DNS failover rule, Include all these rules.

    Leave your gateways leave as no default as it is in the screenshot.

    This should work, I have successfully re-installed 2.0.1 -> upgraded to 2.0.2 and did my rules again and it all works perfect with no trouble. If you need more help PM me your email and we can get you up and working.

    The Fix is above thou , that works and tested twice on 2 installs. :)



  • okay, thank you.
    I will try this tommorow.
    Ist just Failover not LoadBalancing, so Tier 1 and 2 is correct , or?



  • No that is not correct, Please marv, follow my screenshots very carefully and match everything the same as your config. It will work ! :)



  • Ty for your Time, but I have to say that you are wrong ;/
    Both on Tier1 is LoadBalancing, i got then two IP, when i checked my Ip in the internet.
    I installed Pfsene again with CD now (therefore i did this over memstick.img).
    Now it works, why ever oO ::)
    The only things i did after the newinstall was. Setting the Gateways on Tier 1 &2, named it Failover, Add a (one) Nat-Rule- with assiocated Firewallrule. And add in the "lan to any" rule, the Failover Gateway.

    Works.

    But thank you for help so far



  • @robtoronto:

    OK! So I have " Fixed " it.. The issue was just as I figured in my last post.. on the Diagnostics: Routing tables page, The Default route was the " WAN PORT" IP.. I changed my default gateway to my OPT1 Interface and now the default route ip shows the OPT1 IP. Now I am able to reach any port I open from both WAN and OPT1 Ip's …. This is clearly a bug in PFSense, There should be a way to change the default route to a gateway group..this would solve the problem I think.. :) Well my issue is solved correct workaround or not... it's a fix for now..

    Ok I figured out, what is different to my others installations, i swapped the WAN Interface in this Installation.
    If the 2WAN is the one with the higher priority in the Gateway Group all is ok, is the first WAN-Interface with higher Prioty nothing works anymore.
    Thats truly a bug.



  • Marv! I am glad you figured it out… This is very much a bug and I have noticed the same thing... I was not 100% Sure that was what was fixing it the first time I got it working.. Doing that the second time It did not work..then I got my config that I have posted in reply 10 and it works great, with a re-install.. I just tried it AGAIN after you said this because I know for a FACT I had to do the same exact thing.



  • I don't have enough experience with multi-WAN to be able to just spot the issue, but I do know of one potential issue I had seen in the rules someone mentioned.  As far as I know, "Pass" on port forwards bypasses the firewall rules, likely including the ones specifying which gateway to use.  Firewall rules should be used instead of directly using "Pass" on port forwards if you have any special firewall rules with gateway settings.  The same may apply to limiters or traffic shaping.  Depending on how your rules are set up, you may need to duplicate some of the settings to the linked rules for your port forwards (I'm not certain of this, though).


  • Rebel Alliance Developer Netgate

    Indeed, using "pass" on rdr will not give them the proper reply-to behavior to work with multi-wan.

    Use a linked rule instead.

    And if your WAN is a static IP type, make sure you have the gateway selected on the interface config so that the firewall will treat the interface as a WAN (this happens automatically for DHCP/PPPoE/etc) and put reply-to on the firewall rules.



  • @jimp:

    Indeed, using "pass" on rdr will not give them the proper reply-to behavior to work with multi-wan.

    Use a linked rule instead.

    And if your WAN is a static IP type, make sure you have the gateway selected on the interface config so that the firewall will treat the interface as a WAN (this happens automatically for DHCP/PPPoE/etc) and put reply-to on the firewall rules.

    Yeah, did that :)
    Fact is, that i got problems with the WAN1 as Default.
    Lets say WAN1 is UM(the internetprovider)
    Lets say WAN2(opt1) is 1und1.
    If UM is default on WAN1 it doesnt work.
    If i swap the assignement and UM is on WAN2 everything works well, with exact the same rules.


  • Rebel Alliance Developer Netgate

    That still suggests your rules do not get reply-to for some reason. As above, the usual suspects are:

    • Using "pass" on port forwards (rdr pass does not support reply-to, so that can't ever work, not something we can change)
    • Not selecting a gateway on the Interface page for both WAN and WAN2 if using a static IP (this will make the firewall fail to fully treat the interface as a WAN, and skip reply-to)
    • Checking the box in the advanced options to disable reply-to (Which, as the option implies, removes reply-to from rules)

    Post the full contents of config.xml (you can remove keys/passwords) and a copy of /tmp/rules.debug.

    If you choose to obscure IP addresses, leave enough to distinguish them (e.g. last octet), and do not remove them entirely.



  • @jimp:

    That still suggests your rules do not get reply-to for some reason. As above, the usual suspects are:

    • Using "pass" on port forwards (rdr pass does not support reply-to, so that can't ever work, not something we can change)
      i used pass and no RDR ( The checkbox is unchecked)
    • Not selecting a gateway on the Interface page for both WAN and WAN2 if using a static IP (this will make the firewall fail to fully treat the interface as a WAN, and skip reply-to)
      I have set a gateway for the static and none for the dhcp (will choose it on his own)
    • Checking the box in the advanced options to disable reply-to (Which, as the option implies, removes reply-to from rules)
      No, xD. With Wan2 as default it works…

    Post the full contents of config.xml (you can remove keys/passwords) and a copy of /tmp/rules.debug.

    If you choose to obscure IP addresses, leave enough to distinguish them (e.g. last octet), and do not remove them entirely.

    I will it post later, not at home right at the moment


  • Rebel Alliance Developer Netgate

    If you used "pass" on the port forward, it will not work with any non-default route WAN, and it can't work with "pass".

    Change that to "add associated firewall rule" and then save/apply the port forward. Do that for every port forward you have that uses 'pass'



  • @jimp:

    If you used "pass" on the port forward, it will not work with any non-default route WAN, and it can't work with "pass".

    Change that to "add associated firewall rule" and then save/apply the port forward. Do that for every port forward you have that uses 'pass'

    Sry I was not accurate  enough. I used pass in the Firewall rule, not block and not reject.

    BTW: add associated firewall rule is the default setting, when you create a NAT rule.



  • Rebel Alliance Developer Netgate

    I'll still need to see the config.xml and /tmp/rules.debug. A screenshot of that doesn't tell me enough about what it's doing in the background.



  • Hey Jim, Ok I fully understand and now agree with you.; I just got in from a 14 hour shift and have been dealing with recovery of failed modules and stacks we had go down lastnight. I am very tired and do not think I can put my full 100% into this right at this moment. I will grab a copy of my current config, but just to note this is a working copy. I would like to get a 'not' working copy to you also. I would love to do the comparison by my self also, The more I think about it without actually doing it is just racking my brain and have also thought about what you said and you are correct, it very well could be a PFSense configuration issue. From a networking perspective I do know that default route could of been causing the issue. But enough guessing , I will upload a working config today to get you started and I cant promise tomorrow night when I get in from work since I'm sure I will still be swamped at work. But when I get up Monday, I will completely re-install from 2.0.1 -> upgrade to 2.0.2 and cause my same issue.

    and on a note of multiple users, I have people that I know in person and I have read 1 more post on another site. It was no help, but I will try and find that again for you.



  • Here is my raw config.xml & /tmp/rules.debug

    http://m37offroading.ca/PFSENSE/config.xml

    http://m37offroading.ca/PFSENSE/rules.debug

    I have also uploaded them on the redmine post. I will get you copies of the failed config.xml and rules.debug by Monday.


  • Rebel Alliance Developer Netgate

    FYI- You only have the port forward in there once. It should be in there twice (once for each WAN). The WAN rules look OK though.

    rdr on nfe0 proto tcp from any to 24.212.178.88 port $Cloud -> $Server
    pass  in log  quick  on $THOMSONWAN reply-to ( re0 198.84.211.65 )  proto tcp  from any to   $Server port $Cloud  flags S/SA keep state  label "USER_RULE: Allow Server to Cloud on ThomsonWAN"
    pass  in log  quick  on $MOTOROLAWAN reply-to ( nfe0 24.212.178.65 )  proto tcp  from any to   $Server port $Cloud  flags S/SA keep state  label "USER_RULE: Allow Server to Cloud on MotorolaWAN"
    
    


  • I saw port forwards for each.  However, was it your intention to have your port forwards use two IP addresses for the redirect target IP? (10.10.10.20 and 10.10.10.30)  I think using an alias with two (or more) addresses in it like that will make the forwarding cycle through the addresses in the alias on each connection attempt, using a different one each time.



  • ? How could it only be in there once if NAT is working  right now on both IP's, and yes. That example that server runs quad NIC's , the 2 there are used for public traffic.



  • So here we go.
    This is a RAR with a Working Config and a RAR with a non working config.

    I send the password over PM to relevant People, because i dont know that i delete all passwords (or change them in the file to FFF/111)

    http://www.marv21.de/RouterConfig.rar


  • Rebel Alliance Developer Netgate

    @robtoronto:

    ? How could it only be in there once if NAT is working  right now on both IP's, and yes. That example that server runs quad NIC's , the 2 there are used for public traffic.

    Hmm I must have overlooked it, but now the rules.debug file is 404 so I can't check it again. Though Efonne is right about using multiple IPs in the target of a port forward. It will do round-robin connection alternating, it can't forward the same port to both systems and work for both at the same time.
    And if you really want to do load balancing, use the load balancer, not a port forward, since a port forward with two entries would not be smart enough to remove a down host from the pool when it's not in use so half the connections would just fail.


  • Rebel Alliance Developer Netgate

    @Marv21:

    So here we go.
    This is a RAR with a Working Config and a RAR with a non working config.

    I send the password over PM to relevant People, because i dont know that i delete all passwords (or change them in the file to FFF/111)

    http://www.marv21.de/RouterConfig.rar

    In your "don't work" folder the config.xml has two GW_WAN entries, but otherwise I don't see anything there that looks obviously out of place upon first glance. The rules have reply-to, so the default gateway shouldn't matter for items hitting the port forward. Though one of your WANs is getting double NAT, so that doesn't help, but it shouldn't interfere. USB nics (your ue0) are crappy in general but shouldn't affect it either.

    I see you completely reconfigured the two NICs between those two configs, swapping them between "wan" and "opt1". So as it currently is, does the port forward still only work on one WAN? Or does it work on both now?



  • With the "work not" config only the default Gateway works ( in this config the UM WAN ( the dynamic)).

    AT all iam fine, i swaped the assignment of the WAN1 and WAN2 (em0 and ue0) and everything is ok.

    Maybe i forgot to delete a Wan entrie (if i swaped the assignment there a two dynamic Gateways for UM), but i correct that, but i doenst help.



  • Hey, Sorry not sure why it was deleted. It's back on same version. Ok I understand to use load balance if I was going to load balance against 2 web servers to a single nat ip.. but this is not what I am doing or my problem at that.

    my problem is that the multi wan wont nat , even if I have 1 ip of the servers , so forget " dual server ips ". I still have issues with it allowing nat out on the port. this should work…correct ?



  • @robtoronto:

    Hey, Sorry not sure why it was deleted. It's back on same version. Ok I understand to use load balance if I was going to load balance against 2 web servers to a single nat ip.. but this is not what I am doing or my problem at that.

    my problem is that the multi wan wont nat , even if I have 1 ip of the servers , so forget " dual server ips ". I still have issues with it allowing nat out on the port. this should work…correct ?

    BAD Question, but… Are u trieing it from the outside of your LAN?



  • of course ;)


Log in to reply