MUTLI WAN & NAT ??? [SOLVED - Reply#10]

Marv21

Ty for your Time, but I have to say that you are wrong ;/
Both on Tier1 is LoadBalancing, i got then two IP, when i checked my Ip in the internet.
I installed Pfsene again with CD now (therefore i did this over memstick.img).
Now it works, why ever oO ::)
The only things i did after the newinstall was. Setting the Gateways on Tier 1 &2, named it Failover, Add a (one) Nat-Rule- with assiocated Firewallrule. And add in the "lan to any" rule, the Failover Gateway.

Works.

But thank you for help so far

Marv21

@robtoronto:

OK! So I have " Fixed " it.. The issue was just as I figured in my last post.. on the Diagnostics: Routing tables page, The Default route was the " WAN PORT" IP.. I changed my default gateway to my OPT1 Interface and now the default route ip shows the OPT1 IP. Now I am able to reach any port I open from both WAN and OPT1 Ip's …. This is clearly a bug in PFSense, There should be a way to change the default route to a gateway group..this would solve the problem I think.. :) Well my issue is solved correct workaround or not... it's a fix for now..

Ok I figured out, what is different to my others installations, i swapped the WAN Interface in this Installation.
If the 2WAN is the one with the higher priority in the Gateway Group all is ok, is the first WAN-Interface with higher Prioty nothing works anymore.
Thats truly a bug.

robtoronto

Marv! I am glad you figured it out… This is very much a bug and I have noticed the same thing... I was not 100% Sure that was what was fixing it the first time I got it working.. Doing that the second time It did not work..then I got my config that I have posted in reply 10 and it works great, with a re-install.. I just tried it AGAIN after you said this because I know for a FACT I had to do the same exact thing.

Efonnes

I don't have enough experience with multi-WAN to be able to just spot the issue, but I do know of one potential issue I had seen in the rules someone mentioned. As far as I know, "Pass" on port forwards bypasses the firewall rules, likely including the ones specifying which gateway to use. Firewall rules should be used instead of directly using "Pass" on port forwards if you have any special firewall rules with gateway settings. The same may apply to limiters or traffic shaping. Depending on how your rules are set up, you may need to duplicate some of the settings to the linked rules for your port forwards (I'm not certain of this, though).

jimp

Indeed, using "pass" on rdr will not give them the proper reply-to behavior to work with multi-wan.

Use a linked rule instead.

And if your WAN is a static IP type, make sure you have the gateway selected on the interface config so that the firewall will treat the interface as a WAN (this happens automatically for DHCP/PPPoE/etc) and put reply-to on the firewall rules.

Marv21

@jimp:

Indeed, using "pass" on rdr will not give them the proper reply-to behavior to work with multi-wan.

Use a linked rule instead.

And if your WAN is a static IP type, make sure you have the gateway selected on the interface config so that the firewall will treat the interface as a WAN (this happens automatically for DHCP/PPPoE/etc) and put reply-to on the firewall rules.

Yeah, did that :)
Fact is, that i got problems with the WAN1 as Default.
Lets say WAN1 is UM(the internetprovider)
Lets say WAN2(opt1) is 1und1.
If UM is default on WAN1 it doesnt work.
If i swap the assignement and UM is on WAN2 everything works well, with exact the same rules.

jimp

That still suggests your rules do not get reply-to for some reason. As above, the usual suspects are:

Using "pass" on port forwards (rdr pass does not support reply-to, so that can't ever work, not something we can change)
Not selecting a gateway on the Interface page for both WAN and WAN2 if using a static IP (this will make the firewall fail to fully treat the interface as a WAN, and skip reply-to)
Checking the box in the advanced options to disable reply-to (Which, as the option implies, removes reply-to from rules)

Post the full contents of config.xml (you can remove keys/passwords) and a copy of /tmp/rules.debug.

If you choose to obscure IP addresses, leave enough to distinguish them (e.g. last octet), and do not remove them entirely.

Marv21

@jimp:

That still suggests your rules do not get reply-to for some reason. As above, the usual suspects are:

Using "pass" on port forwards (rdr pass does not support reply-to, so that can't ever work, not something we can change)
i used pass and no RDR ( The checkbox is unchecked)

Not selecting a gateway on the Interface page for both WAN and WAN2 if using a static IP (this will make the firewall fail to fully treat the interface as a WAN, and skip reply-to)
I have set a gateway for the static and none for the dhcp (will choose it on his own)

Checking the box in the advanced options to disable reply-to (Which, as the option implies, removes reply-to from rules)
No, xD. With Wan2 as default it works…

Post the full contents of config.xml (you can remove keys/passwords) and a copy of /tmp/rules.debug.

If you choose to obscure IP addresses, leave enough to distinguish them (e.g. last octet), and do not remove them entirely.

I will it post later, not at home right at the moment

jimp

If you used "pass" on the port forward, it will not work with any non-default route WAN, and it can't work with "pass".

Change that to "add associated firewall rule" and then save/apply the port forward. Do that for every port forward you have that uses 'pass'

Marv21

@jimp:

If you used "pass" on the port forward, it will not work with any non-default route WAN, and it can't work with "pass".

Change that to "add associated firewall rule" and then save/apply the port forward. Do that for every port forward you have that uses 'pass'

Sry I was not accurate enough. I used pass in the Firewall rule, not block and not reject.

BTW: add associated firewall rule is the default setting, when you create a NAT rule.

NATRULE.png_thumb

jimp

I'll still need to see the config.xml and /tmp/rules.debug. A screenshot of that doesn't tell me enough about what it's doing in the background.

robtoronto

Hey Jim, Ok I fully understand and now agree with you.; I just got in from a 14 hour shift and have been dealing with recovery of failed modules and stacks we had go down lastnight. I am very tired and do not think I can put my full 100% into this right at this moment. I will grab a copy of my current config, but just to note this is a working copy. I would like to get a 'not' working copy to you also. I would love to do the comparison by my self also, The more I think about it without actually doing it is just racking my brain and have also thought about what you said and you are correct, it very well could be a PFSense configuration issue. From a networking perspective I do know that default route could of been causing the issue. But enough guessing , I will upload a working config today to get you started and I cant promise tomorrow night when I get in from work since I'm sure I will still be swamped at work. But when I get up Monday, I will completely re-install from 2.0.1 -> upgrade to 2.0.2 and cause my same issue.

and on a note of multiple users, I have people that I know in person and I have read 1 more post on another site. It was no help, but I will try and find that again for you.

robtoronto

Here is my raw config.xml & /tmp/rules.debug

http://m37offroading.ca/PFSENSE/config.xml

http://m37offroading.ca/PFSENSE/rules.debug

I have also uploaded them on the redmine post. I will get you copies of the failed config.xml and rules.debug by Monday.

jimp

FYI- You only have the port forward in there once. It should be in there twice (once for each WAN). The WAN rules look OK though.

rdr on nfe0 proto tcp from any to 24.212.178.88 port $Cloud -> $Server
pass  in log  quick  on $THOMSONWAN reply-to ( re0 198.84.211.65 )  proto tcp  from any to   $Server port $Cloud  flags S/SA keep state  label "USER_RULE: Allow Server to Cloud on ThomsonWAN"
pass  in log  quick  on $MOTOROLAWAN reply-to ( nfe0 24.212.178.65 )  proto tcp  from any to   $Server port $Cloud  flags S/SA keep state  label "USER_RULE: Allow Server to Cloud on MotorolaWAN"

Efonnes

I saw port forwards for each. However, was it your intention to have your port forwards use two IP addresses for the redirect target IP? (10.10.10.20 and 10.10.10.30) I think using an alias with two (or more) addresses in it like that will make the forwarding cycle through the addresses in the alias on each connection attempt, using a different one each time.

robtoronto

? How could it only be in there once if NAT is working right now on both IP's, and yes. That example that server runs quad NIC's , the 2 there are used for public traffic.

Marv21

So here we go.
This is a RAR with a Working Config and a RAR with a non working config.

I send the password over PM to relevant People, because i dont know that i delete all passwords (or change them in the file to FFF/111)

http://www.marv21.de/RouterConfig.rar

jimp

@robtoronto:

? How could it only be in there once if NAT is working right now on both IP's, and yes. That example that server runs quad NIC's , the 2 there are used for public traffic.

Hmm I must have overlooked it, but now the rules.debug file is 404 so I can't check it again. Though Efonne is right about using multiple IPs in the target of a port forward. It will do round-robin connection alternating, it can't forward the same port to both systems and work for both at the same time.
And if you really want to do load balancing, use the load balancer, not a port forward, since a port forward with two entries would not be smart enough to remove a down host from the pool when it's not in use so half the connections would just fail.

jimp

@Marv21:

So here we go.
This is a RAR with a Working Config and a RAR with a non working config.

I send the password over PM to relevant People, because i dont know that i delete all passwords (or change them in the file to FFF/111)

http://www.marv21.de/RouterConfig.rar

In your "don't work" folder the config.xml has two GW_WAN entries, but otherwise I don't see anything there that looks obviously out of place upon first glance. The rules have reply-to, so the default gateway shouldn't matter for items hitting the port forward. Though one of your WANs is getting double NAT, so that doesn't help, but it shouldn't interfere. USB nics (your ue0) are crappy in general but shouldn't affect it either.

I see you completely reconfigured the two NICs between those two configs, swapping them between "wan" and "opt1". So as it currently is, does the port forward still only work on one WAN? Or does it work on both now?

Marv21

With the "work not" config only the default Gateway works ( in this config the UM WAN ( the dynamic)).

AT all iam fine, i swaped the assignment of the WAN1 and WAN2 (em0 and ue0) and everything is ok.

Maybe i forgot to delete a Wan entrie (if i swaped the assignment there a two dynamic Gateways for UM), but i correct that, but i doenst help.