Multi WAN Strange Routing

dotdash · Apr 29, 2008, 5:46 PM

It would help if you described how the interfaces were configured- is DMZ bridged to WAN2? Also, knowing the firewall rules on the DMZ, as well as the gateway settings of the machine you are tracerouting would be nice.

Wasca · Apr 30, 2008, 4:12 AM

OK here goes

The DMZ interface is on OPT2 it's a /27 (203.xxx.xxx.160) public network. The DMZ Interface is static and is the first usable IP in the /27 network (203.xxx.xxx.161). My WAN2 ISP routes to the DMZ via my WAN2 interface IP.
I'm using Advanced Outbound NAT for my LAN on both WAN1 and WAN2.

WAN2 192.168.1.0/24 * * * * * NO NAT to LAN Subnet
WAN1 192.168.1.0/24 * * * * * NO NAT to LAN Subnet

Rules on the DMZ

Lets just base this on Ping and Traces since that where the issue lies at the moment.

ICMP DMZ net * ! INTNETWORKS * 203.xxx.xxx.153 <=== If Not LAN send out WAN2 Gateway
ICMP DMZ net * * * * <=== If any send out default routing table

Rules on WAN2

ICMP * * DMZ net * * <=== Allow ICMP to DMZ

Interface settings of the server I'm trying to trace to:

iface eth0 inet static
address 203.xxx.xxx.170
netmask 255.255.255.224
network 203.xxx.xxx.160
broadcast 203.xxx.xxx.191
gateway 203.xxx.xxx.161
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 203.xxx.xxx.161 203.xxx.xxx.9 203.xxx.xxx.10

I have no rules on the WAN inteface as I have no need for antything coming in on that interface (only used for LAN clients browsinig the web)

Here are the static routes I have so as to make use of the DNS Servers on the WAN link by my LAN clients

WAN 203.xxx.xxx.35/32 10.xxx.xxx.214 WAN DNS#1 <=== I realise the gateway is a private network this is what the ADSL2+ ISP hands out, not sure why
WAN 203.xxx.xxx.36/32 10.xxx.xxx.214 WAN DNS#2
WAN2 203.xxx.xxx.10/32 203.xxx.xxx.153 WAN2 DNS#2
WAN2 203.xxx.xxx.9/32 203.xxx.xxx.153 WAN2 DNS#1
WAN2 203.xxx.xxx.98/32 203.xxx.xxx.153 Gateway to IPSEC END POINT out WAN2

I noticed these routes in my routing table

default 10.xxx.xxx.214 UGS 0 1492837 1492 ng0
10.xxx.xxx.214 123.xxx.xxx.68 UH 3 4041 1492 ng0
123.xxx.xxx.68 lo0 UHS 0 0 16384 lo0 <=== I have my suspicions of this entry (WAN IP)
127.0.0.1 127.0.0.1 UH 0 0 16384 lo0
203.xxx.xxx.152/29 link#4 UC 0 0 1500 em3 <=== This is the WAN2 subnet
203.xxx.xxx.158 00:xxx:23:xxx:ea:c1 UHLW 1 15 1500 lo0 <=== This is WAN2 IP. This looks sus also!! why the same interface (lo0) as WAN
203.xxx.xxx.160/27 link#2 UC 0 0 1500 em1 <=== This is the DMZ Subnet
203.xxx.xxx.161 00:04:xxx:a5:xxx:f3 UHLW 1 8 1500 lo0 <=== This is the DMZ Interface IP. Why is it lo0?

Ok so now that you have all that, just recap, when I disconnect my WAN (PPPoE) at the interfaces page I can not trace to the DMZ or the WAN2 interface from externally.

It appears to destroy the route to my WAN2 at this route
203.xxx.xxx.158 00:xxx:23:xxx:ea:c1 UHLW 1 15 1500 lo0

I suppose if you bring down this route maybe it destroys the WAN2 interface also at interface lo0?
123.xxx.xxx.68 lo0 UHS 0 0 16384 lo0

I hope some one can help me, let me know if you need anything else

dotdash · Apr 30, 2008, 3:02 PM

Ok, so I didn't read all of that very carefully, but it appears you have the DMZ setup in a rather unusual fashion.
I suspect you problems are due to this. If the DMZ is coming in through WAN2, than it seems the DMZ should be bridged to the WAN…
I would just put a private subnet on the DMZ, add the VIPs from the /27 to WAN2, and NAT the DMZ; but if you want the servers in the DMZ to have public IPs, the usual configuration is to bridge the interface.

hoba · Apr 30, 2008, 4:15 PM

You can't use public IPs in your dmz if you want to work with multiwan. You have to use NAT. The public IPs are known to the internet to be behind one of the wans only. It might work if you use nat on the interface that these public IPs are not behind only.

Wasca · May 1, 2008, 2:02 PM

Maybe I didn't explain my self properly.

My secondary ISP is on WAN2 (OPT2) It's a /29 subnet. My publicly routable network /27 (OPT3 / called DMZ) is routed to my WAN2 IP from my WAN2 ISP.

I'm only Natting the LAN.

Surely I can have a routable public network off my OPT3 interface? Why would I need to bridge my /29 WAN2 network with the /27 public routable DMZ network?

Ok so the question again is why is my WAN IP appearing in the trace route when I'm tracing to a public IP that is routed via my WAN2? Once that is answered, I may then be able to understand why when my WAN (ADSL2+) goes down (and the WAN interface looses it's IP) I can't trace or gain any access to the public IP's on my OPT3 interface from outside of my office

Trace to 203.82.xxx.aaa This is a public IP on my OPT3 network routed via my WAN2 interface IP

1. customer-reverse-entry.64.xxx.xxx.xxx
2. border-core2.sfo2.servepath.com
3. te-3-4.car2.SanFrancisco1.Level3.net
4. ae-4-4.ebr2.SanJose1.Level3.net
5. ae-82-82.csw3.SanJose1.Level3.net
6. ae-34-89.car4.SanJose1.Level3.net
7. te-7-4-71.sntcca1wch010.wcg.Level3.net
8. GigE-15-0.sntcca1wcx1.wcg.Level3.net
9. pos-9-0-oc48.anhmca1wcx3.wcg.level3.net
10. 64.200.249.178
11. anhmca1wct1-powertel-atm.wcg.net
12. ge-0-2-36.cust-gw03.powertel.net.au
13. 203.185.xxx.ccc <====== This is the upstream (next hop/gateway) router on the WAN2 link
14. xxx-xxx-xxx-68.static.tpgi.com.au <===== This is my ADSL2+ WAN IP (WHY IS THIS NOT MY WAN2 IP??)
15. 203.82.xxx.aaa <====== This is the public IP on the OPT3 network

I pose this question, Could it be that I need to add a Virtual IP like this below on my WAN2?

Type: Proxy Arp
Interface: WAN2
IPAddress(s): Network
203.82.xxx.xxx/27
Description: Proxy Arp for /27 Network on OPT3

or do I use CARP?

Type: Carp
Interface: WAN2
IPAddress(s): Network
203.82.xxx.xxx/27
VHID Group: 1
Advertising Frequency: 0
Description: CARP for /27 Network on OPT3

What do you think?

dotdash · May 1, 2008, 2:57 PM

@Wasca:

Maybe I didn't explain my self properly.

My secondary ISP is on WAN2 (OPT2) It's a /29 subnet. My publicly routable network /27 (OPT3 / called DMZ) is routed to my WAN2 IP from my WAN2 ISP.

I got that part of it. Maybe I did not explain myself properly. Let me put this way- you're doing it wrong. You are slapping an additional subnet coming in via WAN2 on a separate interface, not bridged to the interface the block is routed from, and expecting everything to work smoothly in mult-wan.
Perhaps this sounds short, but people are trying to give you some helpful advice, and it seems like you don't want to hear it. What you are trying may be possible, but it is not a normal or supported configuration. We were trying to point you toward known working methods- if you want to go your own way, it will limit the amount of help others are able to provide.

Wasca · May 2, 2008, 12:52 AM

You are slapping an additional subnet coming in via WAN2 on a separate interface, not bridged to the interface the block is routed from, and expecting everything to work smoothly in mult-wan.

So are you saying you can't route public IP networks to other interfaces in a multi-wan setup, only if you have a single WAN? that sounds very strange.

Ok, So I did a little more searching and found this reply below from this thread http://forum.pfsense.org/index.php/topic,8990.0.html

Maybe you need to clarify first:
Does this /29 subnet get routed by your ISP to the public IP you have on your WAN?

Since you have a public /29 subnet there are multiple approaches:

1: You bridge the OPT1, on which your public IP's are used, to it's WAN.

2: You create virtual IP's on your WAN, use private addresses in your OPT1 and just forward the traffic you need from the VIP's to your private IP's.

3: You route your public IP's. though with only a /29 you would waste one of your 6 IP's and you'd be left with only 5.
(This only works of you have another public IP on WAN that's not within this /29

The first approach is maybe the best if you want the public IP's directly on you servers, and your subnet does NOT get routed to a public address on your WAN. Downside is you use one of your public IP's up for the WAN.
(You still can create NAT-forwardings from the WAN-address to computers in your personal LAN but that's probably not what you want.)

The second approach lets you use all the IP's out of your /29 subnet –> you can forward ports from all 6 IP's.
The downside is that your servers use private IP's which migh crete problems for certain setups.

The third approach would be imo the most clean approach. But it only works if your ISP routes your /29 to another public IP you have on the WAN. You disable NAT for your server-subnet and just route the IP's Smiley

If I was being asked this question, I would answer….

Yes my /27 is routed by my WAN2 ISP. My WAN2 interface IP is 203.82.xxx.aaa and on the WAN2 ISP's router they have a route to my /27 subnet via 203.82.xxx.aaa. The WAN2 interface IP of 203.82.xxx.aaa exists on it's own /29 subnet (The network between my PFSense box and the WAN2 ISP)

I want to do option 3 and this is currently what is setup and working. The problem occurs when my WAN (ADSL2+) connections drops offline I can't access the /27 subnet from externally, but servers on the OPT3 network can still ping out via WAN2 just fine. This can be explained by the first traceroute at the top of this thread (see the trace getting a response from the WAN IP not the WAN2 IP)

I apologise if I'm sounding stubborn but I can't see how Bridging my OPT3 with WAN2 (or the other way round, what ever it is) is the correct method when trying to route a public network, hence why option 3 above is what I currently use and is working, just not when my WAN goes down.

Tell me, does option 3 simply not work when you have more than 1 WAN? is this a limitation of PFSense, I would be highly surprised if it was.

GruensFroeschli · May 2, 2008, 2:27 PM

I'm not sure if i understood your setup correctly.
(Screenshots and not text of the rules would help)

Since you want to route your DMZ through WAN2.
Did you create a rule on the DMZ tab that has as gateway "WAN2" and not *
Did you remove all AoN rules related to your DMZ (you dont want to NAT it)?

I would try to get this working step by step.

1: Have both WAN's up. i assume this is working

2: Get LAN working. what do you want here? loadbalancing the LAN subnet to WAN1 and WAN2?
I see that you already created 2 AoN rules that NAT the LAN subnet to WAN1 and WAN2.

3: Set up the Addressing in the DMZ. Make sure that the client in the DMZ use the pfSense DMZ-interface IP as gateway.

4: assure that there are NO AoN rules for the DMZ.

5: create firewall-rules on DMZ: Assure that the rule allowing traffic on the DMZ uses as gateway the WAN2 (the reply should go back to where the request came from).

6: create firewall-rules on WAN2: allow as "destination your DMZ" access depending on your needs.

Like this should now:

Traffic from the LAN will be NATed to the WAN or WAN2 ip (depends on what you set up)
Traffic from the DMZ will not be NATed, but routed through the WAN2.
Traffic comming in on the WAN2 destined to the DMZ will reach the DMZ.

Maybe you could show screenshots of your rules after the changes you did?
(text is kind of unhandy).

Wasca · May 2, 2008, 3:20 PM

Since you want to route your DMZ through WAN2.
Did you create a rule on the DMZ tab that has as gateway "WAN" and not *
Did you remove all AoN rules related to your DMZ (you dont want to NAT it)?

All rules point to the WAN2 as the gateway.
I only have AoN rules for my LAN (and other natted private subnets) see attached image

1: Have both WAN's up. i assume this is working

Yes both WAN's are up and working with traffic going across them, interface settings attached

2: Get LAN working. what do you want here? loadbalancing the LAN subnet to WAN1 and WAN2?
I see that you already created 2 AoN rules that NAT the LAN subnet to WAN1 and WAN2.

LAN is working, No Load balancing or failover needed (trying to keep it simple to begin with).
I do have some port forward rules on the WAN2 and WAN pointing to PC's on the LAN (used for VNC Listen sessions)

3: Set up the Addressing in the DMZ. Make sure that the client in the DMZ use the pfSense DMZ-interface IP as gateway.

DMZ interface attached.

This is done on all servers, see below

iface eth0 inet static
address 203.xxx.xxx.170
netmask 255.255.255.224
network 203.xxx.xxx.160
broadcast 203.xxx.xxx.191
gateway 203.xxx.xxx.161 <=====THIS IS THE OPT3 (DMZ) interface IP
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 203.xxx.xxx.161 203.xxx.xxx.9 203.xxx.xxx.10 <==== IM USING THE OPT3 INTERFACE AS MY FIRST DNS SERVER

4: assure that there are NO AoN rules for the DMZ.

Nope, none there, I only apply these rules to networks I want natted like the LAN. see attached image

5: create rules on DMZ: Assure that the rule allowing traffic on the DMZ uses as gateway the WAN2 (the reply should go back to where the request came from).

I have all the rules point to the gateway on WAN2, except where the traffic from the DMZ needs to go the LAN or other private networks on the PFSense Router then I use * as the gateway. See attached image.

6: create rules on WAN2: allow as "destination your DMZ" access depending on your needs.

This has been done see attached picture.

Like this should now:

Traffic from the LAN will be NATed to the WAN or WAN2 ip (depends on what you set up)

Traffic from the DMZ will not be NATed, but routed through the WAN2.

Traffic comming in on the WAN2 destined to the DMZ will reach the DMZ.

The setup is like this and has been from the begining.

As I have said in my previous posts, tracing out looks correct and works

This is a trace from one of my DMZ servers 203.xxx.xxx.171

Host Loss% Snt Last Avg Best Wrst StDev
1. 203.xxx.xxx.153 0.0% 17 2.6 2.5 2.3 3.0 0.2 <==== This is my WAN2 Gateway IP
2. 203.xxx.xxx.185 0.0% 17 2.5 2.8 2.4 4.8 0.7
3. qbrisbrdr01-ge01.powertel.net.au 0.0% 17 3.4 3.5 3.1 5.9 0.7
4. Ve1.rq-127creek-core-02.pipenetworks.com 11.8% 17 3.8 844.4 3.5 5772. 1846.
5. Fast0-0-1203.rn-400harris-core-01.pipenetworks.com 0.0% 17 16.0 16.4 15.3 18.7 0.8
6. pyro-xxx.xxxx.com 0.0% 16 16.1 16.7 15.7 21.6 1.4
7. cms1-xxx.xxxx.com 0.0% 16 16.4 16.6 15.7 17.6 0.5

And this is the trace back from the other end

Trace to 203.xxx.xxx.171
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 203.161.136.81 0.0% 7 0.2 0.2 0.1 0.2 0.0
2. RN-400HARRIS-CORE-01.pipenetworks.com 0.0% 7 1.0 49.0 0.8 171.1 81.9
3. Ve4.rq-148brunswick-core-01.pipenetworks.com 16.7% 6 13.3 19.8 13.1 45.6 14.4
4. AS9837.brisbane.pipenetworks.com 0.0% 6 13.5 65.8 13.2 173.0 81.0
5. qbriscust01-ge02.powertel.net.au 0.0% 6 14.1 26.0 14.1 52.2 18.2
6. 203.xxx.xxx.186 0.0% 6 169.3 83.2 14.6 180.2 79.2
7. 123-xxx-xxx-68.static.tpgi.com.au 0.0% 6 50.3 69.3 16.1 257.2 94.0 <==== THIS IS THE WAN IP OF MY PFSense (IT SHOULD BE THE WAN2 IP)
8. 203.xxx.xxx.171 0.0% 6 16.1 16.2 15.8 17.3 0.5

Now the dilemma occurs when the WAN goes down on my PFSense. Imangine taking out hop #7 from the trace above, I now no longer can get access to the DMZ Servers.

I hope this clears up what I have set-up and what problems I'm having.

Does anyone else out there have 2 WANs and a routable network on an optional interface that can test this for me?

What I need tested is tracing in and out of a server on a public routable network hanging off an Optional Interface, make sure the trace goes in and out the same WAN (preferable your secondary WAN)

Thanks

Let me know if you need any more info, I'm very interested to find an answer to this.

![DMZ Rules.gif](/public/imported_attachments/1/DMZ Rules.gif)
![DMZ Rules.gif_thumb](/public/imported_attachments/1/DMZ Rules.gif_thumb)

![WAN Interface.gif](/public/imported_attachments/1/WAN Interface.gif)
![WAN Interface.gif_thumb](/public/imported_attachments/1/WAN Interface.gif_thumb)
![WAN2 Interface.gif](/public/imported_attachments/1/WAN2 Interface.gif)
![WAN2 Interface.gif_thumb](/public/imported_attachments/1/WAN2 Interface.gif_thumb)
![DMZ Interface.gif](/public/imported_attachments/1/DMZ Interface.gif)
![DMZ Interface.gif_thumb](/public/imported_attachments/1/DMZ Interface.gif_thumb)

Wasca · May 4, 2008, 11:57 PM

I was wondering if any of the devs could take another look at this, as I believe there could be a bug causing this issue.

I have complied with the appropriate setup for this kind of network and this is still causing me issues.

If this can't be resolved over the forum, I'm planning on swapping the WAN's so that the backup ADSL2+ connection becomes WAN2 and I use a half bridged setup.

Again, thanks for all your help so far, I really hope we can work this one out, as I don't believe I'm doing anything out of the ordinary.

Thanks