Noteable things when doing Dual WAN (Policy Routing).

Aussie_Bear · Nov 30, 2005, 9:51 PM

Using pfSense version 0.95c

Test setup for pfSense box…

Celeron 1.2Ghz
512MB RDRAM
i820 chipset mobo (ASUS P3C-D)
3x Intel NICs (i82559 chipset)
CD-ROM
Floppy
pfSense 0.95 LiveCD
ISP1 : Telstra Broadband Cable (10Mbit/128k)
ISP2 : Optus Cable (10Mbit/256k)

fxp0 => LAN
fxp1 => WAN
fxp2 => OPT1 (re-designated as WAN2)

WAN => Telstra Cable (due to bpalogin being needed) => BigPond (DHCP)
WAN2 => Optus Cable => DHCP
LAN => Static IP (labelled as 192.168.1.1)

IP of PC 1 on the LAN side => 192.168.1.10
IP of PC 2 on the LAN side => 192.168.1.12

I point PC 1 to WAN (Telstra) and PC 2 to WAN2 (Optus)

Network Layout

WAN (Telstra) WAN2 (Optus)
\ /
pfSense
|
8-Port Switch
| |
PC 1 PC 2

My settings…

For Firewall => NAT Settings…
I've checked Enable advanced outbound NAT in the Outbound section.

Interface Source Destination Destination Port NAT Address NAT Port Description
WAN 192.168.1.0/24 * * * * For Telstra
WAN2 192.168.1.0/24 * * * * For Optus

For Firewall => Rules Settings…

Proto Source Port Destination Port Gateway Description
* 192.168.1.10 * * * * PC 1 -> Telstra
* 192.168.1.12 * * * WAN2 PC 2 -> Optus

ISSUES in regards to using Dual WAN (Policy Routing)

(1) I noticed that the ISP DNS servers of WAN2 are showing up on WAN…
Is there a way to manually force each connection to use specific DNS servers of that ISP?

It doesn't seem to be any problems, it just looks a little odd, that's all.

(2) When PC 2 (IP : 192.168.1.12) is using the Internet via WAN2,
I noticed that there is a noticeable amount of "Collisions" on this interface.

ie :

WAN
In/out packets : 132510/61432 (45.25 MB/3.38 MB)
In/out errors : 0/0
Collisions : 0

WAN2
In/out packets : 122488/11723 (24.79 MB/1.61 MB)
In/out errors : 0/0
Collisions : 1543

It seems that it does affect the performance slightly, and there's a slight delay
before loading webpages and stuff. Is this because of (1) ?

Its odd that only WAN2 has this issue.
WAN seems fine…Not a single collision.

Everything else seems OK, as I'm using this pfSense box like I use M0n0Wall
(Nothing fancy, just simple firewall/router solution).

Anyone else experience the same issue?

Aussie_Bear · Dec 1, 2005, 12:10 AM

Answer to my Question (2)…

After plugging in a M0n0Wall box, I checked the number of Collisions...It turns out to be MORE than pfSense!
So after an hour of testing, its determined that my old cable modem on the Optus line is the cause of the Collisions
issue, NOT pfSense. (The modem runs at 10Mbit at half duplex...This is because of hardware!)

It seems I need to replace my cable modem with a more recent one like the Motorola SB51xx series.
(These run at full duplex and are rated for 100Mbit, but the ISP restricts them to their marketed price plans.)

But I'm not sure why the DNS server changes...Sometimes it takes it from WAN1, while a time later, it takes it
from WAN2.

ZGamer · Dec 1, 2005, 12:21 AM

I would hard code the primary dns server from each isp into the pfsense box to use for distribution…..should fix the dns problem.....else you need to have a separate network to keep it completely separate as it is only capable of handing out one set of dns server lists per ip-range.

Aussie_Bear · Dec 1, 2005, 6:47 AM

Hmmm…How do I go about doing that?

(I'm FreeBSD newbie, just started going through FreeBSD
Handbook...How come Linux distros don't have something
as detailed as this?).

The DNS issue is not technically a major problem, as you
can still connect on both ends without trouble.

Are there any security (or other) implications to this DNS
server swapping between WAN1 and WAN2?

Other than that DNS oddity, everything else seems fine.

I guess I should inform Scott and Co. of this as a minor bug. (???)

RoboK · Dec 1, 2005, 7:44 AM

@Aussie_Bear:

Using pfSense version 0.95c

Test setup for pfSense box…

Celeron 1.2Ghz

512MB RDRAM

i820 chipset mobo (ASUS P3C-D)

3x Intel NICs (i82559 chipset)

CD-ROM

Floppy

pfSense 0.95 LiveCD

ISP1 : Telstra Broadband Cable (10Mbit/128k)

ISP2 : Optus Cable (10Mbit/256k)

fxp0 => LAN
fxp1 => WAN
fxp2 => OPT1 (re-designated as WAN2)

WAN => Telstra Cable (due to bpalogin being needed) => BigPond (DHCP)
WAN2 => Optus Cable => DHCP
LAN => Static IP (labelled as 192.168.1.1)

IP of PC 1 on the LAN side => 192.168.1.10
IP of PC 2 on the LAN side => 192.168.1.12

I point PC 1 to WAN (Telstra) and PC 2 to WAN2 (Optus)

Network Layout

WAN (Telstra) WAN2 (Optus)
\ /
pfSense
|
8-Port Switch
| |
PC 1 PC 2

My settings…

For Firewall => NAT Settings…
I've checked Enable advanced outbound NAT in the Outbound section.

Interface Source Destination Destination Port NAT Address NAT Port Description
WAN 192.168.1.0/24 * * * * For Telstra
WAN2 192.168.1.0/24 * * * * For Optus

For Firewall => Rules Settings…

Proto Source Port Destination Port Gateway Description
* 192.168.1.10 * * * * PC 1 -> Telstra
* 192.168.1.12 * * * WAN2 PC 2 -> Optus

ISSUES in regards to using Dual WAN (Policy Routing)

(1) I noticed that the ISP DNS servers of WAN2 are showing up on WAN…
Is there a way to manually force each connection to use specific DNS servers of that ISP?

It doesn't seem to be any problems, it just looks a little odd, that's all.

(2) When PC 2 (IP : 192.168.1.12) is using the Internet via WAN2,
I noticed that there is a noticeable amount of "Collisions" on this interface.

ie :

WAN
In/out packets : 132510/61432 (45.25 MB/3.38 MB)
In/out errors : 0/0
Collisions : 0

WAN2
In/out packets : 122488/11723 (24.79 MB/1.61 MB)
In/out errors : 0/0
Collisions : 1543

It seems that it does affect the performance slightly, and there's a slight delay
before loading webpages and stuff. Is this because of (1) ?

Its odd that only WAN2 has this issue.
WAN seems fine…Not a single collision.

Everything else seems OK, as I'm using this pfSense box like I use M0n0Wall
(Nothing fancy, just simple firewall/router solution).

Anyone else experience the same issue?

Thanx for very nice and lucid exampe of working dual wan.
;)

ZGamer · Dec 1, 2005, 6:19 PM

In the general settings you can enter the dns server ip addresses…then just uncheck the box for allowing to override.

Aussie_Bear · Dec 2, 2005, 4:11 AM

@RoboK:

Thanx for very nice and lucid example of working dual wan.
;)

No problem. I thought, I might as well do this if it helps the pfSense project in
general. (I've already helped out in testing the Telstra login part).

I've refined the example into a guide over here…

GUIDE : Multi-WAN Optus and Telstra Cable with pfSense.
http://forums.techwatch.com.au/viewtopic.php?t=4802

It just has more details compared to Dan's guide.

@ZGamer:

In the general settings you can enter the dns server ip addresses…then just uncheck the box for allowing to override.

Thanks ZGamer, I'll try that and see how things pan out.
(I thought I had to do some command line stuff…Which I don't really mind doing).

Aussie_Bear · Dec 2, 2005, 8:32 AM

I did what you said ZGamer, but I don't recommend it, if you're using Telstra Cable (Australia Only).

I recommend either putting WAN 1's (Telstra) DNS server first OR leave it using DHCP override,
because you won't be able to login as pfSense will take the other one and assign an IP from WAN 2
and try to put it as WAN 1 !

ZGamer · Dec 2, 2005, 6:05 PM

Correct it will end up given a little bit extra load to one wan interface over the other with the extra dns queries. Ideally I guess you would use pfsense for your dns server and have pfsense cache everything and determine it that way.

Aussie_Bear · Dec 4, 2005, 3:16 AM

One odd thing I've just encountered, is that WAN 2 (OPT 1) is not able to connect to FTP servers.
I always get a "time out". I'm using Firefox web browser to view these FTP servers.

I tried FreeBSD, OpenBSD, Slackware, Debian, etc sites. (Official download link and various mirrors
around the world for each project). All "time out".

To make sure it isn't my connection, I connected a M0n0Wall box to it, and I was able to access FTP!
I double checked by using a Linksys WRT54G router (with third-party Linux firmware installed), and had
no problems with FTP.

I've tried enabling and disabling FTP-Helper. As well, I've opened up ports and such…It did nothing, as
I would still get "time outs". (I've sent all logs via Syslog to a PC on the LAN side, but I don't see any
pf rules blocking FTP connections).

Do any of you folks get the same problem?