2x2 WAN - 2 PPPoE Providers - 2 pfSense Boxes

alfredo

Dear Forum, we have

2 PPPoE Providers
2 pfSense doing CARP internally, not externally.

We have figured out that we cam implement load-balancing and failover, like http://www.tecmint.com/how-to-setup-failover-and-load-balancing-in-pfsense/2/ , but this only works for one box. ;)

Question: How can we make box2 grab both PPPoE connections if box1 fails? Of course, box2 may only have these connections if box1 has indeed failed. box1 and box2 are setup as CARP Cluster with a SYNC link in between.

Thanks for any advice, Alfredo

alfredo

Hallo; anybody???

jimp

You can't do that properly with PPPoE handled on pfSense, not unless your ISP lets both boxes login at the same time on both lines (or if you actually have four lines).

PPPoE and CARP do not mix well. You're almost better off allowing the modems to handle PPPoE and doing proper CARP on small private subnets between the modem and the pfSense WAN(s). Setup 1:1/DMZ in the modem to the CARP VIP. Not pretty since it involves an extra layer of NAT, but you'd end up with more stable failover and multi-WAN for most cases.

alfredo

Hi Jimp,

Thanks for your response. Yeah; am not to keen on Double NAT. Indeed, the a DMZ behind our Vigor 130 Modems would still be behind a 'forced' NAT. I have found another solution; what do you think?

START/STOP PPPoE on Second BOX:


Browse to /etc/rc.carpbackup 
at the bottom of the file type the following: 

	mwexec_bg("ifconfig igb0 down");
	mwexec_bg("ifconfig igb5 down");
	log_error("### CARP BACKUP :: Stopped PPPoE on WAN1/igb0 and WAN2/igb5 :: GAASTRA ###");

?>

Browse to /etc/re.carpmaster 
at the bottom of the file type the following:

	mwexec_bg("ifconfig igb0 up");
	mwexec_bg("ifconfig igb5 up");
	mwexec_bg("/usr/local/sbin/pfSctl -c 'interface reload igb0'");
	mwexec_bg("/usr/local/sbin/pfSctl -c 'interface reload igb5'");
	log_error("### CARP MASTER :: Restarted PPPoE on WAN1/igb0 and WAN2/igb5 :: GAASTRA ###");

?>

It kinda seems to work; but am not quite sure yet…

What do you think

Thanks

Alfredo

jimp

Anything short of static IP addresses on the WANs with HA is asking for trouble. You can hack it up all you want but it won't ever be truly stable with such a configuration.

alfredo

Thanks; so you think double NAT/DMZ is better than starting/stopping PPPoE in such case? We are using all kinds of IPSec and OpenVPN Tunnels…

Would there be any way to make this 'hack' more elegant?

Thanks

Alfredo

jimp

The only way to turn that type of setup into one compatible with CARP is as I mentioned – using static private addresses on pfSense and doing the PPPoE login on the CPE.

Some providers will even route you a /29 or so over PPPoE, if the modem can handle that as well then you'd be perfect. Probably an extra cost for the addresses but it's the only way to do that with PPPoE and avoid NAT.

alfredo

Just communicated with the PPPoE Provider and they mentioned that the next plan up - with more than 1 IP Address - is 10 times the price… ;D

The Problems with double NAT I think are:

Modem has very SLOW CPU - about 1/100th compared to our Huge (8 core i7 - 16GB) pfSense boxes. We think that it might bog down our many IPSec and OpenVPN tunnels. The Manufacturer - DrayTek - mentioned this to us over the phone and recommended to only use it in bridge-mode.
Wouldn't double NAT add another layer of data packet header alteration; which would also permanently slow down our tunnels.
Am not 100% sure, but would this extra NAT layer also affect data packet size with resulting permanent consequences because of stripping?
There is also a significant amount VOIP traffic behind our pfSense boxes.

As such, I was wondering, if you could look at the above code snippets once more, and conclude that they are not too bad for this setup considering all the disadvantages of double NAT. Or would you have some comments/improvements/suggestions to the snippets better.

One thing I am sure of: many-many pfSense fans are facing this same issue. ;)

Thanks,

Alfredo

jimp

That's unfortunate. As I said though, you're welcome to hack things up however you want, but personally I wouldn't trust it in such a configuration and it's not something I want to get involved in.

alfredo

Alright; we have tried the official CARP Solution for our 2x2 WAN taking double NAT as a drawback.
We have setup 3 Gateway groups as shown in most documents: WAN1 and WAN2 Load Balancing (member down), If WAN1 fails, switch to WAN2 (Packet Loss), and If WAN2 fails, switch to WAN1 (Packet Loss). At each of our 3 LANs, we have set the bottom 3 rules to match above groups. We have set gateway monitoring to 8.8.8.8 for WAN1 and 8.8.4.4 for WAN2.

Now comes the problem. When we unplug WAN1, the DNS does not work anymore for the clients. Surfing by IP address does work. So it must be a DNS issue. Could you tell us what is wrong?

System Log Below:

Jan 13 18:59:45 fw1 php-fpm[55656]: /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.
Jan 13 18:59:45 fw1 php-fpm[55656]: /rc.newipsecdns: MONITOR: GW_WAN1 is down, omitting from routing group WAN1_AND_WAN2
Jan 13 19:00:36 fw1 php-fpm[47415]: /rc.dyndns.update: MONITOR: GW_WAN1 is down, omitting from routing group WAN1_TO_WAN2
Jan 13 19:00:37 fw1 php-fpm[55787]: /rc.filter_configure_sync: MONITOR: GW_WAN1 is down, omitting from routing group WAN2_TO_WAN1
Jan 13 19:00:51 fw1 php-fpm[55656]: /rc.newipsecdns: Could not send the message to it-helpdesk@telekom.de – Error: could not connect to the host "smtp.office365.com": ??
Jan 13 19:00:51 fw1 php-fpm[55656]: /rc.newipsecdns: MONITOR: GW_WAN1 is down, omitting from routing group WAN1_TO_WAN2
Jan 13 19:01:42 fw1 php-fpm[47415]: /rc.dyndns.update: Could not send the message to it-helpdesk@telekom.de – Error: could not connect to the host "smtp.office365.com": ??
Jan 13 19:01:42 fw1 php-fpm[47415]: /rc.dyndns.update: MONITOR: GW_WAN1 is down, omitting from routing group WAN2_TO_WAN1
Jan 13 19:01:43 fw1 php-fpm[55787]: /rc.filter_configure_sync: Could not send the message to it-helpdesk@telekom.de – Error: could not connect to the host "smtp.office365.com": ??
Jan 13 19:01:43 fw1 php-fpm[57451]: /rc.filter_configure_sync: MONITOR: GW_WAN1 is down, omitting from routing group WAN1_AND_WAN2
Jan 13 19:01:57 fw1 php-fpm[55656]: /rc.newipsecdns: Could not send the message to it-helpdesk@telekom.de – Error: could not connect to the host "smtp.office365.com": ??
Jan 13 19:01:57 fw1 php-fpm[55656]: /rc.newipsecdns: MONITOR: GW_WAN1 is down, omitting from routing group WAN2_TO_WAN1
Jan 13 19:02:48 fw1 php-fpm[47415]: /rc.dyndns.update: Could not send the message to it-helpdesk@telekom.de – Error: could not connect to the host "smtp.office365.com": ??
Jan 13 19:02:49 fw1 php-fpm[57451]: /rc.filter_configure_sync: Could not send the message to it-helpdesk@telekom.de – Error: could not connect to the host "smtp.office365.com": ??
Jan 13 19:02:49 fw1 php-fpm[57451]: /rc.filter_configure_sync: MONITOR: GW_WAN1 is down, omitting from routing group WAN1_TO_WAN2
Jan 13 19:03:03 fw1 php-fpm[55656]: /rc.newipsecdns: Could not send the message to it-helpdesk@telekom.de – Error: could not connect to the host "smtp.office365.com": ??
Jan 13 19:03:03 fw1 php-fpm[55656]: /rc.newipsecdns: MONITOR: GW_WAN1 is down, omitting from routing group WAN1_AND_WAN2
Jan 13 19:03:56 fw1 php-fpm[57451]: /rc.filter_configure_sync: Could not send the message to it-helpdesk@telekom.de – Error: could not connect to the host "smtp.office365.com": ??
Jan 13 19:03:56 fw1 php-fpm[57451]: /rc.filter_configure_sync: MONITOR: GW_WAN1 is down, omitting from routing group WAN2_TO_WAN1
Jan 13 19:04:09 fw1 php-fpm[55656]: /rc.newipsecdns: Could not send the message to it-helpdesk@telekom.de – Error: could not connect to the host "smtp.office365.com": ??
Jan 13 19:04:09 fw1 php-fpm[55656]: /rc.newipsecdns: MONITOR: GW_WAN1 is down, omitting from routing group WAN1_TO_WAN2
Jan 13 19:05:02 fw1 php-fpm[57451]: /rc.filter_configure_sync: Could not send the message to it-helpdesk@telekom.de – Error: could not connect to the host "smtp.office365.com": ??
Jan 13 19:05:11 fw1 php-fpm[55656]: /rc.newipsecdns: Could not send the message to it-helpdesk@telekom.de – Error: could not connect to the host "smtp.office365.com": ??
Jan 13 19:05:11 fw1 php-fpm[55656]: /rc.newipsecdns: MONITOR: GW_WAN1 is down, omitting from routing group WAN2_TO_WAN1
Jan 13 19:06:17 fw1 php-fpm[55656]: /rc.newipsecdns: Could not send the message to it-helpdesk@telekom.de – Error: could not connect to the host "smtp.office365.com": ??
Jan 13 19:06:17 fw1 php-fpm[55656]: /rc.newipsecdns: MONITOR: GW_WAN1 is down, omitting from routing group WAN1_AND_WAN2

Screen-Shot-2016-01-13-at-20.06.png

alfredo

This has helped us solve the problem:

TIPS:
[1] Diagnostics/Misc
Go to Status -> Interfaces
Go to Diagnostics -> Routes
https://forum.pfsense.org/index.php?topic=43982.15

In System -> General Setup
a. Uncheck:
Allow DNS server list to be overridden by DHCP/PPP on WAN
REF: https://forum.pfsense.org/index.php?topic=43982.0
b. Try setting "Use gateway" to none for the Google DNS servers

[2]
DNS Resolver -> Set "Enable Forwarding Mode:" to true:
Controls whether Unbound will query root servers directly (unchecked, disabled) or if queries will be forwarded to the upstream DNS servers defined under System > General or those obtained by DHCP/PPPoE/etc (checked, enabled). Forwarding mode may be enabled if the upstream DNS servers are trusted and also provide DNSSEC support. Forwarding mode is necessary for Multi-WAN configurations unless default gateway switching is enabled.
REF: https://doc.pfsense.org/index.php/Unbound_DNS_Resolver

[3]
System -> Advanced -> Miscellaneous -> Load Balancing
Set Enable default gateway switching to true:
If the default gateway goes down, switch the default gateway to another available one. This is not enabled by default, as it's unnecessary in most all scenarios, which instead use gateway groups.
REF: https://forum.pfsense.org/index.php?topic=72445.0
REF: https://forum.pfsense.org/index.php?topic=45081.0