Tutorial: HA WAN Failover DHCP with modem and only one available DHCP Lease

[BJ55463]

Hi,

I have 2 pfSense firewalls in HA setup. Uplink is only one modem (in bridge mode) but the modem has an integrated 4-port-switch on lan side. I can only obtain one ip-address via DHCP from the provider. So it is not possible to have the WAN interface of both firewalls active as DHCP client. I didn't found a working and good solution for this problem. So I combined some solutions and added a few commands. I found a few people with the same problem so I want to share my solution.

1. get the interface which is connected to the modem. In my case opt11
2. get the carp vip id for the lan interface (or another interface which should be monitored and which will be used to decide which firewall becomes the wan ip). In my case 2
3. get the real interface for the monitored lan interface. In my case igb1.2510

Setup WAN interface as DHCP for both firewalls. Disconnect wan on the backup firewall. Also set WAN MAC of the second firewall the same as WAN on master firewall.

On both firewalls use Diagnostic -> Command Prompt

run "mkdir /usr/local/etc/devd"

On both firewalls use Diagnostic -> Edit File

edit file "/usr/local/etc/devd/carp.conf" and insert

notify 200 {
  match "system" "CARP";
  match "type" "MASTER";
  action "/usr/local/sbin/pfSctl -c 'interface carpmaster $subsystem';/usr/local/sbin/pfSsh.php playback carpmasterwan '$subsystem'";
};
notify 200 {
  match "system" "CARP";
  match "type" "BACKUP";
  action "/usr/local/sbin/pfSctl -c 'interface carpbackup $subsystem';/usr/local/sbin/pfSsh.php playback carpbackupwan '$subsystem'";
};

now edit file "/etc/phpshellsessions/carpbackupwan"

require_once("config.inc");
require_once("util.inc");
require_once("service-utils.inc");

global $argv;

if( $argv[3] == '$2@igb1.2510' ) {

  interface_bring_down("opt11");
  restart_interface_services("opt11");
  filter_configure();
  system_routing_configure();
  send_event("service reload packages");

  $extras['vpnmode'] = 'client';
  $extras['id'] = 1;
  service_control_stop('openvpn', $extras);

}

Don't forget to change $2@igb1.2510 with 2. and 3. from the notes. For example carp vip 5 and interface igb2 would be $5@igb2
Also change opt11 with 1. from the note (the wan interface)

In my case I have also an openvpn client which should only run if the wan interface should be active. So if you also have one openvpn client bind to wan you have to change the 1 from $extras['id'] = 1; to your openvpn client id. Otherwise you can remove

  $extras['vpnmode'] = 'client';
  $extras['id'] = 1;
  service_control_stop('openvpn', $extras);

Next edit file "/etc/phpshellsessions/carpmasterwan"

require_once("config.inc");
require_once("util.inc");
require_once("service-utils.inc");

global $argv;

if( $argv[3] == '$2@igb1.2510' ) {

  interface_configure("opt11");

  $extras['vpnmode'] = 'client';
  $extras['id'] = 1;
  service_control_restart('openvpn', $extras);

}

Also change $2@igb1.2510 and opt11 and also the vpn part

So now on both firewalls use Diagnostic -> Command Prompt

service devd restart

Now when the LAN CARP IP goes to backup the WAN connection goes down and the openvpn client stops. When the Ip goes to master then the interface will reconnect and the openvpn client starts.

Now the firewall in backup state has no internet connection. For this I have setup the other firewall ip from the CARP network (where the sync traffic goes trough) as a gateway. Then I added a failover group where WAN DHCP Gateway is Tier 1 and then Firewall Ip of the other box on the CARP interface as Tier 5. As trigger I setup "member down". As default gateway I setup the failover group. Now the backup firewall can use the internet without problems. Don't forget to allow the needed traffic from the firewall through the carp interface like pfsense updates. Also do this on the other firewall if this one will go to backup mode.

I hope it would help someone.

Best regards,

Alex

[Gabri.91]

Hi @BJ55463 I had a similar configuration but looks like after 2.7.0 upgrade, VIP IP used as gateway is no more reachable from slave node.

Have you noticed the same?

[BJ55463]

@Gabri-91 Didn't see this problem. But I used this on pfsense+.

Perhaps you can tell me a little bit more about your problem. Perhaps I can help you to solve the problem.

[Gabri.91]

@BJ55463 basically on gateway side I had the same configuration as your, so a gateway group with tieri 1 my WAN gateway and tier 2 my CARP IP, so the slave pfSense was able to reach internet for updates.

After upgrading to 2.7.0 (and ESXi to a minor 7.0.3 patch, don't think it's related), slave pfSense is unable to ping the CARP IP of the main pfSense.

Main is 192.168.5.252, Slave is 192.168.5.254, CARP is 192.168.5.254

This is the gateway

And here on the slave it's offline

Ping not working

Haven't changed any firewall rule so I'd exclude something on that side.

ARP on the slave looks good

Same issue with another interface

Don't know what else I can check TBH. Issue looks to be related to CARP rather than gateway "trick".

[BJ55463]

@Gabri-91 I used the CARP Interface IP of the other pfsense box as TIER 2, not the "modem" interface IP.

First check if CARP Interface Firewall allows Traffic
Second, try to ping each other box over carp interface
Third, take a look at your NAT rules

Also check ESXi vSwitch config. For a few days, I also had problems with ESXi. Promiscuous Mode wasn't enabled anymore on vSwitch for the "modem" interface. But in this case, it was related to HAProxy on a CARP IP.

In my case, I only used physical devices as pfsense firewall, no vms. So I cannot reproduce issues related to vms and esxi.