PFSense router failover redundancy - CARP Virtual IP not Reachable
-
Hello,
Environment
I am having a problem with an PFSense failover redundancy that I trying to implement in a lab environment.
Here is my layout of my environment:
I have to two PFSense devices each configured with a WAN interface on the same subnet as above /29 subnet (69.31.162.48/29).
I assign/bind the em1 interface on each PFSense box with one of the useable IP address in the subnet (69.31.162.50 – 69.31.162.54).
PFSense box 1: 69.31.162.50/29 and PFSense box 2: 69.31.162.51/29
Default gateway for each is 69.31.162.49.
CARP virtual IP address on PFSense box 1 and 2: 69.31.162.54/29
PFSense box 1 is the master (owns the virtual IP address at this time since it is the master)
CARP stands for Common Address Resolution Protocol and the concept is basically the same as Cisco HSRP or VRRP.
A CARP virtual IP address is shared between the two PFSense boxes. One PFSense device is dedicated as a master (active and owns IP) and the other(s) are dedicated as a backup(on standby).
If the master PFSense device goes down the VRRP announcements will stop from the master and the CARP virtual IP address will be moved from the master device to the backup device and the backup device will become the Master.
This environment setup is used for router failover redundancy is case one router goes down the other one will take over as a backup router automatically.
Details:
PFSense box 1:
Interface: em1
IP address: 69.31.162.50/29
GW: 69.31.162.49
CARP Virtual IP: 69.31.162.54/29 (MASTER)
PFSense box 2:
Interface: em1
IP address: 69.31.162.51/29
GW: 69.31.162.49
CARP Virtual IP: 69.31.162.54/29 (BACKUP)
Setup and Problem:
I have a few command prompts open on my computer which goes through another Shaw internet connection.
I have a command prompt windows open continually pinging the following IP addresses:
69.31.162.50
69.31.162.51
69.31.162.54
69.31.162.50 pings and replies continually fine.
69.31.162.51 pings and replies continually fine.
69.31.162.54 pings and gets request times out.
I can ping all hosts in the same subnet from the PFSense boxes themselves.
I can also ping a remote host like 8.8.8.8 and get replies when using 69.31.162.50 or 69.31.162.51 as the ping source address.
When I ping the remote host 8.8.8.8 I get request time outs when using 69.31.162.54 as the ping source address.
Procedure I tried to troubleshoot problem:
1) I delete the CARP virtual IP address off each PFSense box and then change the IP addresses on em1 interface on PFSense box 1 to 69.31.162.54 the ping in command prompt to 69.31.162.54 will start to get replies and 69.31.162.50 will start to get request time outs as expected.
2) I change the IP addresses on em1 interface on PFSense box 1 back to 69.31.162.50 the ping in command prompt to 69.31.162.54 will start to get request time outs again and 69.31.162.50 will start to get replies again as expected.
3) I then add the CARP virtual IP address to PFSense box 1 as 69.31.162.54/29 on em1 interface again, the ping in command prompt to 69.31.162.54 will start to get replies but only get three or four replies get through and then I start to get request time outs again.
Trace Information
I can traceroute from my computer to 69.31.162.50 or .51 I get the below trace:
I:\Users\JHope>tracert -d 69.31.162.51
Tracing route to 69.31.162.51 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 10.192.70.1
2 * * * Request timed out.
3 11 ms 10 ms 11 ms 64.59.149.173
4 12 ms 20 ms 11 ms 206.223.127.20
5 11 ms 11 ms 11 ms 69.31.170.74
6 12 ms 11 ms 16 ms 69.31.170.114
7 37 ms 32 ms 33 ms 69.31.162.51
Trace complete.
I can traceroute from my computer to 69.31.162.54 I get the below trace:
I:\Users\JHope>tracert -d 69.31.162.54
Tracing route to 69.31.162.54 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 10.192.70.1
2 * * * Request timed out.
3 11 ms 10 ms 12 ms 64.59.149.173
4 11 ms 11 ms 11 ms 206.223.127.20
5 12 ms 13 ms 10 ms 69.31.170.74
6 13 ms 13 ms 11 ms 69.31.170.114
7 * * * Request timed out.
8 * * * Request timed out.
9 ^C
Observations:
I see the packet trace is getting to and traversing Smarttnet’s last hop router (69.31.170.114) and then the packet is unable to find where to go next, but the last hop router has received it but cannot communicated with host 69.31.162.54 on the 69.31.162.48/29 subnet.
If I put a tcpdump capture trace on interface em1 on the PFSense box, I see no capture of my ping (ICMP) request from my Shaw internet external IP address.
When I put a tcpdump capture trace on interface em1 on the PFSense box when ping 69.31.162.50 or .51 from my computer, I DO see the capture of my ping (ICMP) request from my Shaw internet external IP address.
Here is the ifconfig of em1 interface from PFSense box 1:
em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether 00:26:9e:1c:c5:12
inet6 fe80::226:9eff:fe1c:c512%em3 prefixlen 64 scopeid 0x4
inet 69.31.162.50 netmask 0xfffffff8 broadcast 69.31.162.55
inet 69.31.162.54 netmask 0xfffffff8 broadcast 69.31.162.55 vhid 1
nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
carp: MASTER vhid 1 advbase 1 advskew 0
Apparently the HW mac addresses for the CARP virtual IP address on vhid1 is 00:00:5E:00:01:01, CARP virtual IP address on vhid2 is 00:00:5E:00:01:02, CARP virtual IP address on vhid3 is 00:00:5E:00:01:03, and so on.
Question and Help Needed
Can you tell me why IP address 69.31.162.54 is not reachable when I set up as a CARP virtual IP address of 69.31.162.54/29 on em1? Does it have something to do with the MAC addresses on the physical interface and virtual interface?
Can you tell me why IP address 69.31.162.54 is reachable, but why for only three or four ping replies when I delete the CARP virtual IP address and change the physical interface to 69.31.162.54, then change it back to 69.31.162.50/29 and then again add a CARP virtual IP address of 69.31.162.54/29 on em1? Does it have something to do with the MAC addresses on the physical interface being real and virtual interface being spoofed or fake? Is so is there a way around this?</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>
-
Can you tell me why IP address 69.31.162.54 is not reachable when I set up as a CARP virtual IP address of 69.31.162.54/29 on em1? Does it have something to do with the MAC addresses on the physical interface and virtual interface?
You're changing what MAC address that IP is associated with, so you'll need to clear the upstream ARP cache after making such changes. Power cycle the modem if it's cable or DSL.
-
Can you print out the ifconfig from em1 on pfsense box 2?
If your provider is also using CARP or VRRP, you might need to reset the VHID to something else to avoid conflict.
Is there anything in the logs to indicate a problem on either machine?