Carp issues with one interface

egarnel · Sep 12, 2006, 2:21 PM

Here is my issue: the wan, lan & carp interfaces on the primary become master. The wan & carp interfaces on the 2nd become backup, the lan stays in master mode for some reason.

Here is my setup:

2 pfsense servers

server A primary RC2h

wan ip aaa.bbb.ccc.27 real .25 virtual
lan ip 192.168.20.14 .4 virtual
carp ip 192.168.4.1 real .3 virtual
mgmt xxx.yyy.ppp.207 no carp

server B secondary RC2

wan ip aaa.bbb.ccc.26 real .25 virtual
lan ip 192.168.20.5 .4 virtual
carp ip 192.168.4.2 real .3 virtual
mgmt xxx.yyy.ppp.215 no carp

ifconfig

em0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>inet 192.168.20.5 netmask 0xffffff00 broadcast 192.168.20.255
inet6 fe80::204:23ff:fec9:3418%em0 prefixlen 64 scopeid 0x1
ether 00:04:23:c9:34:18
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>inet6 fe80::204:23ff:fec9:3419%em1 prefixlen 64 scopeid 0x2
inet aaa.bbb.ccc.26 netmask 0xffffff00 broadcast aaa.bbb.ccc.255
ether 00:04:23:c9:34:19
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
bge0: flags=8843 <up,broadcast,running,simplex,multicast>mtu 1500
options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet xxx.yyy.ppp.207 netmask 0xffffff00 broadcast xxx.yyy.ppp.255
inet6 fe80::216:36ff:fe13:a608%bge0 prefixlen 64 scopeid 0x3
ether 00:16:36:13:a6:08
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
bge1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet 192.168.4.1 netmask 0xffffff00 broadcast 192.168.4.255
inet6 fe80::216:36ff:fe13:a609%bge1 prefixlen 64 scopeid 0x4
ether 00:16:36:13:a6:09
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
pfsync0: flags=41 <up,running>mtu 1348
pfsync: syncdev: lo0 maxupd: 128
lo0: flags=8049 <up,loopback,running,multicast>mtu 16384
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
pflog0: flags=100 <promisc>mtu 33208
carp0: flags=49 <up,loopback,running>mtu 1500
inet 192.168.4.3 netmask 0xffffff00
carp: BACKUP vhid 1 advbase 1 advskew 29
carp1: flags=49 <up,loopback,running>mtu 1500
inet aaa.bbb.ccc.25 netmask 0xffffff00
carp: BACKUP vhid 2 advbase 1 advskew 29
carp2: flags=49 <up,loopback,running>mtu 1500
inet 192.168.20.4 netmask 0xffffff00
carp: MASTER vhid 3 advbase 1 advskew 29

ping 192.168.20.14
PING 192.168.20.14 (192.168.20.14): 56 data bytes
64 bytes from 192.168.20.14: icmp_seq=0 ttl=64 time=0.293 ms
64 bytes from 192.168.20.14: icmp_seq=1 ttl=64 time=0.407 ms
64 bytes from 192.168.20.14: icmp_seq=2 ttl=64 time=0.283 ms
64 bytes from 192.168.20.14: icmp_seq=3 ttl=64 time=0.292 ms

ffff
ifconfig
bge0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet6 fe80::210:18ff:fe06:76b3%bge0 prefixlen 64 scopeid 0x1
inet 192.168.20.14 netmask 0xffffff00 broadcast 192.168.20.255
ether 00:10:18:06:76:b3
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
bge1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet aaa.bbb.ccc.27 netmask 0xffffff00 broadcast aaa.bbb.ccc.255
inet6 fe80::211:43ff:fe5b:72d6%bge1 prefixlen 64 scopeid 0x2
ether 00:11:43:5b:72:d6
media: Ethernet autoselect (100baseTX <half-duplex>)
status: active
bge2: flags=8843 <up,broadcast,running,simplex,multicast>mtu 1500
options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet xxx.yyy.ppp.215 netmask 0xffffff00 broadcast xxx.yyy.ppp.255
inet6 fe80::211:43ff:fe5b:72d7%bge2 prefixlen 64 scopeid 0x3
ether 00:11:43:5b:72:d7
media: Ethernet autoselect (100baseTX <half-duplex>)
status: active
fxp0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>inet 192.168.4.2 netmask 0xffffff00 broadcast 192.168.4.255
inet6 fe80::202:b3ff:febb:9776%fxp0 prefixlen 64 scopeid 0x4
ether 00:02:b3:bb:97:76
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
pflog0: flags=100 <promisc>mtu 33208
enc0: flags=0<> mtu 1536
lo0: flags=8049 <up,loopback,running,multicast>mtu 16384
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
pfsync0: flags=41 <up,running>mtu 1348
pfsync: syncdev: fxp0 maxupd: 128
carp0: flags=49 <up,loopback,running>mtu 1500
inet 192.168.4.3 netmask 0xffffff00
carp: MASTER vhid 1 advbase 1 advskew 0
carp1: flags=49 <up,loopback,running>mtu 1500
inet aaa.bbb.ccc.25 netmask 0xffffff00
carp: MASTER vhid 2 advbase 1 advskew 0
carp2: flags=49 <up,loopback,running>mtu 1500
inet 192.168.20.4 netmask 0xffffff00
carp: MASTER vhid 3 advbase 1 advskew 0

ping 192.168.20.5
PING 192.168.20.5 (192.168.20.5): 56 data bytes
64 bytes from 192.168.20.5: icmp_seq=0 ttl=64 time=0.644 ms
64 bytes from 192.168.20.5: icmp_seq=1 ttl=64 time=0.436 ms
64 bytes from 192.168.20.5: icmp_seq=2 ttl=64 time=0.459 ms</up,loopback,running></up,loopback,running></up,loopback,running></up,running></up,loopback,running,multicast></promisc></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast></half-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,simplex,multicast></half-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast></up,loopback,running></up,loopback,running></up,loopback,running></promisc></up,loopback,running,multicast></up,running></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast>

both servers can ping each other, the gateway & the virtual ip on the lan,wan, carp & mgmt networks. per /cf/conf/config.xml both servers have matching passwords for carp

Also, If I select "disable carp" , why does it not stay disabled?

sullrich · Sep 12, 2006, 3:23 PM

Check that the vhids are the same on the backup and master for the CARP'd ip's. Also verify that pfsync is working, you can ping the dedicated interface that hosts pfsync, etc.

Oh, I notice you don't have a dedicated pfsync interface? Check out the tutorial, you are not doing this like we suggest.

egarnel · Sep 12, 2006, 3:51 PM

vhids match. I ran thru the tutorial. I noticed on slide 18 under CARP settings that it mentions enabling preemption. I do not have a check box for preemption or load balancing. I am running RC2h on the primary & RC2 on the secondary.

The interface labeled carp is the sync interface on both servers

sullrich · Sep 12, 2006, 3:55 PM

pfsync0: flags=41 <up,running>mtu 1348
pfsync: syncdev: lo0 maxupd: 128

You have not set your pfsync interface in the carp settings area.</up,running>

egarnel · Sep 12, 2006, 4:13 PM

hmmmm. The secondary lan interface goes into backup for a few seconds then reverts back to master. thanks for all your help in this matter

Primary
bge1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet 192.168.4.1 netmask 0xffffff00 broadcast 192.168.4.255
inet6 fe80::216:36ff:fe13:a609%bge1 prefixlen 64 scopeid 0x4
ether 00:16:36:13:a6:09
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active

pfsync0: flags=41 <up,running>mtu 1348
pfsync: syncdev: fxp0 maxupd: 128</up,running>
carp0: flags=49 <up,loopback,running>mtu 1500
inet 192.168.4.3 netmask 0xffffff00
carp: MASTER vhid 1 advbase 1 advskew 0
carp1: flags=49 <up,loopback,running>mtu 1500
inet bbb.yyy.xxx.25 netmask 0xffffff00
carp: MASTER vhid 2 advbase 1 advskew 0
carp2: flags=49 <up,loopback,running>mtu 1500
inet 192.168.20.4 netmask 0xffffff00
carp: MASTER vhid 3 advbase 1 advskew 0

Secondary
bge1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=1b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging>inet 192.168.4.1 netmask 0xffffff00 broadcast 192.168.4.255
inet6 fe80::216:36ff:fe13:a609%bge1 prefixlen 64 scopeid 0x4
ether 00:16:36:13:a6:09
media: Ethernet autoselect (1000baseTX <full-duplex>)
status: active
pfsync0: flags=41 <up,running>mtu 1348
pfsync: syncdev: bge1 maxupd: 128</up,running>
lo0: flags=8049 <up,loopback,running,multicast>mtu 16384
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
pflog0: flags=100 <promisc>mtu 33208
carp0: flags=49 <up,loopback,running>mtu 1500
inet 192.168.4.3 netmask 0xffffff00
carp: BACKUP vhid 1 advbase 1 advskew 100
carp1: flags=49 <up,loopback,running>mtu 1500
inet bbb.xxx.yyy.25 netmask 0xffffff00
carp: BACKUP vhid 2 advbase 1 advskew 100
carp2: flags=49 <up,loopback,running>mtu 1500
inet 192.168.20.4 netmask 0xffffff00
carp: MASTER vhid 3 advbase 1 advskew 254</up,loopback,running></up,loopback,running></up,loopback,running></promisc></up,loopback,running,multicast></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast></up,loopback,running></up,loopback,running></up,loopback,running></full-duplex></rxcsum,txcsum,vlan_mtu,vlan_hwtagging></up,broadcast,running,promisc,simplex,multicast>

sullrich · Sep 12, 2006, 4:16 PM

on bge1, both machines need a unique ip in the same subnet.

egarnel · Sep 12, 2006, 4:34 PM

my bad… I copied & pasted the same info twice.
Here is the correct info for the primary carp interface:

fxp0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>mtu 1500
options=b <rxcsum,txcsum,vlan_mtu>inet 192.168.4.2 netmask 0xffffff00 broadcast 192.168.4.255
inet6 fe80::202:b3ff:febb:9776%fxp0 prefixlen 64 scopeid 0x4
ether 00:02:b3:bb:97:76
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active</full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast>

sullrich · Sep 12, 2006, 4:38 PM

Then I am not sure. It looks okay to me. Double check your switch ports, vlans, etc.

Also refer to the archive, a lot of people have asked a LOT of questions about CARP.

egarnel · Sep 12, 2006, 4:46 PM

thanks for your help.

Just for grins, I am going to try to swap out the nic on the primary server. It is an intel 10/100 pro pci plugged into a PCI-X slot. It should work fine and appears to be, but I have a PCI-X nic card available that came from another dell poweredge server that would be consistent with the other nics and also provide 1000 Mbs.

Numbski · Sep 14, 2006, 5:03 PM

Couple of thoughts from one of the people that has asked a LOT of questions. :)

1. Have you looked at the wiki?
2. VHID's need to be unique not just for that set of carp interfaces, but for anything else that might be on that same network segment, as CARP is broadcast, not a peer-to-peer technology.
3. Triple check that you are using the correct subnet mask for your CARP IP's, and not /32.
4. Make sure that your CARP interface has an allow any any statement on it, just to be safe, and that the systems are connected by a crossover cable for additional security.
5. If you must do carp without a dedicated interface, make sure you have an allow statement for CARP and pfSync from the opposing system, and block traffic from any other hosts.

That's about all I can think of.

egarnel · Sep 15, 2006, 2:06 PM

I have done all that you mention. I am using a dedicated interface for carp. Both carp interfaces are connected via the same vlan and xmlrpc updates are successful. I have not had the chance to swap out the nic for a pci-x nic yet, but I will start with a fresh install when I do. I will have to wait until the next maintenance window