CARP VIP not reachable

Eugene

Remove the switch and connect fxp1 of pfSense1 to fxp1 of pfSense2 with crossover cable. Now ping with tcpdump please.

surfdude

Removing the switch doesn't help:

15:38:56.824983 arp who-has 192.168.51.3 tell 192.168.51.2
15:38:57.825252 arp who-has 192.168.51.3 tell 192.168.51.2
15:38:58.826256 arp who-has 192.168.51.3 tell 192.168.51.2

Eugene

do you see these arp-requests on vlan2 of pfSense1?

surfdude

Yes!

Eugene

I suspect we are talking about vlan1 (not vlan2 as in my previous e-mail), right?
doyou see anything when
tcpdump -i vlan1 -n vrrp

and please one more time ifconfig from both boxes.

surfdude

You're right, vlan1 of course. tcpdump -i vlan1 -n vrrp doesn't show anything.

ipconfig of pfsense1:

fxp0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=8 <vlan_mtu>ether 00:50:8b:ec:84:d0
inet 192.168.1.200 netmask 0x80000000 broadcast 255.255.255.255
inet6 fe80::250:8bff:feec:84d0%fxp0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
fxp1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=8 <vlan_mtu>ether 00:50:8b:ec:84:ce
inet6 fe80::250:8bff:feec:84ce%fxp1 prefixlen 64 scopeid 0x2
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
pflog0: flags=100 <promisc>metric 0 mtu 33204
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
enc0: flags=41 <up,running>metric 0 mtu 1536
pfsync0: flags=41 <up,running>metric 0 mtu 1460
pfsync: syncdev: vlan2 syncpeer: 192.168.49.2 maxupd: 128
vlan0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
ether 00:50:8b:ec:84:ce
inet6 fe80::250:8bff:feec:84d0%vlan0 prefixlen 64 scopeid 0x7
inet 192.168.50.1 netmask 0xffffff00 broadcast 192.168.50.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vlan: 3 parent interface: fxp1
vlan1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
ether 00:50:8b:ec:84:ce
inet6 fe80::250:8bff:feec:84d0%vlan1 prefixlen 64 scopeid 0x8
inet 192.168.51.1 netmask 0xffffff00 broadcast 192.168.51.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vlan: 4 parent interface: fxp1
vlan2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
ether 00:50:8b:ec:84:ce
inet6 fe80::250:8bff:feec:84d0%vlan2 prefixlen 64 scopeid 0x9
inet 192.168.49.1 netmask 0xffffff00 broadcast 192.168.49.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vlan: 2 parent interface: fxp1
vlan3: flags=8842 <broadcast,running,simplex,multicast>metric 0 mtu 1500
ether 00:50:8b:ec:84:ce
inet6 fe80::250:8bff:feec:84d0%vlan3 prefixlen 64 tentative scopeid 0xa
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vlan: 5 parent interface: fxp1
carp0: flags=49 <up,loopback,running>metric 0 mtu 1500
inet 192.168.1.60 netmask 0xffffff00
carp: MASTER vhid 50 advbase 1 advskew 0
carp1: flags=49 <up,loopback,running>metric 0 mtu 1500
inet 192.168.50.3 netmask 0xffffff00
carp: MASTER vhid 1 advbase 1 advskew 0
carp2: flags=49 <up,loopback,running>metric 0 mtu 1500
inet 192.168.1.203 netmask 0xffffff00
carp: MASTER vhid 90 advbase 1 advskew 0
carp3: flags=49 <up,loopback,running>metric 0 mtu 1500
inet 192.168.51.3 netmask 0xffffff00
carp: MASTER vhid 10 advbase 1 advskew 0
ng0: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng1: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng2: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng3: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng4: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng5: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng6: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng7: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng8: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng9: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng10: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng11: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng12: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng13: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng14: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng15: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500
ng16: flags=8890 <pointopoint,noarp,simplex,multicast>metric 0 mtu 1500

ipconfig of pfsense2
fxp0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=8 <vlan_mtu>ether 00:50:8b:ec:b5:42
inet 192.168.1.201 netmask 0x80000000 broadcast 255.255.255.255
inet6 fe80::250:8bff:feec:b542%fxp0 prefixlen 64 scopeid 0x1
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
fxp1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=8 <vlan_mtu>ether 00:50:8b:ec:b6:4e
inet6 fe80::250:8bff:feec:b64e%fxp1 prefixlen 64 scopeid 0x2
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
pflog0: flags=100 <promisc>metric 0 mtu 33204
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
enc0: flags=0<> metric 0 mtu 1536
pfsync0: flags=41 <up,running>metric 0 mtu 1460
pfsync: syncdev: vlan2 syncpeer: 224.0.0.240 maxupd: 128
vlan0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
ether 00:50:8b:ec:b6:4e
inet6 fe80::250:8bff:feec:b542%vlan0 prefixlen 64 scopeid 0x7
inet 192.168.50.2 netmask 0xffffff00 broadcast 192.168.50.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vlan: 3 parent interface: fxp1
vlan1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
ether 00:50:8b:ec:b6:4e
inet6 fe80::250:8bff:feec:b542%vlan1 prefixlen 64 scopeid 0x8
inet 192.168.51.2 netmask 0xffffff00 broadcast 192.168.51.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vlan: 4 parent interface: fxp1
vlan2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
ether 00:50:8b:ec:b6:4e
inet6 fe80::250:8bff:feec:b542%vlan2 prefixlen 64 scopeid 0x9
inet 192.168.49.2 netmask 0xffffff00 broadcast 192.168.49.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vlan: 2 parent interface: fxp1
vlan3: flags=8842 <broadcast,running,simplex,multicast>metric 0 mtu 1500
ether 00:50:8b:ec:b6:4e
inet6 fe80::250:8bff:feec:b542%vlan3 prefixlen 64 scopeid 0xa
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vlan: 5 parent interface: fxp1
carp0: flags=49 <up,loopback,running>metric 0 mtu 1500
inet 192.168.1.60 netmask 0xffffff00
carp: BACKUP vhid 50 advbase 1 advskew 100
carp1: flags=49 <up,loopback,running>metric 0 mtu 1500
inet 192.168.50.3 netmask 0xffffff00
carp: BACKUP vhid 1 advbase 1 advskew 100
carp2: flags=49 <up,loopback,running>metric 0 mtu 1500
inet 192.168.1.203 netmask 0xffffff00
carp: BACKUP vhid 90 advbase 1 advskew 100
carp3: flags=49 <up,loopback,running>metric 0 mtu 1500
inet 192.168.51.3 netmask 0xffffff00
carp: BACKUP vhid 10 advbase 1 advskew 100

Edit: New vlan ordner on box 2</up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></full-duplex></broadcast,running,simplex,multicast></full-duplex></up,broadcast,running,simplex,multicast></full-duplex></up,broadcast,running,simplex,multicast></full-duplex></up,broadcast,running,promisc,simplex,multicast></up,running></up,loopback,running,multicast></promisc></full-duplex></vlan_mtu></up,broadcast,running,promisc,simplex,multicast></full-duplex></vlan_mtu></up,broadcast,running,promisc,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></pointopoint,noarp,simplex,multicast></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></full-duplex></broadcast,running,simplex,multicast></full-duplex></up,broadcast,running,simplex,multicast></full-duplex></up,broadcast,running,simplex,multicast></full-duplex></up,broadcast,running,promisc,simplex,multicast></up,running></up,running></up,loopback,running,multicast></promisc></full-duplex></vlan_mtu></up,broadcast,running,promisc,simplex,multicast></full-duplex></vlan_mtu></up,broadcast,running,promisc,simplex,multicast>

Eugene

I am not sure this is the problem but anyway - why vlan1 and vlan2 have different VlanIDs on these boxes?
And if carp works you should see VRRP packets in your tcpdumps generated by master. Try tracing other vlans.

surfdude

Ok, the vlans have now the same order on both boxes (see above). But no effect.

tcpdump -i vlan0 -n vrrp

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan0, link-type EN10MB (Ethernet), capture size 96 bytes
16:18:24.486895 IP 192.168.50.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36
16:18:25.487915 IP 192.168.50.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36
16:18:26.488924 IP 192.168.50.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36
16:18:27.489942 IP 192.168.50.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 0, authtype none, intvl 1s, length 36

tcpdump -i vlan1 -n vrrp

[nothing]

Eugene

tcpdump -i vlan1 -n vrrp on both boxes give nothing?
what about
tcpdump -i vlan2 -n vrrp

surfdude

Yes, nothing on both. vlan2 is also nothing, but thats the SYNC interface

Eugene

it's very very wird. Can you post


	 <interfaces>... and ...
	 <virtualip></virtualip></interfaces> 
```sections from your configs?

surfdude

<interfaces><lan><if>vlan0</if>
<ipaddr>192.168.50.1</ipaddr>
<subnet>24</subnet>
<media><mediaopt><bandwidth>100</bandwidth>
<bandwidthtype>Mb</bandwidthtype></mediaopt></media></lan>
<wan><if>fxp0</if>
<mtu><media><mediaopt><bandwidth>100</bandwidth>
<bandwidthtype>Mb</bandwidthtype>
<spoofmac><disableftpproxy><ipaddr>192.168.1.200</ipaddr>
<subnet>1</subnet>
<gateway>192.168.1.1</gateway></disableftpproxy></spoofmac></mediaopt></media></mtu></wan>
<opt1><descr>DMZ</descr>
<if>vlan1</if>
<bridge><ipaddr>192.168.51.1</ipaddr>
<subnet>24</subnet>
<gateway><spoofmac><mtu><enable></enable></mtu></spoofmac></gateway></bridge></opt1>
<opt2><descr>SYNC</descr>
<if>vlan2</if>
<bridge><enable><ipaddr>192.168.49.1</ipaddr>
<subnet>24</subnet>
<gateway><spoofmac></spoofmac></gateway></enable></bridge></opt2>
<opt3><descr>DMZExt</descr>
<if>vlan3</if>
<bridge><ipaddr>192.168.52.1</ipaddr>
<subnet>24</subnet>
<gateway><spoofmac></spoofmac></gateway></bridge></opt3></interfaces>

<virtualip><vip><mode>carp</mode>
<interface>lan</interface>
<vhid>2</vhid>
<advskew>0</advskew>
<password>test</password>
<descr><type>single</type>
<subnet_bits>24</subnet_bits>
<subnet>192.168.50.3</subnet></descr></vip>
<vip><mode>carp</mode>
<interface>opt1</interface>
<vhid>8</vhid>
<advskew>0</advskew>
<password>test</password>
<descr><type>single</type>
<subnet_bits>24</subnet_bits>
<subnet>192.168.51.3</subnet></descr></vip></virtualip>

surfdude

pfsense 2

<interfaces><lan><if>vlan0</if>
<ipaddr>192.168.50.2</ipaddr>
<subnet>24</subnet>
<media><mediaopt><bandwidth>100</bandwidth>
<bandwidthtype>Mb</bandwidthtype></mediaopt></media></lan>
<wan><if>fxp0</if>
<mtu><ipaddr>192.168.1.201</ipaddr>
<subnet>1</subnet>
<gateway>192.168.1.1</gateway>
<disableftpproxy><dhcphostname><media><mediaopt><bandwidth>100</bandwidth>
<bandwidthtype>Mb</bandwidthtype>
<spoofmac></spoofmac></mediaopt></media></dhcphostname></disableftpproxy></mtu></wan>
<opt1><descr>DMZ</descr>
<if>vlan1</if>
<bridge><enable><ipaddr>192.168.51.2</ipaddr>
<subnet>24</subnet>
<gateway><spoofmac></spoofmac></gateway></enable></bridge></opt1>
<opt2><descr>SYNC</descr>
<if>vlan2</if>
<bridge><enable><ipaddr>192.168.49.2</ipaddr>
<subnet>24</subnet>
<gateway><spoofmac></spoofmac></gateway></enable></bridge></opt2>
<opt3><descr>DMZExt</descr>
<if>vlan3</if>
<bridge><ipaddr>192.168.52.2</ipaddr>
<subnet>24</subnet>
<gateway><spoofmac></spoofmac></gateway></bridge></opt3></interfaces>

<virtualip><vip><mode>carp</mode>
<interface>lan</interface>
<vhid>2</vhid>
<advskew>100</advskew>
<password>test</password>
<descr><type>single</type>
<subnet_bits>24</subnet_bits>
<subnet>192.168.50.3</subnet></descr></vip>
<vip><mode>carp</mode>
<interface>opt1</interface>
<vhid>8</vhid>
<advskew>100</advskew>
<password>test</password>
<descr><type>single</type>
<subnet_bits>24</subnet_bits>
<subnet>192.168.51.3</subnet></descr></vip></virtualip>

Eugene

everything seems to be ok. If we forget about vip for the moment… can you ping real IP?

surfdude

Now i'm getting crazy. I ran a ping for a few minutes and did add a new VIP for testing. While pressing the pfsense "Apply Changes" button, this happens:

ping 192.168.51.3

PING 192.168.51.3 (192.168.51.3): 56 data bytes
64 bytes from 192.168.51.3: icmp_seq=108 ttl=64 time=0.191 ms
64 bytes from 192.168.51.3: icmp_seq=109 ttl=64 time=0.101 ms
–- 192.168.51.3 ping statistics ---
115 packets transmitted, 2 packets received, 98.3% packet loss
round-trip min/avg/max/stddev = 0.101/0.146/0.191/0.045 ms

after this two packets, now its silence again. I can ping the real ip.

Thank you for hour help!

Eugene

may be stupid question but you have rule allowing icmp to this ip, right?
pfctl -sr | grep vlan1

surfdude

I hope so!

pfctl -sr | grep vlan1

block drop in on ! vlan1 inet from 192.168.51.0/24 to any
block drop in on vlan1 inet6 from fe80::250:8bff:feec:84d0 to any
pass out quick on vlan1 all flags S/SA keep state label "let out anything from firewall host itself"
pass out quick on vlan1 proto icmp all keep state (tcp.closed 5) label "let out anything from firewall host itself"
pass out quick on vlan1 all flags S/SA keep state (tcp.closed 5) label "let out anything from firewall host itself"
pass in quick on vlan1 inet proto tcp from 192.168.51.0/24 to any port = domain flags S/SA keep state label "USER_RULE: DNS Requests"
pass in quick on vlan1 inet proto udp from 192.168.51.0/24 to any port = domain keep state label "USER_RULE: DNS Requests"
pass in quick on vlan1 inet proto tcp from 192.168.51.0/24 to any port = http flags S/SA keep state label "USER_RULE"
pass in quick on vlan1 inet from 192.168.51.0/24 to 192.168.51.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on vlan1 all flags S/SA keep state label "USER_RULE"
pass in quick on vlan1 inet proto tcp from 192.168.51.0/24 to any port = https flags S/SA keep state label "USER_RULE"
pass in quick on vlan1 inet proto tcp from any to 127.0.0.1 port = 8022 flags S/SA keep state label "FTP PROXY: Allow traffic to localhost"
pass in quick on vlan1 inet proto tcp from any to 127.0.0.1 port = ftp flags S/SA keep state label "FTP PROXY: Allow traffic to localhost"

Eugene

Ok, sounds interesting. Can you please run tcpdump -i vlan1 -n vrrp on pfsense1 and click edit any vip and then save apply on pfsense1?

surfdude

Still nothing.

But, if I execute this command on pfsense2, I get sometimes a packet:

/sbin/ifconfig carp1 192.168.51.3/24 vhid 8 advskew 100 pass test

Eugene

i think I run out of ideas… Your carp does not work.

pfsctl -sr | grep carp
Try swap nics fxp0 and fxp1.
Sorry