PfSense 2.0RC2: DHCP cluster recover/unknown-state

hbc

Ok, there seems to be the problem. But the behaviour of dhcp4 is not logical to me.

dhcp0 -> ping fails -> invalid argument
dhcp1 -> ping ok
dhcp2 -> ping ok
dhcp3 -> ping fails -> invalid argument
dhcp4 -> ping times out -> 100% packet loss

So dhcp failover works when ping it 0% loss or 100% loss. But why do dhcp0 and dhcp3 fail with invalid argument?

jimp

Double check your interface configuration and subnet masks for those two interfaces.

hbc

Ok, I found a wrong subnet mask in one CARP interface and fixed it.
But this did not help to get it working either.

Then I tried to update both nodes to latest snapshot… now no dhcp failover does work any more.

"time offset too large". But both nodes are served from same ntp server and the time is same on each node.

I wait for next snapshot. Maybe there is a problem when one node is amd64 and the other x86.
I had this time problem some snapshot before.

hbc

Latest snapshot on both today. Got a new state 'partner-down', even peer is ping-able. But those that did not work yesterday kept state with recover/unknown-state.


Failover Group 	My State 	Since 	Peer State 	Since
"dhcp0"  	recover  	2011/05/25 09:35:29  	unknown-state  	2011/05/25 09:35:29  	  	 
"dhcp1"  	partner-down  	2011/05/25 09:37:01  	recover-wait  	2011/05/25 09:35:29  	  	 
"dhcp2"  	partner-down  	2011/05/25 09:37:14  	recover-wait  	2011/05/25 09:35:29  	  	 
"dhcp3"  	recover  	2011/05/25 09:35:29  	unknown-state  	2011/05/25 09:35:29  	  	 
"dhcp4"  	normal  	2011/05/25 09:37:30  	normal  	2011/05/25 09:35:29

Update: Recover needed some time and finished. Now same state as yesterday. Three instances working, two on recover/unknown.

jimp

Are you now able to ping between the IPs on the failing subnets?

If that hasn't changed, it isn't likely to behave any differently.

hbc

I get 'invalid argument' when trying to ping other node from CLI.

But you have to differ here. When I use GUI to ping and select the right interface, I get 100% packet loss.

On CLI I can ping the working dhcp peers without any special parameters, just the others side ip address. 'ping IP'
When I try the same with one of the non-working peers, I get this 'invalid argument' message.

I guess, I have to add the interface. But the use of 'ping -I em1_vlan40 192.168.0.101' tells me something about invalid multicast interface

jimp

Please show "ifconfig -a" from both systems. Pinging from the GUI or the CLI should work as long as you have the right interface chosen.

hbc

Node 1:


$ ifconfig -a
em0: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=1019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwfilter>ether 00:15:17:cc:36:5a
	inet6 fe80::215:17ff:fecc:365a%em0 prefixlen 64 scopeid 0x1 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=88b43 <up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp>metric 0 mtu 1500
	options=1009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwfilter>ether 00:15:17:cc:36:5b
	inet6 fe80::215:17ff:fecc:365b%em1 prefixlen 64 scopeid 0x2 
	inet 192.168.4.2 netmask 0xffffff00 broadcast 192.168.4.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
em2: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:15:17:cc:38:86
	inet 10.10.37.2 netmask 0xffffff00 broadcast 10.10.37.255
	inet6 fe80::215:17ff:fecc:3886%em2 prefixlen 64 scopeid 0x3 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
em3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
	options=19b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4>ether 00:15:17:cc:38:87
	media: Ethernet autoselect
	status: no carrier
bge0: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether 00:19:99:08:95:1f
	inet 192.168.2.20 netmask 0xffffff00 broadcast 192.168.2.255
	inet6 fe80::219:99ff:fe08:951f%bge0 prefixlen 64 scopeid 0x5 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
bge1: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether 00:19:99:08:95:20
	inet 192.168.66.2 netmask 0xffffff00 broadcast 192.168.66.255
	inet6 fe80::219:99ff:fe08:9520%bge1 prefixlen 64 scopeid 0x6 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 
	nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=141 <up,running,promisc>metric 0 mtu 1460
	pfsync: syncdev: em1_vlan60 syncpeer: 192.168.60.3 maxupd: 128
pflog0: flags=100 <promisc>metric 0 mtu 33664
enc0: flags=0<> metric 0 mtu 1536
em0_vlan40: flags=88b43 <up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:15:17:cc:36:5a
	inet6 fe80::215:17ff:fecc:365a%em0_vlan40 prefixlen 64 scopeid 0xc 
	inet 192.168.0.101 netmask 0xffffff00 broadcast 192.168.0.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 40 parent interface: em0
em0_vlan45: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:15:17:cc:36:5a
	inet6 fe80::215:17ff:fecc:365a%em0_vlan45 prefixlen 64 scopeid 0xd 
	inet 10.5.0.2 netmask 0xfffffc00 broadcast 10.5.3.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 45 parent interface: em0
em0_vlan46: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:15:17:cc:36:5a
	inet6 fe80::215:17ff:fecc:365a%em0_vlan46 prefixlen 64 scopeid 0xe 
	inet 192.168.6.2 netmask 0xffffff00 broadcast 192.168.6.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 46 parent interface: em0
em0_vlan47: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:15:17:cc:36:5a
	inet6 fe80::215:17ff:fecc:365a%em0_vlan47 prefixlen 64 scopeid 0xf 
	inet 192.168.7.2 netmask 0xffffff00 broadcast 192.168.7.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 47 parent interface: em0
em1_vlan60: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:15:17:cc:36:5b
	inet6 fe80::215:17ff:fecc:365a%em1_vlan60 prefixlen 64 scopeid 0x10 
	inet 192.168.60.2 netmask 0xffffff00 broadcast 192.168.60.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
	vlan: 60 parent interface: em1
vip10: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.2.10 netmask 0xffffff00 
	carp: MASTER vhid 10 advbase 1 advskew 0
vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.2.22 netmask 0xffffff00 
	carp: MASTER vhid 1 advbase 1 advskew 0
vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.0.1 netmask 0xffffff00 
	carp: MASTER vhid 2 advbase 1 advskew 0
vip3: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 10.5.0.1 netmask 0xfffffc00 
	carp: MASTER vhid 3 advbase 1 advskew 0
vip4: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.6.1 netmask 0xffffff00 
	carp: MASTER vhid 4 advbase 1 advskew 0
vip5: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.7.1 netmask 0xffffff00 
	carp: MASTER vhid 5 advbase 1 advskew 0
vip6: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.66.1 netmask 0xffffff00 
	carp: MASTER vhid 6 advbase 1 advskew 0
vip7: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.4.1 netmask 0xffffff00 
	carp: MASTER vhid 7 advbase 1 advskew 0
vip8: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.60.1 netmask 0xffffff00 
	carp: MASTER vhid 8 advbase 1 advskew 0
vip9: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 10.10.37.1 netmask 0xffffff00 
	carp: MASTER vhid 9 advbase 1 advskew 0
vip11: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.2.31 netmask 0xffffff00 
	carp: MASTER vhid 11 advbase 1 advskew 0</up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp></promisc></up,running,promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></pointopoint,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,promisc,allmulti,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4></broadcast,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwfilter></up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwfilter></up,broadcast,running,promisc,allmulti,simplex,multicast>

Node 2:


$ ifconfig -a
em0: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 00:11:2f:b9:e8:4b
	inet 192.168.2.21 netmask 0xffffff00 broadcast 192.168.2.255
	inet6 fe80::211:2fff:feb9:e84b%em0 prefixlen 64 scopeid 0x1 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
xl0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
	options=80009 <rxcsum,vlan_mtu,linkstate>ether 00:04:76:17:5f:ae
	media: Ethernet autoselect (none)
	status: no carrier
xl1: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
	options=80009 <rxcsum,vlan_mtu,linkstate>ether 00:04:76:17:65:93
	media: Ethernet autoselect (none)
	status: no carrier
em1: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 00:0e:0c:a8:69:99
	inet6 fe80::20e:cff:fea8:6999%em1 prefixlen 64 scopeid 0x4 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6 
	nd6 options=3 <performnud,accept_rtadv>pflog0: flags=100 <promisc>metric 0 mtu 33200
pfsync0: flags=141 <up,running,promisc>metric 0 mtu 1460
	pfsync: syncdev: em1_vlan60 syncpeer: 192.168.60.2 maxupd: 128
enc0: flags=0<> metric 0 mtu 1536
em1_vlan40: flags=88b43 <up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan40 prefixlen 64 scopeid 0xa 
	inet 192.168.0.102 netmask 0xffffff00 broadcast 192.168.0.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 40 parent interface: em1
em1_vlan45: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan45 prefixlen 64 scopeid 0xb 
	inet 10.5.0.3 netmask 0xfffffc00 broadcast 10.5.3.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 45 parent interface: em1
em1_vlan46: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan46 prefixlen 64 scopeid 0xc 
	inet 192.168.6.3 netmask 0xffffff00 broadcast 192.168.6.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 46 parent interface: em1
em1_vlan47: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan47 prefixlen 64 scopeid 0xd 
	inet 192.168.7.3 netmask 0xffffff00 broadcast 192.168.7.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 47 parent interface: em1
em1_vlan66: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan66 prefixlen 64 scopeid 0xe 
	inet 192.168.66.3 netmask 0xffffff00 broadcast 192.168.66.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 66 parent interface: em1
em1_vlan50: flags=88b43 <up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan50 prefixlen 64 scopeid 0xf 
	inet 192.168.4.3 netmask 0xffffff00 broadcast 192.168.4.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 50 parent interface: em1
em1_vlan60: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan60 prefixlen 64 scopeid 0x10 
	inet 192.168.60.3 netmask 0xffffff00 broadcast 192.168.60.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 60 parent interface: em1
em1_vlan1037: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan1037 prefixlen 64 scopeid 0x11 
	inet 10.10.37.3 netmask 0xffffff00 broadcast 10.10.37.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 1037 parent interface: em1
vip10: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.2.10 netmask 0xffffff00 
	carp: BACKUP vhid 10 advbase 1 advskew 100
vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.2.22 netmask 0xffffff00 
	carp: BACKUP vhid 1 advbase 1 advskew 100
vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.0.1 netmask 0xffffff00 
	carp: BACKUP vhid 2 advbase 1 advskew 100
vip3: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 10.5.0.1 netmask 0xfffffc00 
	carp: BACKUP vhid 3 advbase 1 advskew 100
vip4: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.6.1 netmask 0xffffff00 
	carp: BACKUP vhid 4 advbase 1 advskew 100
vip5: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.7.1 netmask 0xffffff00 
	carp: BACKUP vhid 5 advbase 1 advskew 100
vip6: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.66.1 netmask 0xffffff00 
	carp: BACKUP vhid 6 advbase 1 advskew 100
vip7: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.4.1 netmask 0xffffff00 
	carp: BACKUP vhid 7 advbase 1 advskew 100
vip8: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.60.1 netmask 0xffffff00 
	carp: BACKUP vhid 8 advbase 1 advskew 100
vip9: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 10.10.37.1 netmask 0xffffff00 
	carp: BACKUP vhid 9 advbase 1 advskew 100
vip11: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.2.31 netmask 0xffffff00 
	carp: BACKUP vhid 11 advbase 1 advskew 100</up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp></up,running,promisc></promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></pointopoint,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,allmulti,simplex,multicast></rxcsum,vlan_mtu,linkstate></broadcast,simplex,multicast></rxcsum,vlan_mtu,linkstate></broadcast,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,allmulti,simplex,multicast>

jimp

Forgot to ask, also need to see something like the banner from the console that shows what interface names are assigned to which physical/vlan interfaces.

hbc

Ok, I don't understand it quit really. What information is missing?

ifconfig -a shows interface name and corresponding IP adress, vlans are prefixed with corespondig parent interface. Do you mean interface names like LAN, OPT1, WAN? I don't find them in any previously given data like dhcpd.conf. For what do you need this info?

jimp

Because without that it's a lot of needless work trying to figure out how they match up. You should want to make it easier for people who are trying to help you, not harder. :-)

hbc

Ok, here you are:

Node 1:


WAN (wan)          -> em0          -> 192.168.2.21
  LAN (lan)        -> em1_vlan40   -> 192.168.0.102
  WLAN1 (opt1)     -> em1_vlan45   -> 10.5.0.3
  WLAN2 (opt2)     -> em1_vlan46   -> 192.168.6.3
  WLAN3 (opt3)     -> em1_vlan47   -> 192.168.7.3
  STW (opt4)       -> em1_vlan66   -> 192.168.66.3
  BEAMER (opt5)    -> em1_vlan50   -> 192.168.4.3
  IRMC (opt6)      -> em1_vlan60   -> 192.168.60.3
  LABOR (opt7)     -> em1_vlan1037 -> 10.10.37.3

Node 2:


  WAN (wan)       -> bge0       -> 192.168.2.20
  LAN (lan)       -> em0_vlan40 -> 192.168.0.101
  WLAN1 (opt1)    -> em0_vlan45 -> 10.5.0.2
  WLAN2 (opt2)    -> em0_vlan46 -> 192.168.6.2
  WLAN3 (opt3)    -> em0_vlan47 -> 192.168.7.2
  STW (opt4)      -> bge1       -> 192.168.66.2
  BEAMER (opt5)   -> em1        -> 192.168.4.2
  IRMC (opt6)     -> em1_vlan60 -> 192.168.60.2
  LABOR (opt7)    -> em2        -> 10.10.37.2

Node 1 has dedicated (giga) interfaces for most networks, node 2 is just a backup with two physical interfaces and many vlans on LAN side. There may exist a bottle neck in failover state, but primary node hardware can be replaced within 1 hour.

jimp

That all looks ok.

By chance on the failing interfaces are you running captive portal?

hbc

I must disappoint you, but we don't run captive portals on any interface.

jimp

ok. Well try to ping both ways from the command line again and show the full error messages that you get from both directions.

This really has nothing to do with DHCP specifically, and if you fix the connectivity between the firewalls on those interfaces/VLANs then it will likely start to work.

hbc

Here you can see the difference. One works without problem. The other one makes problems. Is my syntax right for ping when specifying interfaces?
Why multicast interface?

[2.0-RC2][root@pfsense01.mydomain.net]/root(8): ping 192.168.66.3
PING 192.168.66.3 (192.168.66.3): 56 data bytes
64 bytes from 192.168.66.3: icmp_seq=0 ttl=64 time=0.234 ms
64 bytes from 192.168.66.3: icmp_seq=1 ttl=64 time=0.271 ms
^C
–- 192.168.66.3 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.234/0.253/0.271/0.019 ms
[2.0-RC2][root@pfsense01.mydomain.net]/root(9): ping 192.168.4.3
PING 192.168.4.3 (192.168.4.3): 56 data bytes
ping: sendto: Invalid argument
ping: sendto: Invalid argument
ping: sendto: Invalid argument
ping: sendto: Invalid argument
^C
–- 192.168.4.3 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
[2.0-RC2][root@pfsense01.mydomain.net]/root(10): ping -I em1 192.168.4.3
ping: invalid multicast interface: `em1'
[2.0-RC2][root@pfsense01.mydomain.net]/root(11):

[2.0-RC2][root@pfsense02.mydomain.net]/root(1): ping 192.168.66.2
PING 192.168.66.2 (192.168.66.2): 56 data bytes
64 bytes from 192.168.66.2: icmp_seq=0 ttl=64 time=0.356 ms
64 bytes from 192.168.66.2: icmp_seq=1 ttl=64 time=0.232 ms
^C
–- 192.168.66.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.232/0.294/0.356/0.062 ms
[2.0-RC2][root@pfsense02.mydomain.net]/root(2): ping 192.168.4.2
PING 192.168.4.2 (192.168.4.2): 56 data bytes
ping: sendto: Invalid argument
ping: sendto: Invalid argument
ping: sendto: Invalid argument
^C
–- 192.168.4.2 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
[2.0-RC2][root@pfsense02.mydomain.net]/root(3): ping -I em1_vlan50 192.168.4.2
ping: invalid multicast interface: `em1_vlan50'

jimp

You should not need to use -I at all. If you ping, it should follow the routing table and go to the local interface.

Check netstat -rn (or Diagnostics > Routes) and see if anything there doesn't look quite right. Also make sure you don't have any overlapping subnets in things like IPsec.

You should be able to just ping one from the other with "ping <ip>" and if that doesn't work, there is definitely something wrong somewhere.</ip>

hbc

Hmmm, ok. That is a good suggestion.

There are differences in routing table. On node 1 there are entries for the peers, on node 2 are these peer routes missing.
But that should not make any influence, since some peer ip addresses are pingable, others not - even with these different routing tables.

Node 1:


Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.2.254      UGS         0 118708283   bge0
10.5.0.0/22        link#13            U           0 69244840 em0_vl
10.5.0.1           link#20            UH          0     1188   vip3
10.5.0.2           link#13            UHS         0        6    lo0
10.10.37.0/24      link#3             U           0    25859    em2
10.10.37.1         link#26            UH          0        0   vip9
10.10.37.2         link#3             UHS         0        0    lo0
127.0.0.1          link#8             UH          0      266    lo0
192.168.0.0/24     link#12            U           0  3064447 em0_vl
192.168.0.1        link#19            UH          0        0   vip2
192.168.0.101      link#12            UHS         0        0    lo0
192.168.4.0/24     link#2             U           0  1920393    em1
192.168.4.1        link#24            UH          0        0   vip7
192.168.4.2        link#2             UHS         0        2    lo0
192.168.6.0/24     link#14            U           0        0 em0_vl
192.168.6.1        link#21            UH          0        0   vip4
192.168.6.2        link#14            UHS         0        0    lo0
192.168.7.0/24     link#15            U           0        0 em0_vl
192.168.7.1        link#22            UH          0        0   vip5
192.168.7.2        link#15            UHS         0        0    lo0
192.168.60.0/24    link#16            U           0 23881393 em1_vl
192.168.60.1       link#25            UH          0        0   vip8
192.168.60.2       link#16            UHS         0        0    lo0
192.168.66.0/24    link#6             U           0 73122252   bge1
192.168.66.1       link#23            UH          0        0   vip6
192.168.66.2       link#6             UHS         0        2    lo0
192.168.2.0/24     link#5             U           0  9838447   bge0
192.168.2.10       link#17            UH          0        0  vip10
192.168.2.20       link#5             UHS         0        0    lo0
192.168.2.22       link#18            UH          0      243   vip1
192.168.2.31       link#27            UH          0        0  vip11

Node 2:


Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.2.254      UGS         0   182600    em0
10.5.0.0/22        link#11            U           0   104151 em1_vl
10.5.0.3           link#11            UHS         0        0    lo0
10.10.37.0/24      link#17            U           0        0 em1_vl
10.10.37.3         link#17            UHS         0        0    lo0
127.0.0.1          link#6             UH          0      526    lo0
192.168.0.0/24     link#10            U           0     1528 em1_vl
192.168.0.102      link#10            UHS         0        2    lo0
192.168.4.0/24     link#15            U           0     1026 em1_vl
192.168.4.3        link#15            UHS         0        0    lo0
192.168.6.0/24     link#12            U           0        0 em1_vl
192.168.6.3        link#12            UHS         0        0    lo0
192.168.7.0/24     link#13            U           0        0 em1_vl
192.168.7.3        link#13            UHS         0        0    lo0
192.168.60.0/24    link#16            U           0   335071 em1_vl
192.168.60.3       link#16            UHS         0        0    lo0
192.168.66.0/24    link#14            U           0    59040 em1_vl
192.168.66.3       link#14            UHS         0        0    lo0
192.168.2.0/24     link#1             U           0   250104    em0
192.168.2.21       link#1             UHS         0        0    lo0