PfSense 2.0RC2: DHCP cluster recover/unknown-state

[jimp]

The easiest way for that to happen is if you messed with the advskew settings on the primary/secondary and set them incorrectly. Primary's advskew should be 1, the slave will fix itself to >20 when you save.

If the advskew on the secondary is less than 20, dhcpd will think it's supposed to be master on both.

[hbc]

This is in CARP settings not in  DHCP, isn't it? Well, I have on primary all skew to 0 and on secondary 100.

Status -> CARP (failover) is on primary node master and on secondary node backup. Thus CARP is working right, but the dhcp failover group for the two instances with static leases does not recover and change to normal state.

Failover Group 	My State 	Since 	Peer State 	Since
"dhcp0"  	recover  	2011/05/23 11:52:23  	unknown-state  	2011/05/23 11:52:23  	  	 
"dhcp1"  	normal  	2011/05/23 11:52:35  	normal  	2011/05/23 11:52:23  	  	 
"dhcp2"  	normal  	2011/05/23 11:52:30  	normal  	2011/05/23 11:52:23  	  	 
"dhcp3"  	recover  	2011/05/23 11:52:23  	unknown-state  	2011/05/23 11:52:23  	  	 
"dhcp4"  	normal  	2011/05/23 11:52:29  	normal  	2011/05/23 11:52:23 

[jimp]

I'd have to see the dhcpd.conf file from both routers to be able to say much more.

Once upon a time firewall rules could have been blocking the traffic but the way the code works now, if you have the dhcpd failover peer defined, it automatically adds rules to pass the failover traffic.

[hbc]

But the data exchange between ports 519 and 520 is within same subnet and works for the other dhcp instances without having to open ports explicitly. How shall I show you the configuration files? Private message oder here?

[jimp]

There shouldn't be anything really private in the dhcpd.conf, you can just post them inline here or attach them as .txt files. Pasting them into a post and putting them in a "code" block (the hash mark button) should be fine.

I'm mainly worried about the failover declarations and if they are consistent between all of the subnets.

The 519/520 traffic is within the same subnet, sure, but some people have rulesets that would block this traffic. The rules are internal now so that isn't likely the issue.

[hbc]

Ok, here are the two files:

Node 1: Master

option domain-name "mydomain.net";
option ldap-server code 95 = text;
option domain-search-list code 119 = text;

default-lease-time 7200;
max-lease-time 86400;
log-facility local7;
ddns-update-style none;
one-lease-per-client true;
deny duplicates;
ping-check true;
authoritative;
failover peer "dhcp0" {
  primary;
  address 192.168.0.101;
  port 519;
  peer address 192.168.0.102;
  peer port 520;
  max-response-delay 10;
  max-unacked-updates 10;
  split 128;
  mclt 600;

  load balance max seconds 3;
}
failover peer "dhcp1" {
  primary;
  address 10.5.0.2;
  port 519;
  peer address 10.5.0.3;
  peer port 520;
  max-response-delay 10;
  max-unacked-updates 10;
  split 128;
  mclt 600;

  load balance max seconds 3;
}
failover peer "dhcp2" {
  primary;
  address 192.168.66.2;
  port 519;
  peer address 192.168.66.3;
  peer port 520;
  max-response-delay 10;
  max-unacked-updates 10;
  split 128;
  mclt 600;

  load balance max seconds 3;
}
failover peer "dhcp3" {
  primary;
  address 192.168.4.2;
  port 519;
  peer address 192.168.4.3;
  peer port 520;
  max-response-delay 10;
  max-unacked-updates 10;
  split 128;
  mclt 600;

  load balance max seconds 3;
}
failover peer "dhcp4" {
  primary;
  address 192.168.60.2;
  port 519;
  peer address 192.168.60.3;
  peer port 520;
  max-response-delay 10;
  max-unacked-updates 10;
  split 128;
  mclt 600;

  load balance max seconds 3;
}
subnet 192.168.0.0 netmask 255.255.255.0 {
	pool {
		option domain-name-servers 192.168.0.1,192.168.3.3;
		deny dynamic bootp clients;
		deny unknown clients;
		failover peer "dhcp0";
		range 192.168.0.250 192.168.0.250;
	}
	option routers 192.168.0.1;
	option domain-name-servers 192.168.0.1,192.168.3.3;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
host s_lan_0 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.3;
	option host-name B203-AP;
}
host s_lan_1 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.4;
	option host-name C402-AP;
}
host s_lan_2 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.5;
	option host-name C503-AP1;
}
host s_lan_3 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.6;
	option host-name B303-AP;
}
host s_lan_4 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.7;
	option host-name A12-AP;
}
host s_lan_5 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.8;
	option host-name C219-AP;
}
host s_lan_6 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.9;
	option host-name B210-AP;
}
host s_lan_7 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.10;
	option host-name A205-AP1;
}
host s_lan_8 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.11;
	option host-name M04-AP;
}
host s_lan_9 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.12;
	option host-name E105-AP;
}
host s_lan_10 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.13;
	option host-name E220-AP;
}
host s_lan_11 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.14;
	option host-name D213-AP;
}
host s_lan_12 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.15;
	option host-name PBT-AP1;
}
host s_lan_13 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.16;
	option host-name PBT-AP2;
}
host s_lan_14 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.17;
	option host-name PBT-AP3;
}
host s_lan_15 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.18;
	option host-name PBT-AP4;
}
host s_lan_16 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.19;
	option host-name PBT-AP5;
}
host s_lan_17 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.20;
	option host-name PBT-AP6;
}
host s_lan_18 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.21;
	option host-name M01-AP;
}
host s_lan_19 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.22;
	option host-name M02-AP;
}
host s_lan_20 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.23;
	option host-name C503-AP2;
}
host s_lan_21 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.24;
	option host-name G102-AP;
}
host s_lan_22 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.25;
	option host-name D210-AP;
}
host s_lan_23 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.26;
	option host-name G002-AP;
}
host s_lan_24 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.27;
	option host-name G004-AP;
}
host s_lan_25 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.28;
	option host-name A205-AP2;
}
host s_lan_26 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.29;
	option host-name B202-AP;
}
host s_lan_27 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.30;
	option host-name C503-AP3;
}
host s_lan_28 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.31;
	option host-name P1XX-AP2;
}
host s_lan_29 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.32;
	option host-name P1XX-AP1;
}
host s_lan_30 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.33;
	option host-name E110-AP;
}
host s_lan_31 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.34;
	option host-name K004-AP;
}
host s_lan_32 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.35;
	option host-name O100-AP;
}
host s_lan_33 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.36;
	option host-name PBT-AP7;
}
host s_lan_34 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.37;
	option host-name O300-AP;
}
host s_lan_35 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.38;
	option host-name O200-AP;
}
host s_lan_36 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.39;
	option host-name A014-AP;
}
subnet 10.5.0.0 netmask 255.255.252.0 {
	pool {
		option domain-name-servers 10.5.0.1,192.168.3.3;
		deny dynamic bootp clients;
		failover peer "dhcp1";
		range 10.5.0.10 10.5.3.254;
	}
	option routers 10.5.0.1;
	option domain-name "wlan.mydomain.net";
	option domain-search-list "mydomain.net";
	option domain-name-servers 10.5.0.1,192.168.3.3;
	default-lease-time 1800;
	max-lease-time 3600;
	option netbios-name-servers 192.168.3.10,192.168.3.2;
	option netbios-node-type 8;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
subnet 192.168.6.0 netmask 255.255.255.0 {
	pool {
		option domain-name-servers 192.168.6.1,192.168.3.3;
		range 192.168.6.10 192.168.6.250;
	}
	option routers 192.168.6.1;
	option domain-name "dfn.mydomain.net";
	option domain-name-servers 192.168.6.1,192.168.3.3;
	default-lease-time 1800;
	max-lease-time 3600;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
subnet 192.168.7.0 netmask 255.255.255.0 {
	pool {
		option domain-name-servers 192.168.7.1,192.168.3.3;
		range 192.168.7.10 192.168.7.250;
	}
	option routers 192.168.7.1;
	option domain-name "belwue.mydomain.net";
	option domain-name-servers 192.168.7.1,192.168.3.3;
	default-lease-time 1800;
	max-lease-time 3600;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
subnet 192.168.66.0 netmask 255.255.255.0 {
	pool {
		option domain-name-servers 192.168.66.1,192.168.3.3;
		deny dynamic bootp clients;
		failover peer "dhcp2";
		range 192.168.66.160 192.168.66.250;
	}
	option routers 192.168.66.1;
	option domain-name "stw.mydomain.net";
	option domain-search-list "mydomain.net";
	option domain-name-servers 192.168.66.1,192.168.3.3;
	option netbios-name-servers 192.168.3.10,192.168.3.2;
	option netbios-node-type 8;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
subnet 192.168.4.0 netmask 255.255.255.0 {
	pool {
		option domain-name-servers 192.168.4.1,192.168.3.3;
		deny dynamic bootp clients;
		deny unknown clients;
		failover peer "dhcp3";
		range 192.168.4.200 192.168.4.200;
	}
	option routers 192.168.4.1;
	option domain-name "beamer.mydomain.net";
	option domain-name-servers 192.168.4.1,192.168.3.3;
	option netbios-name-servers 192.168.3.10,192.168.3.2;
	option netbios-node-type 8;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
host s_opt5_0 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.10;
	option host-name meovis-server;
}
host s_opt5_1 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.11;
	option host-name meovisDisplay1;
}
host s_opt5_2 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.12;
	option host-name meovisDisplay2;
}
host s_opt5_3 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.13;
	option host-name meovisDisplay3;
}
host s_opt5_4 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.98;
	option host-name Beam-P302;
}
host s_opt5_5 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.99;
	option host-name Beam-P203;
}
host s_opt5_6 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.101;
	option host-name beam-a14;
}
host s_opt5_7 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.102;
	option host-name beam-b306;
}
host s_opt5_8 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.103;
	option host-name beam-b310;
}
host s_opt5_9 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.104;
	option host-name beam-c320;
}
host s_opt5_10 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.105;
	option host-name beam-c322;
}
host s_opt5_11 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.106;
	option host-name beam-d213;
}
host s_opt5_12 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.107;
	option host-name beam-e103;
}
host s_opt5_13 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.108;
	option host-name beam-e104;
}
host s_opt5_14 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.109;
	option host-name beam-g004;
}
host s_opt5_15 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.110;
	option host-name beam-g103;
}
host s_opt5_16 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.111;
	option host-name beam-p101;
}
host s_opt5_17 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.112;
	option host-name beam-p201;
}
host s_opt5_18 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.113;
	option host-name beam-p202;
}
host s_opt5_19 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.114;
	option host-name beam-p301;
}
host s_opt5_20 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.115;
	option host-name beam-p303;
}
host s_opt5_21 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.116;
	option host-name NEC-G003;
}
host s_opt5_22 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.117;
	option host-name NEC-G002;
}
host s_opt5_23 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.118;
	option host-name NEC-G004;
}
host s_opt5_24 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.119;
	option host-name NEC-B308;
}
host s_opt5_25 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.120;
	option host-name NEC-B309;
}
host s_opt5_26 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.121;
	option host-name NEC-B209;
}
host s_opt5_27 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.122;
	option host-name NEC-B208;
}
host s_opt5_28 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.123;
	option host-name NEC-C220;
}
host s_opt5_29 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.124;
	option host-name NEC-A103;
}
host s_opt5_30 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.125;
	option host-name NEC-D212;
}
host s_opt5_31 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.126;
	option host-name NEC-D112;
}
host s_opt5_32 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.127;
	option host-name NEC-D318;
}
host s_opt5_33 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.128;
	option host-name NEC-D210;
}
host s_opt5_34 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.129;
	option host-name NEC-D414;
}
host s_opt5_35 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.130;
	option host-name NEC-D107a;
}
host s_opt5_36 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.131;
	option host-name NEC-D107b;
}
host s_opt5_37 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.132;
	option host-name NEC-D106;
}
host s_opt5_38 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.133;
	option host-name NEC-E005;
}
host s_opt5_39 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.134;
	option host-name NEC-C215;
}
host s_opt5_40 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.135;
	option host-name NEC-B310;
}
host s_opt5_41 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.136;
	option host-name NEC-B306;
}
host s_opt5_42 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.137;
	option host-name NEC-B307;
}
host s_opt5_43 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.138;
	option host-name NEC-E110;
}
host s_opt5_44 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.139;
	option host-name NEC-C320;
}
host s_opt5_45 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.140;
	option host-name NEC-D213;
}
host s_opt5_46 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.141;
	option host-name NEC-C219;
}
host s_opt5_47 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.142;
	option host-name NEC-C321;
}
host s_opt5_48 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.143;
	option host-name NEC-A0112;
}
host s_opt5_49 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.144;
	option host-name NEC-G005;
}
host s_opt5_50 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.145;
	option host-name NEC-G103;
}
host s_opt5_51 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.146;
	option host-name NEC-G302;
}
host s_opt5_52 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.147;
	option host-name NEC-1250;
}
host s_opt5_53 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.148;
	option host-name NEC-D414a;
}
subnet 192.168.60.0 netmask 255.255.255.0 {
	pool {
		option domain-name-servers 192.168.60.1,192.168.3.3;
		deny dynamic bootp clients;
		deny unknown clients;
		failover peer "dhcp4";
		range 192.168.60.200 192.168.60.210;
	}
	option routers 192.168.60.1;
	option domain-name "irmc.mydomain.net";
	option domain-name-servers 192.168.60.1,192.168.3.3;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
host s_opt6_0 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.10;
	option host-name rz-vc-IRMC;
}
host s_opt6_1 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.21;
	option host-name vmhost1-IRMC;
}
host s_opt6_2 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.22;
	option host-name vmhost2-IRMC;
}
host s_opt6_3 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.23;
	option host-name vmhost3-IRMC;
}
host s_opt6_4 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.24;
	option host-name vmhost4-IRMC;
}
host s_opt6_5 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.25;
	option host-name vmhost5-IRMC;
}
host s_opt6_6 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.26;
	option host-name vmhost6-IRMC;
}
host s_opt6_7 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.28;
	option host-name vmhost8-IRMC;
}

Node 2: Backup

option domain-name "mydomain.net";
option ldap-server code 95 = text;
option domain-search-list code 119 = text;

default-lease-time 7200;
max-lease-time 86400;
log-facility local7;
ddns-update-style none;
one-lease-per-client true;
deny duplicates;
ping-check true;
authoritative;
failover peer "dhcp0" {
  secondary;
  address 192.168.0.102;
  port 520;
  peer address 192.168.0.101;
  peer port 519;
  max-response-delay 10;
  max-unacked-updates 10;
  mclt 600;

  load balance max seconds 3;
}
failover peer "dhcp1" {
  secondary;
  address 10.5.0.3;
  port 520;
  peer address 10.5.0.2;
  peer port 519;
  max-response-delay 10;
  max-unacked-updates 10;
  mclt 600;

  load balance max seconds 3;
}
failover peer "dhcp2" {
  secondary;
  address 192.168.66.3;
  port 520;
  peer address 192.168.66.2;
  peer port 519;
  max-response-delay 10;
  max-unacked-updates 10;
  mclt 600;

  load balance max seconds 3;
}
failover peer "dhcp3" {
  secondary;
  address 192.168.4.3;
  port 520;
  peer address 192.168.4.2;
  peer port 519;
  max-response-delay 10;
  max-unacked-updates 10;
  mclt 600;

  load balance max seconds 3;
}
failover peer "dhcp4" {
  secondary;
  address 192.168.60.3;
  port 520;
  peer address 192.168.60.2;
  peer port 519;
  max-response-delay 10;
  max-unacked-updates 10;
  mclt 600;

  load balance max seconds 3;
}
subnet 192.168.0.0 netmask 255.255.255.0 {
	pool {
		option domain-name-servers 192.168.0.1,192.168.3.3;
		deny dynamic bootp clients;
		deny unknown clients;
		failover peer "dhcp0";
		range 192.168.0.250 192.168.0.250;
	}
	option routers 192.168.0.1;
	option domain-name-servers 192.168.0.1,192.168.3.3;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
host s_lan_0 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.3;
	option host-name B203-AP;
}
host s_lan_1 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.4;
	option host-name C402-AP;
}
host s_lan_2 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.5;
	option host-name C503-AP1;
}
host s_lan_3 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.6;
	option host-name B303-AP;
}
host s_lan_4 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.7;
	option host-name A12-AP;
}
host s_lan_5 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.8;
	option host-name C219-AP;
}
host s_lan_6 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.9;
	option host-name B210-AP;
}
host s_lan_7 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.10;
	option host-name A205-AP1;
}
host s_lan_8 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.11;
	option host-name M04-AP;
}
host s_lan_9 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.12;
	option host-name E105-AP;
}
host s_lan_10 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.13;
	option host-name E220-AP;
}
host s_lan_11 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.14;
	option host-name D213-AP;
}
host s_lan_12 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.15;
	option host-name PBT-AP1;
}
host s_lan_13 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.16;
	option host-name PBT-AP2;
}
host s_lan_14 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.17;
	option host-name PBT-AP3;
}
host s_lan_15 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.18;
	option host-name PBT-AP4;
}
host s_lan_16 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.19;
	option host-name PBT-AP5;
}
host s_lan_17 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.20;
	option host-name PBT-AP6;
}
host s_lan_18 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.21;
	option host-name M01-AP;
}
host s_lan_19 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.22;
	option host-name M02-AP;
}
host s_lan_20 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.23;
	option host-name C503-AP2;
}
host s_lan_21 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.24;
	option host-name G102-AP;
}
host s_lan_22 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.25;
	option host-name D210-AP;
}
host s_lan_23 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.26;
	option host-name G002-AP;
}
host s_lan_24 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.27;
	option host-name G004-AP;
}
host s_lan_25 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.28;
	option host-name A205-AP2;
}
host s_lan_26 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.29;
	option host-name B202-AP;
}
host s_lan_27 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.30;
	option host-name C503-AP3;
}
host s_lan_28 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.31;
	option host-name P1XX-AP2;
}
host s_lan_29 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.32;
	option host-name P1XX-AP1;
}
host s_lan_30 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.33;
	option host-name E110-AP;
}
host s_lan_31 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.34;
	option host-name K004-AP;
}
host s_lan_32 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.35;
	option host-name O100-AP;
}
host s_lan_33 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.36;
	option host-name PBT-AP7;
}
host s_lan_34 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.37;
	option host-name O300-AP;
}
host s_lan_35 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.38;
	option host-name O200-AP;
}
host s_lan_36 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.0.39;
	option host-name A014-AP;
}
subnet 10.5.0.0 netmask 255.255.252.0 {
	pool {
		option domain-name-servers 10.5.0.1,192.168.3.3;
		deny dynamic bootp clients;
		failover peer "dhcp1";
		range 10.5.0.10 10.5.3.254;
	}
	option routers 10.5.0.1;
	option domain-name "wlan.mydomain.net";
	option domain-search-list "mydomain.net";
	option domain-name-servers 10.5.0.1,192.168.3.3;
	default-lease-time 1800;
	max-lease-time 3600;
	option netbios-name-servers 192.168.3.10,192.168.3.2;
	option netbios-node-type 8;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
subnet 192.168.6.0 netmask 255.255.255.0 {
	pool {
		option domain-name-servers 192.168.6.1,192.168.3.3;
		range 192.168.6.10 192.168.6.250;
	}
	option routers 192.168.6.1;
	option domain-name "dfn.mydomain.net";
	option domain-name-servers 192.168.6.1,192.168.3.3;
	default-lease-time 1800;
	max-lease-time 3600;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
subnet 192.168.7.0 netmask 255.255.255.0 {
	pool {
		option domain-name-servers 192.168.7.1,192.168.3.3;
		range 192.168.7.10 192.168.7.250;
	}
	option routers 192.168.7.1;
	option domain-name "belwue.mydomain.net";
	option domain-name-servers 192.168.7.1,192.168.3.3;
	default-lease-time 1800;
	max-lease-time 3600;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
subnet 192.168.66.0 netmask 255.255.255.0 {
	pool {
		option domain-name-servers 192.168.66.1,192.168.3.3;
		deny dynamic bootp clients;
		failover peer "dhcp2";
		range 192.168.66.160 192.168.66.250;
	}
	option routers 192.168.66.1;
	option domain-name "stw.mydomain.net";
	option domain-search-list "mydomain.net";
	option domain-name-servers 192.168.66.1,192.168.3.3;
	option netbios-name-servers 192.168.3.10,192.168.3.2;
	option netbios-node-type 8;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
subnet 192.168.4.0 netmask 255.255.255.0 {
	pool {
		option domain-name-servers 192.168.4.1,192.168.3.3;
		deny dynamic bootp clients;
		deny unknown clients;
		failover peer "dhcp3";
		range 192.168.4.200 192.168.4.200;
	}
	option routers 192.168.4.1;
	option domain-name "beamer.mydomain.net";
	option domain-name-servers 192.168.4.1,192.168.3.3;
	option netbios-name-servers 192.168.3.10,192.168.3.2;
	option netbios-node-type 8;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
host s_opt5_0 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.10;
	option host-name meovis-server;
}
host s_opt5_1 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.11;
	option host-name meovisDisplay1;
}
host s_opt5_2 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.12;
	option host-name meovisDisplay2;
}
host s_opt5_3 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.13;
	option host-name meovisDisplay3;
}
host s_opt5_4 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.98;
	option host-name Beam-P302;
}
host s_opt5_5 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.99;
	option host-name Beam-P203;
}
host s_opt5_6 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.101;
	option host-name beam-a14;
}
host s_opt5_7 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.102;
	option host-name beam-b306;
}
host s_opt5_8 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.103;
	option host-name beam-b310;
}
host s_opt5_9 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.104;
	option host-name beam-c320;
}
host s_opt5_10 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.105;
	option host-name beam-c322;
}
host s_opt5_11 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.106;
	option host-name beam-d213;
}
host s_opt5_12 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.107;
	option host-name beam-e103;
}
host s_opt5_13 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.108;
	option host-name beam-e104;
}
host s_opt5_14 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.109;
	option host-name beam-g004;
}
host s_opt5_15 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.110;
	option host-name beam-g103;
}
host s_opt5_16 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.111;
	option host-name beam-p101;
}
host s_opt5_17 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.112;
	option host-name beam-p201;
}
host s_opt5_18 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.113;
	option host-name beam-p202;
}
host s_opt5_19 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.114;
	option host-name beam-p301;
}
host s_opt5_20 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.115;
	option host-name beam-p303;
}
host s_opt5_21 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.116;
	option host-name NEC-G003;
}
host s_opt5_22 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.117;
	option host-name NEC-G002;
}
host s_opt5_23 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.118;
	option host-name NEC-G004;
}
host s_opt5_24 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.119;
	option host-name NEC-B308;
}
host s_opt5_25 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.120;
	option host-name NEC-B309;
}
host s_opt5_26 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.121;
	option host-name NEC-B209;
}
host s_opt5_27 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.122;
	option host-name NEC-B208;
}
host s_opt5_28 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.123;
	option host-name NEC-C220;
}
host s_opt5_29 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.124;
	option host-name NEC-A103;
}
host s_opt5_30 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.125;
	option host-name NEC-D212;
}
host s_opt5_31 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.126;
	option host-name NEC-D112;
}
host s_opt5_32 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.127;
	option host-name NEC-D318;
}
host s_opt5_33 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.128;
	option host-name NEC-D210;
}
host s_opt5_34 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.129;
	option host-name NEC-D414;
}
host s_opt5_35 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.130;
	option host-name NEC-D107a;
}
host s_opt5_36 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.131;
	option host-name NEC-D107b;
}
host s_opt5_37 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.132;
	option host-name NEC-D106;
}
host s_opt5_38 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.133;
	option host-name NEC-E005;
}
host s_opt5_39 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.134;
	option host-name NEC-C215;
}
host s_opt5_40 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.135;
	option host-name NEC-B310;
}
host s_opt5_41 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.136;
	option host-name NEC-B306;
}
host s_opt5_42 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.137;
	option host-name NEC-B307;
}
host s_opt5_43 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.138;
	option host-name NEC-E110;
}
host s_opt5_44 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.139;
	option host-name NEC-C320;
}
host s_opt5_45 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.140;
	option host-name NEC-D213;
}
host s_opt5_46 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.141;
	option host-name NEC-C219;
}
host s_opt5_47 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.142;
	option host-name NEC-C321;
}
host s_opt5_48 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.143;
	option host-name NEC-A0112;
}
host s_opt5_49 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.144;
	option host-name NEC-G005;
}
host s_opt5_50 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.145;
	option host-name NEC-G103;
}
host s_opt5_51 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.146;
	option host-name NEC-G302;
}
host s_opt5_52 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.147;
	option host-name NEC-1250;
}
host s_opt5_53 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.4.148;
	option host-name NEC-D414a;
}
subnet 192.168.60.0 netmask 255.255.255.0 {
	pool {
		option domain-name-servers 192.168.60.1,192.168.3.3;
		deny dynamic bootp clients;
		deny unknown clients;
		failover peer "dhcp4";
		range 192.168.60.200 192.168.60.210;
	}
	option routers 192.168.60.1;
	option domain-name "irmc.mydomain.net";
	option domain-name-servers 192.168.60.1,192.168.3.3;
	option ntp-servers 192.168.3.3,192.168.3.4;

}
host s_opt6_0 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.10;
	option host-name rz-vc-IRMC;
}
host s_opt6_1 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.21;
	option host-name vmhost1-IRMC;
}
host s_opt6_2 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.22;
	option host-name vmhost2-IRMC;
}
host s_opt6_3 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.23;
	option host-name vmhost3-IRMC;
}
host s_opt6_4 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.24;
	option host-name vmhost4-IRMC;
}
host s_opt6_5 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.25;
	option host-name vmhost5-IRMC;
}
host s_opt6_6 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.26;
	option host-name vmhost6-IRMC;
}
host s_opt6_7 {
	hardware ethernet 00:xx:xx:xx:xx:xx;
	fixed-address 192.168.60.28;
	option host-name vmhost8-IRMC;
}

[jimp]

The failover peers look ok there.

Can you actually ping the IP of the secondary on that subnet from the primary? And vice versa?

[hbc]

Ok, there seems to be the problem. But the behaviour of dhcp4 is not logical to me.

dhcp0 -> ping fails -> invalid argument
dhcp1 -> ping ok
dhcp2 -> ping ok
dhcp3 -> ping fails -> invalid argument
dhcp4 -> ping times out -> 100% packet loss

So dhcp failover works when ping it 0% loss or 100% loss. But why do dhcp0 and dhcp3 fail with invalid argument?

[jimp]

Double check your interface configuration and subnet masks for those two interfaces.

[hbc]

Ok, I found a wrong subnet mask in one CARP interface and fixed it.
But this did not help to get it working either.

Then I tried to update both nodes to latest snapshot… now no dhcp failover does work any more.

"time offset too large". But both nodes are served from same ntp server and the time is same on each node.

I wait for next snapshot. Maybe there is a problem when one node is amd64 and the other x86.
I had this time problem some snapshot before.

[hbc]

Latest snapshot on both today. Got a new state 'partner-down', even peer is ping-able. But those that did not work yesterday kept state with recover/unknown-state.

Failover Group 	My State 	Since 	Peer State 	Since
"dhcp0"  	recover  	2011/05/25 09:35:29  	unknown-state  	2011/05/25 09:35:29  	  	 
"dhcp1"  	partner-down  	2011/05/25 09:37:01  	recover-wait  	2011/05/25 09:35:29  	  	 
"dhcp2"  	partner-down  	2011/05/25 09:37:14  	recover-wait  	2011/05/25 09:35:29  	  	 
"dhcp3"  	recover  	2011/05/25 09:35:29  	unknown-state  	2011/05/25 09:35:29  	  	 
"dhcp4"  	normal  	2011/05/25 09:37:30  	normal  	2011/05/25 09:35:29 

Update: Recover needed some time and finished. Now same state as yesterday. Three instances working, two on recover/unknown.

[jimp]

Are you now able to ping between the IPs on the failing subnets?

If that hasn't changed, it isn't likely to behave any differently.

[hbc]

I get 'invalid argument' when trying to ping other node from CLI.

But you have to differ here. When I use GUI to ping and select the right interface, I get 100% packet loss.

On CLI I can ping the working dhcp peers without any special parameters, just the others side ip address. 'ping IP'
When I try the same with one of the non-working peers, I get this 'invalid argument' message.

I guess, I have to add the interface. But the use of 'ping -I em1_vlan40 192.168.0.101' tells me something about invalid multicast interface

[jimp]

Please show "ifconfig -a" from both systems. Pinging from the GUI or the CLI should work as long as you have the right interface chosen.

[hbc]

Node 1:

$ ifconfig -a
em0: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=1019b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwfilter>ether 00:15:17:cc:36:5a
	inet6 fe80::215:17ff:fecc:365a%em0 prefixlen 64 scopeid 0x1 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
em1: flags=88b43 <up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp>metric 0 mtu 1500
	options=1009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwfilter>ether 00:15:17:cc:36:5b
	inet6 fe80::215:17ff:fecc:365b%em1 prefixlen 64 scopeid 0x2 
	inet 192.168.4.2 netmask 0xffffff00 broadcast 192.168.4.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
em2: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:15:17:cc:38:86
	inet 10.10.37.2 netmask 0xffffff00 broadcast 10.10.37.255
	inet6 fe80::215:17ff:fecc:3886%em2 prefixlen 64 scopeid 0x3 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
em3: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
	options=19b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4>ether 00:15:17:cc:38:87
	media: Ethernet autoselect
	status: no carrier
bge0: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether 00:19:99:08:95:1f
	inet 192.168.2.20 netmask 0xffffff00 broadcast 192.168.2.255
	inet6 fe80::219:99ff:fe08:951f%bge0 prefixlen 64 scopeid 0x5 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
bge1: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=8009b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate>ether 00:19:99:08:95:20
	inet 192.168.66.2 netmask 0xffffff00 broadcast 192.168.66.255
	inet6 fe80::219:99ff:fe08:9520%bge1 prefixlen 64 scopeid 0x6 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 
	nd6 options=3 <performnud,accept_rtadv>pfsync0: flags=141 <up,running,promisc>metric 0 mtu 1460
	pfsync: syncdev: em1_vlan60 syncpeer: 192.168.60.3 maxupd: 128
pflog0: flags=100 <promisc>metric 0 mtu 33664
enc0: flags=0<> metric 0 mtu 1536
em0_vlan40: flags=88b43 <up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:15:17:cc:36:5a
	inet6 fe80::215:17ff:fecc:365a%em0_vlan40 prefixlen 64 scopeid 0xc 
	inet 192.168.0.101 netmask 0xffffff00 broadcast 192.168.0.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 40 parent interface: em0
em0_vlan45: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:15:17:cc:36:5a
	inet6 fe80::215:17ff:fecc:365a%em0_vlan45 prefixlen 64 scopeid 0xd 
	inet 10.5.0.2 netmask 0xfffffc00 broadcast 10.5.3.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 45 parent interface: em0
em0_vlan46: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:15:17:cc:36:5a
	inet6 fe80::215:17ff:fecc:365a%em0_vlan46 prefixlen 64 scopeid 0xe 
	inet 192.168.6.2 netmask 0xffffff00 broadcast 192.168.6.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 46 parent interface: em0
em0_vlan47: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:15:17:cc:36:5a
	inet6 fe80::215:17ff:fecc:365a%em0_vlan47 prefixlen 64 scopeid 0xf 
	inet 192.168.7.2 netmask 0xffffff00 broadcast 192.168.7.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 47 parent interface: em0
em1_vlan60: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:15:17:cc:36:5b
	inet6 fe80::215:17ff:fecc:365a%em1_vlan60 prefixlen 64 scopeid 0x10 
	inet 192.168.60.2 netmask 0xffffff00 broadcast 192.168.60.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
	vlan: 60 parent interface: em1
vip10: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.2.10 netmask 0xffffff00 
	carp: MASTER vhid 10 advbase 1 advskew 0
vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.2.22 netmask 0xffffff00 
	carp: MASTER vhid 1 advbase 1 advskew 0
vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.0.1 netmask 0xffffff00 
	carp: MASTER vhid 2 advbase 1 advskew 0
vip3: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 10.5.0.1 netmask 0xfffffc00 
	carp: MASTER vhid 3 advbase 1 advskew 0
vip4: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.6.1 netmask 0xffffff00 
	carp: MASTER vhid 4 advbase 1 advskew 0
vip5: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.7.1 netmask 0xffffff00 
	carp: MASTER vhid 5 advbase 1 advskew 0
vip6: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.66.1 netmask 0xffffff00 
	carp: MASTER vhid 6 advbase 1 advskew 0
vip7: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.4.1 netmask 0xffffff00 
	carp: MASTER vhid 7 advbase 1 advskew 0
vip8: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.60.1 netmask 0xffffff00 
	carp: MASTER vhid 8 advbase 1 advskew 0
vip9: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 10.10.37.1 netmask 0xffffff00 
	carp: MASTER vhid 9 advbase 1 advskew 0
vip11: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.2.31 netmask 0xffffff00 
	carp: MASTER vhid 11 advbase 1 advskew 0</up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp></promisc></up,running,promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></pointopoint,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,linkstate></up,broadcast,running,promisc,allmulti,simplex,multicast></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4></broadcast,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,vlan_hwfilter></up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,tso4,vlan_hwfilter></up,broadcast,running,promisc,allmulti,simplex,multicast> 

Node 2:

$ ifconfig -a
em0: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 00:11:2f:b9:e8:4b
	inet 192.168.2.21 netmask 0xffffff00 broadcast 192.168.2.255
	inet6 fe80::211:2fff:feb9:e84b%em0 prefixlen 64 scopeid 0x1 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
xl0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
	options=80009 <rxcsum,vlan_mtu,linkstate>ether 00:04:76:17:5f:ae
	media: Ethernet autoselect (none)
	status: no carrier
xl1: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
	options=80009 <rxcsum,vlan_mtu,linkstate>ether 00:04:76:17:65:93
	media: Ethernet autoselect (none)
	status: no carrier
em1: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 00:0e:0c:a8:69:99
	inet6 fe80::20e:cff:fea8:6999%em1 prefixlen 64 scopeid 0x4 
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
plip0: flags=8810 <pointopoint,simplex,multicast>metric 0 mtu 1500
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6 
	nd6 options=3 <performnud,accept_rtadv>pflog0: flags=100 <promisc>metric 0 mtu 33200
pfsync0: flags=141 <up,running,promisc>metric 0 mtu 1460
	pfsync: syncdev: em1_vlan60 syncpeer: 192.168.60.2 maxupd: 128
enc0: flags=0<> metric 0 mtu 1536
em1_vlan40: flags=88b43 <up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan40 prefixlen 64 scopeid 0xa 
	inet 192.168.0.102 netmask 0xffffff00 broadcast 192.168.0.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 40 parent interface: em1
em1_vlan45: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan45 prefixlen 64 scopeid 0xb 
	inet 10.5.0.3 netmask 0xfffffc00 broadcast 10.5.3.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 45 parent interface: em1
em1_vlan46: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan46 prefixlen 64 scopeid 0xc 
	inet 192.168.6.3 netmask 0xffffff00 broadcast 192.168.6.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 46 parent interface: em1
em1_vlan47: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan47 prefixlen 64 scopeid 0xd 
	inet 192.168.7.3 netmask 0xffffff00 broadcast 192.168.7.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 47 parent interface: em1
em1_vlan66: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan66 prefixlen 64 scopeid 0xe 
	inet 192.168.66.3 netmask 0xffffff00 broadcast 192.168.66.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 66 parent interface: em1
em1_vlan50: flags=88b43 <up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan50 prefixlen 64 scopeid 0xf 
	inet 192.168.4.3 netmask 0xffffff00 broadcast 192.168.4.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 50 parent interface: em1
em1_vlan60: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan60 prefixlen 64 scopeid 0x10 
	inet 192.168.60.3 netmask 0xffffff00 broadcast 192.168.60.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 60 parent interface: em1
em1_vlan1037: flags=8b43 <up,broadcast,running,promisc,allmulti,simplex,multicast>metric 0 mtu 1500
	options=3 <rxcsum,txcsum>ether 00:0e:0c:a8:69:99
	inet6 fe80::211:2fff:feb9:e84b%em1_vlan1037 prefixlen 64 scopeid 0x11 
	inet 10.10.37.3 netmask 0xffffff00 broadcast 10.10.37.255
	nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	vlan: 1037 parent interface: em1
vip10: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.2.10 netmask 0xffffff00 
	carp: BACKUP vhid 10 advbase 1 advskew 100
vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.2.22 netmask 0xffffff00 
	carp: BACKUP vhid 1 advbase 1 advskew 100
vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.0.1 netmask 0xffffff00 
	carp: BACKUP vhid 2 advbase 1 advskew 100
vip3: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 10.5.0.1 netmask 0xfffffc00 
	carp: BACKUP vhid 3 advbase 1 advskew 100
vip4: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.6.1 netmask 0xffffff00 
	carp: BACKUP vhid 4 advbase 1 advskew 100
vip5: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.7.1 netmask 0xffffff00 
	carp: BACKUP vhid 5 advbase 1 advskew 100
vip6: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.66.1 netmask 0xffffff00 
	carp: BACKUP vhid 6 advbase 1 advskew 100
vip7: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.4.1 netmask 0xffffff00 
	carp: BACKUP vhid 7 advbase 1 advskew 100
vip8: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.60.1 netmask 0xffffff00 
	carp: BACKUP vhid 8 advbase 1 advskew 100
vip9: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 10.10.37.1 netmask 0xffffff00 
	carp: BACKUP vhid 9 advbase 1 advskew 100
vip11: flags=49 <up,loopback,running>metric 0 mtu 1500
	inet 192.168.2.31 netmask 0xffffff00 
	carp: BACKUP vhid 11 advbase 1 advskew 100</up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum></up,broadcast,running,promisc,allmulti,simplex,multicast,staticarp></up,running,promisc></promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></pointopoint,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,allmulti,simplex,multicast></rxcsum,vlan_mtu,linkstate></broadcast,simplex,multicast></rxcsum,vlan_mtu,linkstate></broadcast,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,allmulti,simplex,multicast> 

[jimp]

Forgot to ask, also need to see something like the banner from the console that shows what interface names are assigned to which physical/vlan interfaces.

[hbc]

Ok, I don't understand it quit really. What information is missing?

ifconfig -a shows interface name and corresponding IP adress, vlans are prefixed with corespondig parent interface. Do you mean interface names like LAN, OPT1, WAN? I don't find them in any previously given data like dhcpd.conf. For what do you need this info?

[jimp]

Because without that it's a lot of needless work trying to figure out how they match up. You should want to make it easier for people who are trying to help you, not harder. :-)

[hbc]

Ok, here you are:

Node 1:

WAN (wan)          -> em0          -> 192.168.2.21
  LAN (lan)        -> em1_vlan40   -> 192.168.0.102
  WLAN1 (opt1)     -> em1_vlan45   -> 10.5.0.3
  WLAN2 (opt2)     -> em1_vlan46   -> 192.168.6.3
  WLAN3 (opt3)     -> em1_vlan47   -> 192.168.7.3
  STW (opt4)       -> em1_vlan66   -> 192.168.66.3
  BEAMER (opt5)    -> em1_vlan50   -> 192.168.4.3
  IRMC (opt6)      -> em1_vlan60   -> 192.168.60.3
  LABOR (opt7)     -> em1_vlan1037 -> 10.10.37.3

Node 2:

  WAN (wan)       -> bge0       -> 192.168.2.20
  LAN (lan)       -> em0_vlan40 -> 192.168.0.101
  WLAN1 (opt1)    -> em0_vlan45 -> 10.5.0.2
  WLAN2 (opt2)    -> em0_vlan46 -> 192.168.6.2
  WLAN3 (opt3)    -> em0_vlan47 -> 192.168.7.2
  STW (opt4)      -> bge1       -> 192.168.66.2
  BEAMER (opt5)   -> em1        -> 192.168.4.2
  IRMC (opt6)     -> em1_vlan60 -> 192.168.60.2
  LABOR (opt7)    -> em2        -> 10.10.37.2

Node 1 has dedicated (giga) interfaces for most networks, node 2 is just a backup with two physical interfaces and many vlans on LAN side. There may exist a bottle neck in failover state, but primary node hardware can be replaced within 1 hour.

[jimp]

That all looks ok.

By chance on the failing interfaces are you running captive portal?