PFsense 2.2.4 on Watchguard X550e with Cluster lost network function



  • We run PFsense on two Watchguard x550
    Version 2.2.4-Release i386 nanobsd (4g)

    GB CF Card, Bios Update Watchguard installed

    Carp runs.
    four networkcard are in use(WAN, LAN, DMZ, CLUSTER)
    Now my question.

    after some runtime a network lost function.
    Reboot pfsense runs

    For yesterday backup box lost funktion on cluster.

    on local console i check with Ping on the other box i get  "ping: sendto: No route to host".
    the local address on this box a rechable.,

    a solution for a short time where "#ifconfig sk1 down & ifconfig sk1 up" but this only helps a short time.

    /boot/loader.conf:
    loader_color="NO"
    console=comconsole
    autoboot_delay="5"
    beastie_disable="YES"
    vm.kmem_size="435544320"
    vm.kmem_size_max="535544320"
    hw.usb.no_pf="1"
    hint.ata.0.mode=PIO4
    if_sk_load="yes"

    Services:
    apinger Gateway Monitoring Daemon
    lcdproc LCD Driver
    ntpd         NTP clock sync
    sshd    Secure Shell Daemon
    unbound DNS Resolver

    Installend Packages:
    LCDproc-dev

    Infos vom dmesg:

    k0: promiscuous mode disabled
    sk0: promiscuous mode enabled
    carp: VHID 2@sk0: INIT -> BACKUP
    carp: VHID 2@sk0: BACKUP -> MASTER (preempting a slower master)
    carp: VHID 3@sk2: MASTER -> BACKUP (more frequent advertisement received)
    carp: VHID 1@sk1: MASTER -> BACKUP (more frequent advertisement received)
    carp: VHID 2@sk0: MASTER -> BACKUP (more frequent advertisement received)
    carp: VHID 3@sk2: BACKUP -> MASTER (preempting a slower master)
    carp: VHID 2@sk0: BACKUP -> MASTER (preempting a slower master)
    arp: xxx.xxx.xxx.xxx moved from 00:00:5e:00:01:03 to 00:90:7f:42:7b:b7 on sk2
    carp: VHID 1@sk1: BACKUP -> MASTER (master down)
    arp: xxx.xxx.xxx.xxx moved from 00:00:5e:00:01:01 to 00:90:7f:42:7b:b8 on sk1
    sk3: link state changed to DOWN
    sk3: link state changed to UP
    sk3: link state changed to DOWN
    sk3: link state changed to UP
    carp: VHID 1@sk1: MASTER -> BACKUP (more frequent advertisement received)
    carp: VHID 2@sk0: MASTER -> BACKUP (more frequent advertisement received)
    arp: xxx.xxx.xxx.xxx moved from 00:90:7f:42:7b:b9 to 00:00:5e:00:01:02 on sk0
    carp: VHID 3@sk2: MASTER -> BACKUP (more frequent advertisement received)
    carp: VHID 2@sk0: BACKUP -> MASTER (master down)
    carp: VHID 2@sk0: MASTER -> BACKUP (more frequent advertisement received)
    carp: demoted by 240 to 240 (interface down)
    sk0: link state changed to DOWN
    carp: demoted by 240 to 480 (interface down)
    sk2: link state changed to DOWN
    carp: demoted by 240 to 720 (interface down)
    sk1: link state changed to DOWN
    sk3: link state changed to DOWN
    ifa_del_loopback_route: deletion failed: 3
    carp: VHID 1@sk1: INIT -> BACKUP
    carp: demoted by -240 to 480 (interface up)
    sk1: link state changed to UP
    carp: VHID 1@sk1: BACKUP -> MASTER (master down)
    sk3: link state changed to UP
    sk3: link state changed to DOWN
    sk3: link state changed to UP
    carp: VHID 1@sk1: INIT -> BACKUP
    carp: VHID 1@sk1: INIT -> BACKUP
    carp: VHID 1@sk1: BACKUP -> MASTER (master down)
    ifa_add_loopback_route: insertion failed: 17
    carp: VHID 1@sk1: INIT -> BACKUP
    carp: demoted by 240 to 720 (interface down)
    sk1: link state changed to DOWN
    carp: VHID 1@sk1: INIT -> BACKUP
    carp: demoted by -240 to 480 (interface up)
    sk1: link state changed to UP
    carp: VHID 1@sk1: BACKUP -> MASTER (master down)
    ifa_add_loopback_route: insertion failed: 17
    sk3: link state changed to DOWN
    sk3: link state changed to UP
    sk3: link state changed to DOWN
    sk3: link state changed to UP
    carp: VHID 1@sk1: MASTER -> BACKUP (more frequent advertisement received)
    carp: VHID 2@sk0: INIT -> BACKUP
    carp: demoted by -240 to 240 (interface up)
    sk0: link state changed to UP
    ifa_del_loopback_route: deletion failed: 3
    carp: demoted by -240 to 0 (vhid removed)
    sk2: promiscuous mode disabled
    sk2: promiscuous mode enabled
    carp: demoted by 240 to 240 (interface down)
    carp: demoted by 240 to 480 (interface down)
    sk1: link state changed to DOWN
    carp: VHID 3@sk2: INIT -> BACKUP
    carp: demoted by -240 to 240 (interface up)
    sk2: link state changed to UP
    carp: VHID 1@sk1: INIT -> BACKUP
    carp: demoted by -240 to 0 (interface up)
    sk1: link state changed to UP
    carp: demoted by 240 to 240 (interface down)
    sk0: link state changed to DOWN
    sonewconn: pcb 0xc73ea6e4: Listen queue overflow: 2 already in queue awaiting acceptance (1 occurrences)
    carp: VHID 2@sk0: INIT -> BACKUP
    carp: demoted by -240 to 0 (interface up)
    sk0: link state changed to UP
    carp: demoted by 240 to 240 (interface down)
    sk0: link state changed to DOWN
    sk3: link state changed to DOWN
    sk3: link state changed to UP
    carp: VHID 2@sk0: INIT -> BACKUP
    carp: demoted by -240 to 0 (interface up)
    sk0: link state changed to UP
    carp: demoted by 240 to 240 (interface down)
    sk2: link state changed to DOWN
    carp: demoted by 240 to 480 (interface down)
    sk0: link state changed to DOWN
    carp: demoted by 240 to 720 (interface down)
    sk1: link state changed to DOWN
    sk3: link state changed to DOWN
    
    
    Sep 4 12:39:54	check_reload_status: Carp backup event
    Sep 4 12:39:54	kernel: carp: demoted by 240 to 480 (interface down)
    Sep 4 12:39:54	kernel: sk0: link state changed to DOWN
    Sep 4 12:39:54	check_reload_status: Linkup starting sk0
    Sep 4 12:39:56	php-fpm[93761]: /rc.carpbackup: Carp cluster member "xxx.xxx.xxx.xxx - WAN Gateway IP (2@sk0)" has resumed the state "BACKUP" for vhid 2@sk0
    Sep 4 12:39:56	php-fpm[93761]: /rc.linkup: Hotplug event detected for WAN(wan) but ignoring since interface is configured with static IP (xxx.xxx.xxx.xxx  )
    Sep 4 12:40:14	check_reload_status: updating dyndns WANGW
    Sep 4 12:40:14	check_reload_status: Restarting ipsec tunnels
    Sep 4 12:40:14	check_reload_status: Restarting OpenVPN tunnels/interfaces
    Sep 4 12:40:14	check_reload_status: Reloading filter
    Sep 4 12:40:23	check_reload_status: Carp backup event
    Sep 4 12:40:23	kernel: carp: demoted by 240 to 720 (interface down)
    Sep 4 12:40:23	kernel: sk1: link state changed to DOWN
    Sep 4 12:40:23	check_reload_status: Linkup starting sk1
    Sep 4 12:40:24	php-fpm[52693]: /rc.carpbackup: Carp cluster member "xxx.xxx.xxx.xxx  - Lan Gateway IP (1@sk1)" has resumed the state "BACKUP" for vhid 1@sk1
    Sep 4 12:40:24	php-fpm[52693]: /rc.linkup: Hotplug event detected for LAN(lan) but ignoring since interface is configured with static IP (xxx.xxx.xxx.xxx )
    Sep 4 13:11:17	check_reload_status: Linkup starting sk3
    

    Thx



  • I have the same problem. Some of my LAN links (LAN, DMZ) fail after a while. A simple

    ifconfig sk2 down
    ifconfig sk2 up

    (sometimes sk1, sky or another physical interface)

    solves the problem temperarily. This happens after a while (e.g. a few days) or under high network traffic (backups). But I have permanently traffic (e.g. multiple webcams that are recorded on a NAS). I did not find any appropriate log entries for that.



  • This is a known problem with these boxes, mine started doing it on 2.2.4 and worse on 2.2.5.  Do yourselves a favor, either drop back to 2.2.3 and see how that works out, or like I did, scrap the red box and build a computer, buy a 4 port network card and go from there.

    Myself and several others, all in the same boat and we tried everything to no avail.  Normally, pulling the network cable out of the port that failed, then plugging it back on solves it, or like you are doing.

    I noticed with high traffic it would do it, it just made my systems and network to unreliable to say the least.

    Hope this helps.


Log in to reply