Ping to WAN or LAN fails (pfsense 2.3)



  • Hi,

    We have the problem that circa every 4 hours the server cannot ping or the internet (8.8.8.8 ) or the internal network. If we reboot the server then everything is fine again. This began to happen after we updated to pfsense 2.3.

    We now have a script running every minute that will test connection to the internet and to the internal IP addresses. If it fails the server will reboot itself and the script won't run for 1 hour.

    I'm new here so what can I give you for some more information?

    Thanks!



  • Sooooo-

    Based on your comment -  "or the internal network"…

    I would like you to do an experiment.  I want you to transfer some files on your internal network. Doesn't matter... just something.  Notice that the lights on your switch where the firewall is plugged into do not follow that data transfer.  Your comment "or the internal network" does not make sense since the firewall is not a part of internal pings, transfers or otherwise. Other than actually maybe giving the ip addresses of the internal equipment. But that would be an easy summation. You could look at the box and see its not getting an address from the DHCP server if that was the cause..

    The reason is that the subnet programmed into your devices tell it whether to point its traffic at your pfSense box or not. Anything on your network destine for another device on your same  (subnet) network does not pass through or to the firewall.

    If you have some weird or different setup however then all bets are off. And we don't read minds. Don't be afraid to explain things in as much detail as possible...    :)



  • With internal network I mean that the firwall cannot ping other VLANs anymore.

    So the internet was down, I pinged 8.8.8.8 on the firewall and that worked, then I pinged 192.168.10.1 on the firewall and that failed. (I think with the error: host is down)

    The firewall could ping every device in his subnet(192.168.1.0/24), but pings fails to:

    192.168.10.1
    192.168.30.1
    192.168.40.1

    This is what happend now:

    
    Apr 29 10:07:14	shutdown		reboot by root:
    Apr 29 10:07:12	root		/usr/local/bin/ping_check_fw.sh: 192.168.30.1 unreachable, reboot now!
    Apr 29 10:06:40	xinetd	14112	Reconfigured: new=0 old=1 dropped=0 (services)
    Apr 29 10:06:40	xinetd	14112	readjusting service 6969-udp
    Apr 29 10:06:40	xinetd	14112	Swapping defaults
    Apr 29 10:06:40	xinetd	14112	Starting reconfiguration
    Apr 29 10:06:40	php-fpm	41509	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use LANGTW.
    Apr 29 10:06:40	xinetd	14112	Reconfigured: new=0 old=1 dropped=0 (services)
    Apr 29 10:06:40	xinetd	14112	readjusting service 6969-udp
    Apr 29 10:06:40	xinetd	14112	Swapping defaults
    Apr 29 10:06:40	xinetd	14112	Starting reconfiguration
    Apr 29 10:06:39	php-fpm	41509	/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use CiscoVPN.
    Apr 29 10:06:39	check_reload_status		Reloading filter
    Apr 29 10:06:39	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Apr 29 10:06:39	check_reload_status		Restarting ipsec tunnels
    Apr 29 10:06:39	check_reload_status		updating dyndns LANGTW
    Apr 29 10:06:38	check_reload_status		Reloading filter
    Apr 29 10:06:38	check_reload_status		Restarting OpenVPN tunnels/interfaces
    Apr 29 10:06:38	check_reload_status		Restarting ipsec tunnels
    Apr 29 10:06:38	check_reload_status		updating dyndns CiscoVPN
    
    


  • Thank You!  That is all important information.

    Im going towards a driver conflict/error/crapout myself…

    Can you elaborate on your equipment a little more. It tends to help draw out someone who may have had the same problem with a similar setup.



  • Is this what u need?

    Copyright (c) 1992-2016 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    	The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 10.3-RELEASE #6 05adf0a(RELENG_2_3_0): Mon Apr 11 18:52:07 CDT 2016
        root@ce23-amd64-builder:/builder/pfsense-230/tmp/obj/builder/pfsense-230/tmp/FreeBSD-src/sys/pfSense amd64
    FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
    CPU: Intel(R) Atom(TM) CPU D525   @ 1.80GHz (1800.02-MHz K8-class CPU)
      Origin="GenuineIntel"  Id=0x106ca  Family=0x6  Model=0x1c  Stepping=10
      Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x40e31d <sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe>AMD Features=0x20100800 <syscall,nx,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant, performance statistics
    real memory  = 4294967296 (4096 MB)
    avail memory = 4087820288 (3898 MB)
    Event timer "LAPIC" quality 400
    ACPI APIC Table: <031511 APIC1031>
    FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
     cpu0 (BSP): APIC ID:  0
     cpu1 (AP/HT): APIC ID:  1
     cpu2 (AP): APIC ID:  2
     cpu3 (AP/HT): APIC ID:  3
    random: <software, yarrow="">initialized
    ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/64 (20150515/tbfadt-644)
    ioapic0: Changing APIC ID to 4
    ioapic0 <version 2.0="">irqs 0-23 on motherboard
    wlan: mac acl policy registered
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff806208b0, 0) error 1
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80620960, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80620a10, 0) error 1
    iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80647bb0, 0) error 1
    iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80647c60, 0) error 1
    iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80647d10, 0) error 1
    netmap: loaded module
    kbd1 at kbdmux0
    cryptosoft0: <software crypto="">on motherboard
    padlock0: No ACE support.
    acpi0: <031511 XSDT1031> on motherboard
    acpi0: Power Button (fixed)
    cpu0: <acpi cpu="">on acpi0
    cpu1: <acpi cpu="">on acpi0
    cpu2: <acpi cpu="">on acpi0
    cpu3: <acpi cpu="">on acpi0
    attimer0: <at timer="">port 0x40-0x43 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
    Event timer "RTC" frequency 32768 Hz quality 0
    hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 14318180 Hz quality 950
    Event timer "HPET" frequency 14318180 Hz quality 450
    Event timer "HPET1" frequency 14318180 Hz quality 440
    Event timer "HPET2" frequency 14318180 Hz quality 440
    Event timer "HPET3" frequency 14318180 Hz quality 440
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    vgapci0: <vga-compatible display="">port 0xbc00-0xbc07 mem 0xfe580000-0xfe5fffff,0xd0000000-0xdfffffff,0xfe600000-0xfe6fffff irq 16 at device 2.0 on pci0
    agp0: <intel pineview="" svga="" controller="">on vgapci0
    agp0: aperture size is 256M, detected 8188k stolen memory
    vgapci0: Boot video device
    vgapci1: <vga-compatible display="">mem 0xfe480000-0xfe4fffff at device 2.1 on pci0
    uhci0: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb880-0xb89f irq 16 at device 26.0 on pci0
    usbus0 on uhci0
    uhci1: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb800-0xb81f irq 21 at device 26.1 on pci0
    usbus1 on uhci1
    uhci2: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb480-0xb49f irq 19 at device 26.2 on pci0
    usbus2 on uhci2
    ehci0: <intel 82801i="" (ich9)="" usb="" 2.0="" controller="">mem 0xfe577c00-0xfe577fff irq 18 at device 26.7 on pci0
    usbus3: EHCI version 1.0
    usbus3 on ehci0
    hdac0: <intel 82801i="" hda="" controller="">mem 0xfe570000-0xfe573fff irq 22 at device 27.0 on pci0
    pcib1: <acpi pci-pci="" bridge="">irq 17 at device 28.0 on pci0
    pci1: <acpi pci="" bus="">on pcib1
    pcib2: <acpi pci-pci="" bridge="">irq 18 at device 28.2 on pci0
    pci2: <acpi pci="" bus="">on pcib2
    em0: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k="">port 0xcc00-0xcc1f mem 0xfe7e0000-0xfe7fffff,0xfe7dc000-0xfe7dffff irq 18 at device 0.0 on pci2
    em0: Using MSIX interrupts with 3 vectors
    em0: Ethernet address: 00:30:18:a2:fb:bd
    em0: netmap queues/slots: TX 1/1024, RX 1/1024
    pcib3: <acpi pci-pci="" bridge="">irq 19 at device 28.3 on pci0
    pci3: <acpi pci="" bus="">on pcib3
    em1: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k="">port 0xdc00-0xdc1f mem 0xfe8e0000-0xfe8fffff,0xfe8dc000-0xfe8dffff irq 19 at device 0.0 on pci3
    em1: Using MSIX interrupts with 3 vectors
    em1: Ethernet address: 00:30:18:a2:fb:be
    em1: netmap queues/slots: TX 1/1024, RX 1/1024
    pcib4: <acpi pci-pci="" bridge="">irq 17 at device 28.4 on pci0
    pci4: <acpi pci="" bus="">on pcib4
    xhci0: <asmedia asm1042="" usb="" 3.0="" controller="">mem 0xfe9f8000-0xfe9fffff irq 16 at device 0.0 on pci4
    xhci0: 32 bytes context size, 32-bit DMA
    usbus4 on xhci0
    uhci3: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb400-0xb41f irq 23 at device 29.0 on pci0
    usbus5 on uhci3
    uhci4: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb080-0xb09f irq 19 at device 29.1 on pci0
    usbus6 on uhci4
    uhci5: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb000-0xb01f irq 18 at device 29.2 on pci0
    usbus7 on uhci5
    ehci1: <intel 82801i="" (ich9)="" usb="" 2.0="" controller="">mem 0xfe577800-0xfe577bff irq 23 at device 29.7 on pci0
    usbus8: EHCI version 1.0
    usbus8 on ehci1
    pcib5: <acpi pci-pci="" bridge="">at device 30.0 on pci0
    pci5: <acpi pci="" bus="">on pcib5
    em2: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.1.0="">port 0xec00-0xec3f mem 0xfebe0000-0xfebfffff,0xfebc0000-0xfebdffff irq 18 at device 4.0 on pci5
    em2: Ethernet address: 00:30:18:a0:f2:a1
    em2: netmap queues/slots: TX 1/256, RX 1/256
    em3: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.1.0="">port 0xe880-0xe8bf mem 0xfeb80000-0xfeb9ffff,0xfeb60000-0xfeb7ffff irq 19 at device 6.0 on pci5
    em3: Ethernet address: 00:30:18:a0:f2:a2
    em3: netmap queues/slots: TX 1/256, RX 1/256
    em4: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.1.0="">port 0xe800-0xe83f mem 0xfeb20000-0xfeb3ffff,0xfeb00000-0xfeb1ffff irq 16 at device 7.0 on pci5
    em4: Ethernet address: 00:30:18:a0:f2:a3
    em4: netmap queues/slots: TX 1/256, RX 1/256
    isab0: <pci-isa bridge="">at device 31.0 on pci0
    isa0: <isa bus="">on isab0
    ahci0: <intel ich9="" ahci="" sata="" controller="">port 0xa400-0xa407,0xac00-0xac03,0xa880-0xa887,0xa800-0xa803,0xa480-0xa49f mem 0xfe577000-0xfe5777ff irq 19 at device 31.2 on pci0
    ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier not supported
    ahcich0: <ahci channel="">at channel 0 on ahci0
    ahcich1: <ahci channel="">at channel 1 on ahci0
    ahcich2: <ahci channel="">at channel 2 on ahci0
    ahcich3: <ahci channel="">at channel 3 on ahci0
    ahcich4: <ahci channel="">at channel 4 on ahci0
    ahcich5: <ahci channel="">at channel 5 on ahci0
    ahciem0: <ahci enclosure="" management="" bridge="">on ahci0
    acpi_button0: <power button="">on acpi0
    uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
    uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
    atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0
    atkbd0: <at keyboard="">irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    orm0: <isa option="" roms="">at iomem 0xce000-0xcefff,0xcf000-0xcffff,0xd0000-0xd0fff on isa0
    sc0: <system console="">at flags 0x100 on isa0
    sc0: VGA <16 virtual consoles, flags=0x300>
    vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    ppc0: cannot reserve I/O port range
    Timecounters tick every 1.000 msec
    IPsec: Initialized Security Association Processing.
    hdacc0: <via vt1708s_4="" hda="" codec="">at cad 0 on hdac0
    hdaa0: <via vt1708s_4="" audio="" function="" group="">at nid 1 on hdacc0
    hdaa0: No presence detection support at nid 29
    pcm0: <via vt1708s_4="" (analog="" 2.0+hp="" 2.0)="">at nid 28,29 and 26,31,30,27 on hdaa0
    pcm1: <via vt1708s_4="" (rear-panel="" digital)="">at nid 32 on hdaa0
    random: unblocking device.
    usbus0: 12Mbps Full Speed USB v1.0
    usbus1: 12Mbps Full Speed USB v1.0
    usbus2: 12Mbps Full Speed USB v1.0
    usbus3: 480Mbps High Speed USB v2.0
    usbus4: 5.0Gbps Super Speed USB v3.0
    usbus5: 12Mbps Full Speed USB v1.0
    usbus6: 12Mbps Full Speed USB v1.0
    usbus7: 12Mbps Full Speed USB v1.0
    usbus8: 480Mbps High Speed USB v2.0
    ugen0.1: <intel>at usbus0
    uhub0: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus0
    ugen1.1: <intel>at usbus1
    uhub1: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus1
    ugen3.1: <intel>at usbus3
    uhub2: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus3
    ugen2.1: <intel>at usbus2
    uhub3: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus2
    ugen5.1: <intel>at usbus5
    uhub4: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus5
    ugen4.1: <0x1b21> at usbus4
    uhub5: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus4
    ugen7.1: <intel>at usbus7
    uhub6: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus7
    ugen6.1: <intel>at usbus6
    uhub7: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus6
    ugen8.1: <intel>at usbus8
    uhub8: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus8
    ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
    ses0: <ahci sgpio="" enclosure="" 1.00="" 0001="">SEMB S-E-S 2.00 device
    ses0: SEMB SES Device
    ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
    ada0: <st980412asg 0002sdm1="">ATA8-ACS SATA 2.x device
    ada0: Serial Number 5VF00YCC
    ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
    ada0: Command Queueing enabled
    ada0: 76319MB (156301488 512 byte sectors)
    ada0: Previously was known as ad4
    SMP: AP CPU #1 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #3 Launched!
    Timecounter "TSC" frequency 1800018072 Hz quality 1000
    uhub5: 4 ports with 4 removable, self powered
    Root mount waiting for: usbus8 usbus7 usbus6 usbus5 usbus3 usbus2 usbus1 usbus0
    uhub0: 2 ports with 2 removable, self powered
    uhub1: 2 ports with 2 removable, self powered
    uhub3: 2 ports with 2 removable, self powered
    uhub4: 2 ports with 2 removable, self powered
    uhub6: 2 ports with 2 removable, self powered
    uhub7: 2 ports with 2 removable, self powered
    Root mount waiting for: usbus8 usbus3
    Root mount waiting for: usbus8 usbus3
    uhub8: 6 ports with 6 removable, self powered
    uhub2: 6 ports with 6 removable, self powered
    Trying to mount root from ufs:/dev/ad4s1a [rw]...
    em1: promiscuous mode enabled
    carp: demoted by 240 to 240 (interface down)
    em2: promiscuous mode enabled
    carp: demoted by 240 to 480 (interface down)
    em0: promiscuous mode enabled
    carp: demoted by 240 to 720 (interface down)
    pflog0: promiscuous mode enabled
    carp: VHID 2@em0: INIT -> BACKUP
    carp: demoted by -240 to 480 (interface up)
    em0: link state changed to UP
    em3: link state changed to UP
    DUMMYNET 0 with IPv6 initialized (100409)
    load_dn_sched dn_sched FIFO loaded
    load_dn_sched dn_sched QFQ loaded
    load_dn_sched dn_sched RR loaded
    load_dn_sched dn_sched WF2Q+ loaded
    load_dn_sched dn_sched PRIO loaded
    carp: VHID 3@em2: INIT -> BACKUP
    carp: demoted by -240 to 240 (interface up)
    em2: link state changed to UP
    carp: VHID 1@em1: INIT -> BACKUP
    carp: demoted by -240 to 0 (interface up)
    em1: link state changed to UP
    carp: VHID 3@em2: BACKUP -> MASTER (preempting a slower master)
    carp: VHID 2@em0: BACKUP -> MASTER (master down)
    carp: VHID 1@em1: BACKUP -> MASTER (master down)
    pflog0: promiscuous mode disabled
    Waiting (max 60 seconds) for system process `vnlru' to stop...done
    Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
    Waiting (max 60 seconds) for system process `syncer' to stop...
    Syncing disks, vnodes remaining...0 0 done
    All buffers synced.
    Uptime: 31m40s
    Rebooting...
    Copyright (c) 1992-2016 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    	The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 10.3-RELEASE #6 05adf0a(RELENG_2_3_0): Mon Apr 11 18:52:07 CDT 2016
        root@ce23-amd64-builder:/builder/pfsense-230/tmp/obj/builder/pfsense-230/tmp/FreeBSD-src/sys/pfSense amd64
    FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
    CPU: Intel(R) Atom(TM) CPU D525   @ 1.80GHz (1800.02-MHz K8-class CPU)
      Origin="GenuineIntel"  Id=0x106ca  Family=0x6  Model=0x1c  Stepping=10
      Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x40e31d <sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe>AMD Features=0x20100800 <syscall,nx,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant, performance statistics
    real memory  = 4294967296 (4096 MB)
    avail memory = 4087820288 (3898 MB)
    Event timer "LAPIC" quality 400
    ACPI APIC Table: <031511 APIC1031>
    FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
    FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
     cpu0 (BSP): APIC ID:  0
     cpu1 (AP/HT): APIC ID:  1
     cpu2 (AP): APIC ID:  2
     cpu3 (AP/HT): APIC ID:  3
    random: <software, yarrow="">initialized
    ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/64 (20150515/tbfadt-644)
    ioapic0: Changing APIC ID to 4
    ioapic0 <version 2.0="">irqs 0-23 on motherboard
    wlan: mac acl policy registered
    netmap: loaded module
    kbd1 at kbdmux0
    cryptosoft0: <software crypto="">on motherboard
    padlock0: No ACE support.
    acpi0: <031511 XSDT1031> on motherboard
    acpi0: Power Button (fixed)
    cpu0: <acpi cpu="">on acpi0
    cpu1: <acpi cpu="">on acpi0
    cpu2: <acpi cpu="">on acpi0
    cpu3: <acpi cpu="">on acpi0
    attimer0: <at timer="">port 0x40-0x43 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
    Event timer "RTC" frequency 32768 Hz quality 0
    hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 14318180 Hz quality 950
    Event timer "HPET" frequency 14318180 Hz quality 450
    Event timer "HPET1" frequency 14318180 Hz quality 440
    Event timer "HPET2" frequency 14318180 Hz quality 440
    Event timer "HPET3" frequency 14318180 Hz quality 440
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    vgapci0: <vga-compatible display="">port 0xbc00-0xbc07 mem 0xfe580000-0xfe5fffff,0xd0000000-0xdfffffff,0xfe600000-0xfe6fffff irq 16 at device 2.0 on pci0
    agp0: <intel pineview="" svga="" controller="">on vgapci0
    agp0: aperture size is 256M, detected 8188k stolen memory
    vgapci0: Boot video device
    vgapci1: <vga-compatible display="">mem 0xfe480000-0xfe4fffff at device 2.1 on pci0
    uhci0: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb880-0xb89f irq 16 at device 26.0 on pci0
    usbus0 on uhci0
    uhci1: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb800-0xb81f irq 21 at device 26.1 on pci0
    usbus1 on uhci1
    uhci2: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb480-0xb49f irq 19 at device 26.2 on pci0
    usbus2 on uhci2
    ehci0: <intel 82801i="" (ich9)="" usb="" 2.0="" controller="">mem 0xfe577c00-0xfe577fff irq 18 at device 26.7 on pci0
    usbus3: EHCI version 1.0
    usbus3 on ehci0
    hdac0: <intel 82801i="" hda="" controller="">mem 0xfe570000-0xfe573fff irq 22 at device 27.0 on pci0
    pcib1: <acpi pci-pci="" bridge="">irq 17 at device 28.0 on pci0
    pci1: <acpi pci="" bus="">on pcib1
    pcib2: <acpi pci-pci="" bridge="">irq 18 at device 28.2 on pci0
    pci2: <acpi pci="" bus="">on pcib2
    em0: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k="">port 0xcc00-0xcc1f mem 0xfe7e0000-0xfe7fffff,0xfe7dc000-0xfe7dffff irq 18 at device 0.0 on pci2
    em0: Using MSIX interrupts with 3 vectors
    em0: Ethernet address: 00:30:18:a2:fb:bd
    em0: netmap queues/slots: TX 1/1024, RX 1/1024
    pcib3: <acpi pci-pci="" bridge="">irq 19 at device 28.3 on pci0
    pci3: <acpi pci="" bus="">on pcib3
    em1: <intel(r) 1000="" pro="" network="" connection="" 7.6.1-k="">port 0xdc00-0xdc1f mem 0xfe8e0000-0xfe8fffff,0xfe8dc000-0xfe8dffff irq 19 at device 0.0 on pci3
    em1: Using MSIX interrupts with 3 vectors
    em1: Ethernet address: 00:30:18:a2:fb:be
    em1: netmap queues/slots: TX 1/1024, RX 1/1024
    pcib4: <acpi pci-pci="" bridge="">irq 17 at device 28.4 on pci0
    pci4: <acpi pci="" bus="">on pcib4
    xhci0: <asmedia asm1042="" usb="" 3.0="" controller="">mem 0xfe9f8000-0xfe9fffff irq 16 at device 0.0 on pci4
    xhci0: 32 bytes context size, 32-bit DMA
    usbus4 on xhci0
    uhci3: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb400-0xb41f irq 23 at device 29.0 on pci0
    usbus5 on uhci3
    uhci4: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb080-0xb09f irq 19 at device 29.1 on pci0
    usbus6 on uhci4
    uhci5: <intel 82801i="" (ich9)="" usb="" controller="">port 0xb000-0xb01f irq 18 at device 29.2 on pci0
    usbus7 on uhci5
    ehci1: <intel 82801i="" (ich9)="" usb="" 2.0="" controller="">mem 0xfe577800-0xfe577bff irq 23 at device 29.7 on pci0
    usbus8: EHCI version 1.0
    usbus8 on ehci1
    pcib5: <acpi pci-pci="" bridge="">at device 30.0 on pci0
    pci5: <acpi pci="" bus="">on pcib5
    em2: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.1.0="">port 0xec00-0xec3f mem 0xfebe0000-0xfebfffff,0xfebc0000-0xfebdffff irq 18 at device 4.0 on pci5
    em2: Ethernet address: 00:30:18:a0:f2:a1
    em2: netmap queues/slots: TX 1/256, RX 1/256
    em3: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.1.0="">port 0xe880-0xe8bf mem 0xfeb80000-0xfeb9ffff,0xfeb60000-0xfeb7ffff irq 19 at device 6.0 on pci5
    em3: Ethernet address: 00:30:18:a0:f2:a2
    em3: netmap queues/slots: TX 1/256, RX 1/256
    em4: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.1.0="">port 0xe800-0xe83f mem 0xfeb20000-0xfeb3ffff,0xfeb00000-0xfeb1ffff irq 16 at device 7.0 on pci5
    em4: Ethernet address: 00:30:18:a0:f2:a3
    em4: netmap queues/slots: TX 1/256, RX 1/256
    isab0: <pci-isa bridge="">at device 31.0 on pci0
    isa0: <isa bus="">on isab0
    ahci0: <intel ich9="" ahci="" sata="" controller="">port 0xa400-0xa407,0xac00-0xac03,0xa880-0xa887,0xa800-0xa803,0xa480-0xa49f mem 0xfe577000-0xfe5777ff irq 19 at device 31.2 on pci0
    ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier not supported
    ahcich0: <ahci channel="">at channel 0 on ahci0
    ahcich1: <ahci channel="">at channel 1 on ahci0
    ahcich2: <ahci channel="">at channel 2 on ahci0
    ahcich3: <ahci channel="">at channel 3 on ahci0
    ahcich4: <ahci channel="">at channel 4 on ahci0
    ahcich5: <ahci channel="">at channel 5 on ahci0
    ahciem0: <ahci enclosure="" management="" bridge="">on ahci0
    acpi_button0: <power button="">on acpi0
    uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
    uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
    atkbdc0: <keyboard controller="" (i8042)="">port 0x60,0x64 irq 1 on acpi0
    atkbd0: <at keyboard="">irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    orm0: <isa option="" roms="">at iomem 0xce000-0xcefff,0xcf000-0xcffff,0xd0000-0xd0fff on isa0
    sc0: <system console="">at flags 0x100 on isa0
    sc0: VGA <16 virtual consoles, flags=0x300>
    vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    ppc0: cannot reserve I/O port range
    Timecounters tick every 1.000 msec
    IPsec: Initialized Security Association Processing.
    hdacc0: <via vt1708s_4="" hda="" codec="">at cad 0 on hdac0
    hdaa0: <via vt1708s_4="" audio="" function="" group="">at nid 1 on hdacc0
    hdaa0: No presence detection support at nid 29
    pcm0: <via vt1708s_4="" (analog="" 2.0+hp="" 2.0)="">at nid 28,29 and 26,31,30,27 on hdaa0
    pcm1: <via vt1708s_4="" (rear-panel="" digital)="">at nid 32 on hdaa0
    random: unblocking device.
    usbus0: 12Mbps Full Speed USB v1.0
    usbus1: 12Mbps Full Speed USB v1.0
    usbus2: 12Mbps Full Speed USB v1.0
    usbus3: 480Mbps High Speed USB v2.0
    usbus4: 5.0Gbps Super Speed USB v3.0
    usbus5: 12Mbps Full Speed USB v1.0
    usbus6: 12Mbps Full Speed USB v1.0
    usbus7: 12Mbps Full Speed USB v1.0
    usbus8: 480Mbps High Speed USB v2.0
    ugen1.1: <intel>at usbus1
    uhub0: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus1
    ugen0.1: <intel>at usbus0
    uhub1: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus0
    ugen3.1: <intel>at usbus3
    uhub2: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus3
    ugen2.1: <intel>at usbus2
    uhub3: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus2
    ugen5.1: <intel>at usbus5
    uhub4: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus5
    ugen4.1: <0x1b21> at usbus4
    uhub5: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus4
    ugen7.1: <intel>at usbus7
    uhub6: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus7
    ugen6.1: <intel>at usbus6
    uhub7: <intel 1="" 9="" uhci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus6
    ugen8.1: <intel>at usbus8
    uhub8: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus8
    ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
    ses0: <ahci sgpio="" enclosure="" 1.00="" 0001="">SEMB S-E-S 2.00 device
    ses0: SEMB SES Device
    ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
    ada0: <st980412asg 0002sdm1="">ATA8-ACS SATA 2.x device
    ada0: Serial Number 5VF00YCC
    ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
    ada0: Command Queueing enabled
    ada0: 76319MB (156301488 512 byte sectors)
    ada0: Previously was known as ad4
    SMP: AP CPU #1 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #3 Launched!
    Timecounter "TSC" frequency 1800017469 Hz quality 1000
    uhub5: 4 ports with 4 removable, self powered
    Root mount waiting for: usbus8 usbus7 usbus6 usbus5 usbus3 usbus2 usbus1 usbus0
    uhub0: 2 ports with 2 removable, self powered
    uhub1: 2 ports with 2 removable, self powered
    uhub3: 2 ports with 2 removable, self powered
    uhub4: 2 ports with 2 removable, self powered
    uhub6: 2 ports with 2 removable, self powered
    uhub7: 2 ports with 2 removable, self powered
    Root mount waiting for: usbus8 usbus3
    Root mount waiting for: usbus8 usbus3
    uhub8: 6 ports with 6 removable, self powered
    uhub2: 6 ports with 6 removable, self powered
    Trying to mount root from ufs:/dev/ad4s1a [rw]...
    em1: promiscuous mode enabled
    carp: demoted by 240 to 240 (interface down)
    em2: promiscuous mode enabled
    carp: demoted by 240 to 480 (interface down)
    em0: promiscuous mode enabled
    carp: demoted by 240 to 720 (interface down)
    carp: VHID 2@em0: INIT -> BACKUP
    carp: demoted by -240 to 480 (interface up)
    em0: link state changed to UP
    pflog0: promiscuous mode enabled
    em3: link state changed to UP
    DUMMYNET 0 with IPv6 initialized (100409)
    load_dn_sched dn_sched FIFO loaded
    load_dn_sched dn_sched QFQ loaded
    load_dn_sched dn_sched RR loaded
    load_dn_sched dn_sched WF2Q+ loaded
    load_dn_sched dn_sched PRIO loaded
    carp: VHID 3@em2: INIT -> BACKUP
    carp: demoted by -240 to 240 (interface up)
    em2: link state changed to UP
    carp: VHID 1@em1: INIT -> BACKUP
    carp: demoted by -240 to 0 (interface up)
    em1: link state changed to UP
    carp: VHID 3@em2: BACKUP -> MASTER (preempting a slower master)
    carp: VHID 2@em0: BACKUP -> MASTER (master down)
    carp: VHID 1@em1: BACKUP -> MASTER (master down)</st980412asg></ahci></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></via></via></via></via></generic></system></isa></at></keyboard></power></ahci></ahci></ahci></ahci></ahci></ahci></ahci></intel></isa></pci-isa></intel(r)></intel(r)></intel(r)></acpi></acpi></intel></intel></intel></intel></asmedia></acpi></acpi></intel(r)></acpi></acpi></intel(r)></acpi></acpi></acpi></acpi></intel></intel></intel></intel></intel></vga-compatible></intel></vga-compatible></acpi></acpi></high></at></at></acpi></acpi></acpi></acpi></software></version></software,></lahf></syscall,nx,lm></sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe></st980412asg></ahci></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></intel></via></via></via></via></generic></system></isa></at></keyboard></power></ahci></ahci></ahci></ahci></ahci></ahci></ahci></intel></isa></pci-isa></intel(r)></intel(r)></intel(r)></acpi></acpi></intel></intel></intel></intel></asmedia></acpi></acpi></intel(r)></acpi></acpi></intel(r)></acpi></acpi></acpi></acpi></intel></intel></intel></intel></intel></vga-compatible></intel></vga-compatible></acpi></acpi></high></at></at></acpi></acpi></acpi></acpi></software></version></software,></lahf></syscall,nx,lm></sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> 
    

  • Rebel Alliance Global Moderator

    "then I pinged 192.168.10.1 on the firewall and that failed."

    You mean pfsense could not ping its own interface?

    Looks like something changed and your openvpn reset?  You have gateways on your lan??

    "Reloading endpoints that may use LANGTW."

    So there are downstream networks from pfsense?  Could draw out your network?



  • Gateways:
    Name Gateway         Monitor         RTT          RTTsd Loss         Status Description
    WANGW 62.58.41.65 8.8.8.8         9.658ms 1.333ms  0.0% Online WAN Gateway
    LANGTW 192.168.1.253 192.168.1.253 0.586ms 0.333ms 0.0% Online Main Switch
    CiscoVPN 192.168.1.248 192.168.1.248 0.736ms 0.205ms 0.0% Online Cisco ASA 5505 MR

    Static Routes:

    192.168.10.0/24 LANGTW - 192.168.1.253 LAN Office Subnet  
    192.168.160.0/24 CiscoVPN - 192.168.1.248 LAN VPN Australia  
    192.168.20.0/24 LANGTW - 192.168.1.253 LAN DTP Subnet  
    192.168.30.0/23 LANGTW - 192.168.1.253 LAN R&D Subnet  
    192.168.40.0/24 LANGTW - 192.168.1.253 LAN Production Subnet  
    192.168.50.0/24 CiscoVPN - 192.168.1.248 LAN MR systems Subnet  
    192.168.60.0/24 LANGTW - 192.168.1.253 LAN Overige Subnet  
    192.168.70.0/24 CiscoVPN - 192.168.1.248 LAN VPN Clients Subnet

    The openvpn server is disabled.

    Apr 29 11:20:19	dpinger		send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 192.168.1.248 bind_addr 192.168.1.101 identifier "CiscoVPN "
    Apr 29 11:20:19	dpinger		send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 192.168.1.253 bind_addr 192.168.1.101 identifier "LANGTW "
    Apr 29 11:20:19	dpinger		send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr 8.8.8.8 bind_addr 62.58.41.67 identifier "WANGW "
    Apr 29 10:43:16	dpinger		CiscoVPN 192.168.1.248: Clear latency 729us stddev 198us loss 5%
    Apr 29 10:43:16	dpinger		LANGTW 192.168.1.253: Clear latency 564us stddev 270us loss 5%
    Apr 29 10:43:15	dpinger		WANGW 8.8.8.8: Clear latency 9368us stddev 621us loss 5%
    

  • Rebel Alliance Global Moderator

    Why do you have a LAN GW??  So all these networks are down stream?

    and you also have a vpn connection over the same network??



  • Sorry, but I don't know. As I said I'm new with this and the FW was already configured before I worked with this.


  • Rebel Alliance Global Moderator

    so can pfsense ping its gateways to get to these other networks when they go down??

    192.168.1.253 seems to be the gateway to get to these network.. What is that device?

    192.168.20.0/24  LANGTW - 192.168.1.253  LAN  DTP Subnet     
    192.168.30.0/23  LANGTW - 192.168.1.253  LAN  R&D Subnet     
    192.168.40.0/24  LANGTW - 192.168.1.253  LAN  Production Subnet

    And seems you also have carp setup?

    carp: VHID 3@em2: BACKUP -> MASTER (preempting a slower master)
    carp: VHID 2@em0: BACKUP -> MASTER (master down)
    carp: VHID 1@em1: BACKUP -> MASTER (master down)

    Your really going to have to draw up this network if your wanting help..  Looks like you also have loss on your network

    Apr 29 10:43:16 dpinger LANGTW 192.168.1.253: Clear latency 564us stddev 270us loss 5%

    So your rebooting pfsense when it can not get somewhere, or your rebooting some other server when it can not talk to pfsense?



  • The 192.168.1.253 is the main Switch. So every switch is connected to that one and the main switch is connected to both the firewalls.

    We have 2 Firewalls if the first reboots or turns off the second firewall will be the MASTER.

    Yes, I'm rebooting the first firewall when it cannot ping 8.8.8.8 or one of the subnets.



  • Are you using IPsec?


  • Rebel Alliance Global Moderator

    You sure about that?  This seems to indicate that your master is down

    carp: VHID 1@em1: BACKUP -> MASTER (master down)

    Is that off the pfsense that you didn't reboot?

    First thing I would suggest vs rebooting.. Is actually look to why it can not talk to whatever..  What are the errors in the log when it can not talk.. In carp setup it should failover if there is an issue with it, etc..  So sounds like you have a problem with carp not working as well.



  • Yes, I'm using upsec. See attachment



    ![Ipsec rules.png](/public/imported_attachments/1/Ipsec rules.png)
    ![Ipsec rules.png_thumb](/public/imported_attachments/1/Ipsec rules.png_thumb)



  • It is off the pfsense that I rebooted.

    And I think your right about the CARP. The second firewall only takes over if the first firewall turns off or reboots.



  • Guessing you're hitting the issue described here.
    https://forum.pfsense.org/index.php?topic=110710.msg618388#msg618388

    which we're working on tracking down. The workaround there should suffice in the mean time for the few who are routinely hitting that.



  • I see that 2.3.1 is released. Will this update solve the problem that we have to run on 1 cpu core to ensure internet connection?



  • @Gadmin:

    I see that 2.3.1 is released. Will this update solve the problem

    Yes, 2.3.1 and newer fix that issue.