[Solved] WAN Down after upgrade 2.2–>2.2.1



  • Hi Guys,

    Everything was working fine in my 2.2 pfSense box. I upgrade it to 2.2.1 and  :o no WAN/Internet anymore.
    What I did:

    1. Check Firewall and NAT configuration –> Result: nothing change from 2.2  before
    2. Reboot pfSense and Modem --> no joy
    3. Test Modem with a second PC --> Modem is OK
    4. WireShark the Modem<->WAN traffic --> Nothing is coming out from WAN, it is completely dead
    5. Capture package for WAN in pfSense --> Nothing is showing up
    6. WAN is up with a green arrow according to pfSense main page WTF?
    7. Try to swap WAN msk0 with an OPT em2 NIC Port --> WAN remains dead (not related to driver)
      /8) Finally I checked the LOGs --> Yes I am still learning how to fight with pfSense

    and finally I found something interesting:

    kernel: arpresolve: can't allocate llinfo for 192.168.178.1 on msk0
    

    every singe second…(yes msk0 is the WAN NIC) I think I got the problem.

    Here some console jazz:

    [2.2.1-RELEASE][root@appasense.localdomain]/root: netstat -rn
    Routing tables
    
    Internet:
    Destination        Gateway            Flags      Netif Expire
    127.0.0.1          link#8             UH          lo0
    172.27.36.0/24     link#3             U           em1
    172.27.36.1        link#3             UHS         lo0
    172.27.37.0/24     link#2             U           em0
    172.27.37.1        link#2             UHS         lo0
    172.27.38.0/24     link#4             U           em2
    172.27.38.1        link#4             UHS         lo0
    198.168.178.0/24   link#1             U          msk0
    198.168.178.2      link#1             UHS         lo0
    
    Internet6:
    Destination                       Gateway                       Flags      Netif Expire
    ::1                               link#8                        UH          lo0
    fe80::%msk0/64                    link#1                        U          msk0
    fe80::22cf:30ff:fe4c:39da%msk0    link#1                        UHS         lo0
    fe80::%em0/64                     link#2                        U           em0
    fe80::215:17ff:fe37:c73a%em0      link#2                        UHS         lo0
    fe80::%em1/64                     link#3                        U           em1
    fe80::215:17ff:fe37:c73b%em1      link#3                        UHS         lo0
    fe80::%em2/64                     link#4                        U           em2
    fe80::6a05:caff:fe1e:701f%em2     link#4                        UHS         lo0
    fe80::%lo0/64                     link#8                        U           lo0
    fe80::1%lo0                       link#8                        UHS         lo0
    ff01::%msk0/32                    fe80::22cf:30ff:fe4c:39da%msk0 U          msk0
    ff01::%em0/32                     fe80::215:17ff:fe37:c73a%em0  U           em0
    ff01::%em1/32                     fe80::215:17ff:fe37:c73b%em1  U           em1
    ff01::%em2/32                     fe80::6a05:caff:fe1e:701f%em2 U           em2
    ff01::%lo0/32                     ::1                           U           lo0
    ff02::%msk0/32                    fe80::22cf:30ff:fe4c:39da%msk0 U          msk0
    ff02::%em0/32                     fe80::215:17ff:fe37:c73a%em0  U           em0
    ff02::%em1/32                     fe80::215:17ff:fe37:c73b%em1  U           em1
    ff02::%em2/32                     fe80::6a05:caff:fe1e:701f%em2 U           em2
    ff02::%lo0/32                     ::1                           U           lo0
    

    and

    [2.2.1-RELEASE][root@appasense.localdomain]/root: sysctl dev.msk.0
    dev.msk.0.%desc: Marvell Technology Group Ltd. Yukon Optima Id 0xbc Rev 0x01
    dev.msk.0.%driver: msk
    dev.msk.0.%location:
    dev.msk.0.%pnpinfo:
    dev.msk.0.%parent: mskc0
    dev.msk.0.stats.rx.ucast_frames: 0
    dev.msk.0.stats.rx.bcast_frames: 1279
    dev.msk.0.stats.rx.pause_frames: 0
    dev.msk.0.stats.rx.mcast_frames: 86
    dev.msk.0.stats.rx.crc_errs: 0
    dev.msk.0.stats.rx.good_octets: 131258
    dev.msk.0.stats.rx.bad_octets: 0
    dev.msk.0.stats.rx.frames_64: 792
    dev.msk.0.stats.rx.frames_65_127: 455
    dev.msk.0.stats.rx.frames_128_255: 68
    dev.msk.0.stats.rx.frames_256_511: 42
    dev.msk.0.stats.rx.frames_512_1023: 8
    dev.msk.0.stats.rx.frames_1024_1518: 0
    dev.msk.0.stats.rx.frames_1519_max: 0
    dev.msk.0.stats.rx.frames_too_long: 0
    dev.msk.0.stats.rx.jabbers: 0
    dev.msk.0.stats.rx.overflows: 0
    dev.msk.0.stats.tx.ucast_frames: 0
    dev.msk.0.stats.tx.bcast_frames: 1
    dev.msk.0.stats.tx.pause_frames: 2
    dev.msk.0.stats.tx.mcast_frames: 4
    dev.msk.0.stats.tx.octets: 616
    dev.msk.0.stats.tx.frames_64: 3
    dev.msk.0.stats.tx.frames_65_127: 3
    dev.msk.0.stats.tx.frames_128_255: 1
    dev.msk.0.stats.tx.frames_256_511: 0
    dev.msk.0.stats.tx.frames_512_1023: 0
    dev.msk.0.stats.tx.frames_1024_1518: 0
    dev.msk.0.stats.tx.frames_1519_max: 0
    dev.msk.0.stats.tx.colls: 0
    dev.msk.0.stats.tx.late_colls: 0
    dev.msk.0.stats.tx.excess_colls: 0
    dev.msk.0.stats.tx.multi_colls: 0
    dev.msk.0.stats.tx.single_colls: 0
    dev.msk.0.stats.tx.underflows: 0
    

    Now what to do?

    Thanks in advance,

    PS: This Weekend I will be out, excuse me if I don't follow up with you.



  • According to your 'netstat' output, you're missing your default gateway entry. This should read 'Destination: default', with your WAN gateway set as the route out. Check your routing config and make sure your default gateway is set to the router interface.



  • Thanks muswellhillbilly you are right. I will ask to a friend who has access to my pfSense box to check that.
    It is the first time I do a pfSense upgrade, very strange to my eyes that something like this could happen.

    Before everything was fine, I am 100% sure the default gateway was there.  ???



  • Further investigations… I now believe the problem is not the "default gateway" but apinger...

    Here my WAN Interface:

    and my (drammatic) gateway situation:

    not good.

    So let´s start by restoring the previuous config when everything was working in the 2.2.
    Done, let's now check dmesg in the SSH console:

    Timecounter "TSC-low" frequency 1655616379 Hz quality 800
    Trying to mount root from ufs:/dev/ada0s1a [rw]...
    msk0: link state changed to DOWN
    pflog0: promiscuous mode enabled
    em2: link state changed to UP
    msk0: link state changed to UP
    em1: link state changed to UP
    msk0: watchdog timeout
    msk0: link state changed to DOWN
    msk0: link state changed to UP
    msk0: watchdog timeout
    msk0: link state changed to DOWN
    msk0: link state changed to UP
    [2.2.1-RELEASE][root@appasense.localdomain]/root:
    

    That watchdog timeout is like a punch in the eye, let´s reboot the machine and see.
    dmesg after reboot:

    cpu4 (AP): APIC ID:  4
     cpu5 (AP): APIC ID:  5
    ioapic0 <version 2.1=""> irqs 0-23 on motherboard
    ioapic1 <version 2.1=""> irqs 24-55 on motherboard
    wlan: mac acl policy registered
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /bo                                                                                                        ot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8060b270, 0) error 1
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /b                                                                                                        oot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8060b320, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw                                                                                                        /.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in                                                                                                         /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8060b3d0, 0) error 1
    iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /bo                                                                                                        ot/loader.conf.
    module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff80632a40, 0) error 1
    iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /b                                                                                                        oot/loader.conf.
    module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80632af0, 0) error 1
    iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi                                                                                                        /.
    iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in                                                                                                         /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80632ba0, 0) error 1
    random: <software, yarrow=""> initialized
    kbd1 at kbdmux0
    cryptosoft0: <software crypto=""> on motherboard
    padlock0: No ACE support.
    acpi0: <091412 XSDT1501> on motherboard
    acpi0: Power Button (fixed)
    acpi0: reservation of fec00000, 1000 (3) failed
    acpi0: reservation of fee00000, 1000 (3) failed
    acpi0: reservation of ffb80000, 80000 (3) failed
    acpi0: reservation of fec10000, 20 (3) failed
    acpi0: reservation of fed80000, 1000 (3) failed
    acpi0: reservation of 0, a0000 (3) failed
    acpi0: reservation of 100000, dfe00000 (3) failed
    cpu0: <acpi cpu=""> on acpi0
    cpu1: <acpi cpu=""> on acpi0
    cpu2: <acpi cpu=""> on acpi0
    cpu3: <acpi cpu=""> on acpi0
    cpu4: <acpi cpu=""> on acpi0
    cpu5: <acpi cpu=""> on acpi0
    attimer0: <at timer=""> port 0x40-0x43 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    atrtc0: <at realtime="" clock=""> port 0x70-0x71 irq 8 on acpi0
    Event timer "RTC" frequency 32768 Hz quality 0
    hpet0: <high precision="" event="" timer=""> iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 14318180 Hz quality 950
    Event timer "HPET" frequency 14318180 Hz quality 450
    Event timer "HPET1" frequency 14318180 Hz quality 450
    Event timer "HPET2" frequency 14318180 Hz quality 450
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
    acpi_ec0: <embedded controller:="" gpe="" 0xa=""> port 0x62,0x66 on acpi0
    pcib0: <acpi host-pci="" bridge=""> port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus=""> on pcib0
    pcib1: <acpi pci-pci="" bridge=""> irq 52 at device 4.0 on pci0
    pci6: <acpi pci="" bus=""> on pcib1
    atapci0: <jmicron jmb363="" udma133="" controller=""> port 0xec00-0xec07,0xe880-0xe883,0x                                                                                                        e800-0xe807,0xe480-0xe483,0xe400-0xe40f mem 0xfe9fe000-0xfe9fffff irq 44 at devi                                                                                                        ce 0.0 on pci6
    ahci0: <jmicron jmb363="" ahci="" sata="" controller=""> at channel -1 on atapci0
    ahci0: AHCI v1.00 with 2 3Gbps ports, Port Multiplier supported
    ahci0: quirks=0x1 <noforce>ahcich0: <ahci channel=""> at channel 0 on ahci0
    ahcich1: <ahci channel=""> at channel 1 on ahci0
    ata2: <ata channel=""> at channel 0 on atapci0
    pcib2: <acpi pci-pci="" bridge=""> irq 53 at device 6.0 on pci0
    pci5: <acpi pci="" bus=""> on pcib2
    mskc0: <marvell yukon="" 88e8059="" gigabit="" ethernet=""> port 0xd800-0xd8ff mem 0xfe8fc00                                                                                                        0-0xfe8fffff irq 51 at device 0.0 on pci5
    msk0: <marvell technology="" group="" ltd.="" yukon="" optima="" id="" 0xbc="" rev="" 0x01=""> on mskc0
    miibus0: <mii bus=""> on msk0
    e1000phy0: <marvell phyg65g="" gigabit="" phy=""> PHY 0 on miibus0
    e1000phy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 100                                                                                                        0baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
    pcib3: <acpi pci-pci="" bridge=""> irq 53 at device 7.0 on pci0
    pci4: <acpi pci="" bus=""> on pcib3
    xhci0: <nec upd720200="" usb="" 3.0="" controller=""> mem 0xfe7fe000-0xfe7fffff irq 50 at de                                                                                                        vice 0.0 on pci4
    xhci0: 32 byte context size.
    usbus0 on xhci0
    pcib4: <acpi pci-pci="" bridge=""> irq 54 at device 10.0 on pci0
    pci3: <acpi pci="" bus=""> on pcib4
    em0: <intel(r) 1000="" pro="" network="" connection="" 7.4.2=""> port 0xcc00-0xcc1f mem 0xfe6e0                                                                                                        000-0xfe6fffff,0xfe6c0000-0xfe6dffff irq 47 at device 0.0 on pci3
    em0: Using an MSI interrupt
    em1: <intel(r) 1000="" pro="" network="" connection="" 7.4.2=""> port 0xc880-0xc89f mem 0xfe6a0                                                                                                        000-0xfe6bffff,0xfe680000-0xfe69ffff irq 44 at device 0.1 on pci3
    em1: Using an MSI interrupt
    pcib5: <acpi pci-pci="" bridge=""> irq 54 at device 13.0 on pci0
    pci2: <acpi pci="" bus=""> on pcib5
    em2: <intel(r) 1000="" pro="" network="" connection="" 7.4.2=""> port 0xbc00-0xbc1f mem 0xfe5e0                                                                                                        000-0xfe5fffff,0xfe500000-0xfe57ffff,0xfe5dc000-0xfe5dffff irq 40 at device 0.0                                                                                                         on pci2
    em2: Using MSIX interrupts with 3 vectors
    ahci1: <amd sb7x0="" sb8x0="" sb9x0="" ahci="" sata="" controller=""> port 0xa000-0xa007,0x9000-0x                                                                                                        9003,0x8000-0x8007,0x7000-0x7003,0x6000-0x600f mem 0xfe3fe000-0xfe3fe3ff irq 19                                                                                                         at device 17.0 on pci0
    ahci1: AHCI v1.20 with 6 6Gbps ports, Port Multiplier supported
    ahcich2: <ahci channel=""> at channel 0 on ahci1
    ahcich3: <ahci channel=""> at channel 1 on ahci1
    ahcich4: <ahci channel=""> at channel 2 on ahci1
    ahcich5: <ahci channel=""> at channel 3 on ahci1
    ahcich6: <ahci channel=""> at channel 4 on ahci1
    ahcich7: <ahci channel=""> at channel 5 on ahci1
    ohci0: <amd sb7x0="" sb8x0="" sb9x0="" usb="" controller=""> mem 0xfe3fb000-0xfe3fbfff irq 18 a                                                                                                        t device 18.0 on pci0
    usbus1 on ohci0
    ehci0: <amd sb7x0="" sb8x0="" sb9x0="" usb="" 2.0="" controller=""> mem 0xfe3fe400-0xfe3fe4ff irq                                                                                                         17 at device 18.2 on pci0
    usbus2: EHCI version 1.0
    usbus2 on ehci0
    ohci1: <amd sb7x0="" sb8x0="" sb9x0="" usb="" controller=""> mem 0xfe3fc000-0xfe3fcfff irq 18 a                                                                                                        t device 19.0 on pci0
    usbus3 on ohci1
    ehci1: <amd sb7x0="" sb8x0="" sb9x0="" usb="" 2.0="" controller=""> mem 0xfe3fe800-0xfe3fe8ff irq                                                                                                         17 at device 19.2 on pci0
    usbus4: EHCI version 1.0
    usbus4 on ehci1
    isab0: <pci-isa bridge=""> at device 20.3 on pci0
    isa0: <isa bus=""> on isab0
    pcib6: <acpi pci-pci="" bridge=""> at device 20.4 on pci0
    pci1: <acpi pci="" bus=""> on pcib6
    vgapci0: <vga-compatible display=""> irq 20 at device 5.0 on pci1
    vgapci0: Boot video device
    ral0: <ralink technology="" rt2860=""> mem 0xfe4f0000-0xfe4fffff irq 21 at device 6.0                                                                                                         on pci1
    ohci2: <amd sb7x0="" sb8x0="" sb9x0="" usb="" controller=""> mem 0xfe3fd000-0xfe3fdfff irq 18 a                                                                                                        t device 20.5 on pci0
    usbus5 on ohci2
    ohci3: <amd sb7x0="" sb8x0="" sb9x0="" usb="" controller=""> mem 0xfe3ff000-0xfe3fffff irq 18 a                                                                                                        t device 22.0 on pci0
    usbus6 on ohci3
    ehci2: <amd sb7x0="" sb8x0="" sb9x0="" usb="" 2.0="" controller=""> mem 0xfe3fec00-0xfe3fecff irq                                                                                                         17 at device 22.2 on pci0
    usbus7: EHCI version 1.0
    usbus7 on ehci2
    acpi_button0: <power button=""> on acpi0
    orm0: <isa option="" roms=""> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0
    gpioapu0: Environment returned System Product Name
    sc0: <system console=""> at flags 0x100 on isa0
    sc0: VGA <16 virtual consoles, flags=0x300>
    vga0: <generic isa="" vga=""> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    atkbdc0: <keyboard controller="" (i8042)=""> at port 0x60,0x64 on isa0
    atkbd0: <at keyboard=""> irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    fdc0: <enhanced floppy="" controller=""> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
    ppc0: cannot reserve I/O port range
    acpi_throttle0: <acpi cpu="" throttling=""> on cpu0
    hwpstate0: <cool`n'quiet 2.0=""> on cpu0
    Timecounters tick every 1.000 msec
    IPsec: Initialized Security Association Processing.
    random: unblocking device.
    usbus0: 5.0Gbps Super Speed USB v3.0
    usbus1: 12Mbps Full Speed USB v1.0
    usbus2: 480Mbps High Speed USB v2.0
    usbus3: 12Mbps Full Speed USB v1.0
    usbus4: 480Mbps High Speed USB v2.0
    usbus5: 12Mbps Full Speed USB v1.0
    usbus6: 12Mbps Full Speed USB v1.0
    usbus7: 480Mbps High Speed USB v2.0
    ugen0.1: <0x1033> at usbus0
    uhub0: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
    ugen1.1: <ati> at usbus1
    uhub1: <ati 1="" 9="" ohci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus1
    ugen3.1: <ati> at usbus3
    uhub2: <ati 1="" 9="" ohci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus3
    ugen2.1: <ati> at usbus2
    uhub3: <ati 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus2
    ugen4.1: <ati> at usbus4
    uhub4: <ati 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus4
    ugen5.1: <ati> at usbus5
    uhub5: <ati 1="" 9="" ohci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus5
    ugen7.1: <ati> at usbus7
    uhub6: <ati 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus7
    ugen6.1: <ati> at usbus6
    uhub7: <ati 1="" 9="" ohci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus6
    uhub5: 2 ports with 2 removable, self powered
    uhub7: 4 ports with 4 removable, self powered
    uhub1: 5 ports with 5 removable, self powered
    uhub2: 5 ports with 5 removable, self powered
    uhub0: 4 ports with 4 removable, self powered
    uhub6: 4 ports with 4 removable, self powered
    uhub3: 5 ports with 5 removable, self powered
    uhub4: 5 ports with 5 removable, self powered
    ada0 at ahcich2 bus 0 scbus3 target 0 lun 0
    ada0: <samsung 840="" ssd="" evo="" 120gb="" ext0bb6q=""> ATA-9 SATA 3.x device
    ada0: Serial Number S1D5NSADC69928Z
    ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
    ada0: Command Queueing enabled
    ada0: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C)
    ada0: quirks=0x1<4K>
    ada0: Previously was known as ad10
    ada1 at ahcich3 bus 0 scbus4 target 0 lun 0
    ada1: <wdc wd1200bevs-75ust0="" 01.01a01=""> ATA-8 SATA 1.x device
    ada1: Serial Number WD-WXE907G52186
    ada1: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes)
    ada1: Command Queueing enabled
    ada1: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C)
    ada1: Previously was known as ad12
    SMP: AP CPU #3 Launched!
    SMP: AP CPU #1 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #4 Launched!
    SMP: AP CPU #5 Launched!
    Timecounter "TSC-low" frequency 1655616593 Hz quality 800
    Trying to mount root from ufs:/dev/ada0s1a [rw]...
    msk0: link state changed to DOWN
    pflog0: promiscuous mode enabled
    em2: link state changed to UP
    msk0: link state changed to UP
    em1: link state changed to UP</wdc></samsung></ati></ati></ati></ati></ati></ati></ati></ati></ati></ati></ati></ati></ati></ati></cool`n'quiet></acpi></enhanced></at></keyboard></generic></system></isa></power></amd></amd></amd></ralink></vga-compatible></acpi></acpi></isa></pci-isa></amd></amd></amd></amd></ahci></ahci></ahci></ahci></ahci></ahci></amd></intel(r)></acpi></acpi></intel(r)></intel(r)></acpi></acpi></nec></acpi></acpi></marvell></mii></marvell></marvell></acpi></acpi></ata></ahci></ahci></noforce></jmicron></jmicron></acpi></acpi></acpi></acpi></embedded></high></at></at></acpi></acpi></acpi></acpi></acpi></acpi></software></software,></version></version>
    

    mmmm don´t like the module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8060b320, 0) error 1, now I have to google what does it means. Anyway it looks like is not related to the gateway problem.

    Let´s check now the routing tables:

    [2.2.1-RELEASE][root@appasense.localdomain]/root: netstat -rn
    Routing tables
    
    Internet:
    Destination        Gateway            Flags      Netif Expire
    default            192.168.178.1      UGS        msk0
    127.0.0.1          link#8             UH          lo0
    172.27.36.0/24     link#3             U           em1
    172.27.36.1        link#3             UHS         lo0
    172.27.37.0/24     link#2             U           em0
    172.27.37.1        link#2             UHS         lo0
    172.27.38.0/24     link#4             U           em2
    172.27.38.1        link#4             UHS         lo0
    192.168.178.0/24   link#1             U          msk0
    192.168.178.2      link#1             UHS         lo0
    
    Internet6:
    Destination                       Gateway                       Flags      Netif Expire
    ::1                               link#8                        UH          lo0
    fe80::%msk0/64                    link#1                        U          msk0
    fe80::22cf:30ff:fe4c:39da%msk0    link#1                        UHS         lo0
    fe80::%em0/64                     link#2                        U           em0
    fe80::215:17ff:fe37:c73a%em0      link#2                        UHS         lo0
    fe80::%em1/64                     link#3                        U           em1
    fe80::215:17ff:fe37:c73b%em1      link#3                        UHS         lo0
    fe80::%em2/64                     link#4                        U           em2
    fe80::6a05:caff:fe1e:701f%em2     link#4                        UHS         lo0
    fe80::%lo0/64                     link#8                        U           lo0
    fe80::1%lo0                       link#8                        UHS         lo0
    ff01::%msk0/32                    fe80::22cf:30ff:fe4c:39da%msk0 U          msk0
    ff01::%em0/32                     fe80::215:17ff:fe37:c73a%em0  U           em0
    ff01::%em1/32                     fe80::215:17ff:fe37:c73b%em1  U           em1
    ff01::%em2/32                     fe80::6a05:caff:fe1e:701f%em2 U           em2
    ff01::%lo0/32                     ::1                           U           lo0
    ff02::%msk0/32                    fe80::22cf:30ff:fe4c:39da%msk0 U          msk0
    ff02::%em0/32                     fe80::215:17ff:fe37:c73a%em0  U           em0
    ff02::%em1/32                     fe80::215:17ff:fe37:c73b%em1  U           em1
    ff02::%em2/32                     fe80::6a05:caff:fe1e:701f%em2 U           em2
    ff02::%lo0/32                     ::1                           U           lo0
    

    …and default is now there but the gateway is still down and red as before…

    What about looking the apinger logs?

    Mar 23 23:33:00 	apinger: ALARM: WANGW(192.168.178.1) *** down ***
    Mar 23 23:32:50 	apinger: Starting Alarm Pinger, apinger(48668)
    Mar 23 23:32:38 	apinger: Exiting on signal 15.
    Mar 23 22:54:18 	apinger: Starting Alarm Pinger, apinger(12521)
    Mar 23 22:45:11 	apinger: SIGHUP received, reloading configuration.
    Mar 23 22:44:39 	apinger: Starting Alarm Pinger, apinger(11998)
    Mar 23 22:36:13 	apinger: Starting Alarm Pinger, apinger(11883)
    Mar 22 23:16:03 	apinger: Starting Alarm Pinger, apinger(11790)
    Mar 19 21:57:21 	apinger: Starting Alarm Pinger, apinger(10679)
    Mar 19 21:09:28 	apinger: Starting Alarm Pinger, apinger(11363)
    Mar 19 20:14:33 	apinger: Starting Alarm Pinger, apinger(10557)
    Mar 19 19:53:13 	apinger: Starting Alarm Pinger, apinger(11210)
    Mar 19 19:51:52 	apinger: ALARM: WANGW(192.168.178.1) *** down ***
    Mar 19 19:51:42 	apinger: Starting Alarm Pinger, apinger(16748)
    Mar 19 19:51:10 	apinger: No usable targets found, exiting
    Mar 19 19:51:10 	apinger: Starting Alarm Pinger, apinger(21440)
    Mar 19 19:51:00 	apinger: No usable targets found, exiting
    Mar 19 19:51:00 	apinger: alarm canceled (config reload): WANGW(192.168.178.1) *** down ***
    Mar 19 19:51:00 	apinger: SIGHUP received, reloading configuration.
    Mar 19 19:50:42 	apinger: SIGHUP received, reloading configuration.
    Mar 19 19:48:11 	apinger: ALARM: WANGW(192.168.178.1) *** down ***
    Mar 19 19:48:01 	apinger: Starting Alarm Pinger, apinger(30239)
    Mar 19 19:19:14 	apinger: No usable targets found, exiting
    Mar 19 19:19:14 	apinger: alarm canceled (config reload): WANGW(192.168.178.1) *** down ***
    Mar 19 19:19:14 	apinger: SIGHUP received, reloading configuration.
    Mar 19 19:19:13 	apinger: Could not bind socket on address(192.168.178.2) for monitoring address 192.168.178.1(WANGW) with error Can't assign requested address
    Mar 19 19:19:12 	apinger: Could not bind socket on address(192.168.178.2) for monitoring address 192.168.178.1(WANGW) with error Can't assign requested address
    Mar 19 19:19:11 	apinger: Could not bind socket on address(192.168.178.2) for monitoring address 192.168.178.1(WANGW) with error Can't assign requested address
    Mar 19 19:19:10 	apinger: Could not bind socket on address(192.168.178.2) for monitoring address 192.168.178.1(WANGW) with error Can't assign requested address
    Mar 19 19:19:09 	apinger: Could not bind socket on address(192.168.178.2) for monitoring address 192.168.178.1(WANGW) with error Can't assign requested address
    Mar 19 19:19:08 	apinger: Could not bind socket on address(192.168.178.2) for monitoring address 192.168.178.1(WANGW) with error Can't assign requested address
    Mar 19 19:15:20 	apinger: SIGHUP received, reloading configuration.
    Mar 19 07:04:08 	apinger: ALARM: WANGW(192.168.178.1) *** down ***
    Mar 19 07:03:57 	apinger: alarm canceled (config reload): WANGW(192.168.178.1) *** down ***
    Mar 19 07:03:57 	apinger: SIGHUP received, reloading configuration.
    Mar 19 07:03:55 	apinger: Could not bind socket on address(192.168.178.3) for monitoring address 192.168.178.1(WANGW) with error Can't assign requested address
    Mar 19 07:03:19 	apinger: ALARM: WANGW(192.168.178.1) *** down ***
    Mar 19 07:03:08 	apinger: SIGHUP received, reloading configuration.
    Mar 19 07:03:08 	apinger: alarm canceled (config reload): WANGW(192.168.178.1) *** down ***
    Mar 19 07:03:08 	apinger: SIGHUP received, reloading configuration.
    Mar 19 07:03:05 	apinger: Could not bind socket on address(192.168.178.2) for monitoring address 192.168.178.1(WANGW) with error Can't assign requested address
    Mar 19 07:02:06 	apinger: SIGHUP received, reloading configuration.
    Mar 19 01:09:32 	apinger: SIGHUP received, reloading configuration.
    Mar 19 01:06:40 	apinger: ALARM: WANGW(192.168.178.1) *** down ***
    Mar 19 01:06:30 	apinger: Starting Alarm Pinger, apinger(9753)
    Mar 19 01:05:11 	apinger: No usable targets found, exiting
    Mar 19 01:05:11 	apinger: alarm canceled (config reload): WANGW(192.168.178.1) *** down ***
    Mar 19 01:05:11 	apinger: SIGHUP received, reloading configuration.
    Mar 19 01:05:11 	apinger: Could not bind socket on address(192.168.178.2) for monitoring address 192.168.178.1(WANGW) with error Can't assign requested address
    Mar 19 01:05:10 	apinger: Could not bind socket on address(192.168.178.2) for monitoring address 192.168.178.1(WANGW) with error Can't assign requested address
    

    Not good, I would like to speak with the service apinger and ask if it has smoked some wired stuff…
    For tonight is now enough, I need to sleep.

    PS: I am tempted to format the machine and reinstall everything from scratch (first the 2.2 and then upgrade to 2.2.1 and see if it is a reproducible issue...)






  • Hi guys
    I'm with the same problem.
    Its like a network driver problem.

    This problem was solved on a old version, but appears again.
    https://forum.pfsense.org/index.php?topic=50489.0

    is there any other possibility to correct this problem? I applied the configuration on /boot/loader.conf.local but same issue



  • Hi rolandosf,

    do you have a msk0 NIC too?

    BTW I swapped the fmsk0 NIC with a em0 NIC in the WAN interface and still no joy. I can redo the test and report some data/logs if needed.

    Take care,

    Z


  • Netgate Administrator

    You appear to have typo'd the WAN address as 198 instead of 192. Thus the gateway is now outside the wan subnet.

    Additionally to avoid those watchdog timeouts on the msk interface you should set hw.msk.msi_disable=1. You can do that by running this command at the console:

    echo 'hw.msk.msi_disable=1' >> /boot/loader.conf.local
    

    You will have to reboot for that to take effect.

    Steve



  • I want to kill myself. I have no idea why yesterday the WAN was 198.xxxxxxxx, as I turned on the pfSense box today I got:

    I tried the hw.msk.msi_disable=1 trick and now:

    
    SMP: AP CPU #1 Launched!
    SMP: AP CPU #3 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #5 Launched!
    SMP: AP CPU #4 Launched!
    Timecounter "TSC-low" frequency 1655612688 Hz quality 800
    Trying to mount root from ufs:/dev/ada0s1a [rw]...
    msk0: link state changed to DOWN
    pflog0: promiscuous mode enabled
    em2: link state changed to UP
    msk0: link state changed to UP
    em1: link state changed to UP
    msk0: watchdog timeout
    msk0: link state changed to DOWN
    msk0: link state changed to UP
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    interrupt storm detected on "irq51:"; throttling interrupt source
    
    

    here the final result:

    back to square 1.

    I think Lucifer is now the master of my pfSense, I need to go back to 2.2






  • Today I reinstalled the 2.2.1 fresh. Still no joy. the msk0 driver was doing his show and taking the WAN port down.
    When I use an Intel NIC em in the everything is cool. I ordered a new Dual Intel NIC.

    Lesson learned:

    1. FreeBSD needs Intel NIC, all the others will ruin your day soon or later.
    2. Never trust the automatic pfSense update.

Log in to reply