4. Pfsense start problematic, need some advise

Pfsense start problematic, need some advise

  • Y

    been using it for long time, no problem at all.

    yesterday night, network start unreachable.
    i can login the pfsense webUI, WAN got IP, but can NOT ping (dns is well config)

    Crash reporter:

    Crash report begins.  Anonymous machine information:

    amd64
    11.1-RELEASE-p7
    FreeBSD 11.1-RELEASE-p7 #10 r313908+986837ba7e9(RELENG_2_4): Mon Mar 26 18:08:25 CDT 2018    root@buildbot2.netgate.com:/builder/ce-243/tmp/obj/builder/ce-243/tmp/FreeBSD-src/sys/pfSense

    Crash report details:

    PHP Errors:
    [09-May-2018 14:01:14 Asia/Hong_Kong] PHP Parse error:  syntax error, unexpected end of file in /etc/inc/system.inc on line 2234
    [09-May-2018 14:01:14 Asia/Hong_Kong] PHP Stack trace:
    [09-May-2018 14:01:14 Asia/Hong_Kong] PHP  1. {main}() /etc/ecl.php:0
    [09-May-2018 14:01:14 Asia/Hong_Kong] PHP  2. require_once() /etc/ecl.php:21

    No FreeBSD crash data found.

    FOUND:

    /var/run
    108% of 3.4MiB - ufs in RAM

    Shell Output - ls -la /var/run
    -rw–-----  1 root  wheel    3407872 May  9 14:01 cpucontrol.core
    rm /var/run/cpucontrol.core
    Seems fixed this part but cant find the cause

    VPN server can not login with iOS device (just upgrade to 11.3.1 last night)
    confirm the username and passw0rd are correct, even with new client config, still failed
    log:

    May 9 14:15:12 openvpn 70379 192.168.5.238:60610 TLS Auth Error: Auth Username/Password verification failed for peer
    May 9 14:15:12 openvpn 70379 192.168.5.238:60610 WARNING: Failed running command (–auth-user-pass-verify): external program exited with error status: 1
    May 9 14:14:58 openvpn 70379 192.168.5.238:60610 peer info: IV_PROTO=2
    May 9 14:14:58 openvpn 70379 192.168.5.238:60610 peer info: IV_TCPNL=1
    May 9 14:14:58 openvpn 70379 192.168.5.238:60610 peer info: IV_NCP=2
    May 9 14:14:58 openvpn 70379 192.168.5.238:60610 peer info: IV_PLAT=ios
    May 9 14:14:58 openvpn 70379 192.168.5.238:60610 peer info: IV_VER=3.2
    May 9 14:14:58 openvpn 70379 192.168.5.238:60610 peer info: IV_GUI_VER=net.openvpn.connect.ios_1.2.9-0

    system log

    May 9 14:16:01 kernel pid 80989 (php-fpm), uid 0: exited on signal 11 (core dumped)
    May 9 14:15:41 kernel Failed to fully fault in a core file segment at VA 0x8009c5000 with size 0x2000 to be written at offset 0x109000 for process php-fpm
    May 9 14:15:41 kernel vm_fault: pager read error, pid 80989 (php-fpm)
    May 9 14:15:41 kernel vnode_pager_generic_getpages_done: I/O read error 5
    May 9 14:15:41 kernel g_vfs_done():ufsid/58401b3de03cb078[READ(offset=12529262592, length=4096)]error = 5
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): Error 5, Retries exhausted
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): RES: 41 40 00 67 75 00 01 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 ff 66 75 40 01 00 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): Retrying command
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): RES: 41 40 00 67 75 00 01 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 ff 66 75 40 01 00 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): Retrying command
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): RES: 41 40 00 67 75 00 01 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 ff 66 75 40 01 00 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): Retrying command
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): RES: 41 40 00 67 75 00 01 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 ff 66 75 40 01 00 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): Retrying command
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): RES: 41 40 00 67 75 00 01 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 ff 66 75 40 01 00 00 00 00 00
    May 9 14:15:41 kernel vm_fault: pager read error, pid 80989 (php-fpm)
    May 9 14:15:41 kernel vnode_pager_generic_getpages_done: I/O read error 5
    May 9 14:15:41 kernel g_vfs_done():ufsid/58401b3de03cb078[READ(offset=12529262592, length=4096)]error = 5
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): Error 5, Retries exhausted
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): RES: 41 40 00 67 75 00 01 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 ff 66 75 40 01 00 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): Retrying command
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): RES: 41 40 00 67 75 00 01 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 ff 66 75 40 01 00 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): Retrying command
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): RES: 41 40 00 67 75 00 01 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 ff 66 75 40 01 00 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): Retrying command
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): RES: 41 40 00 67 75 00 01 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 ff 66 75 40 01 00 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): Retrying command
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): RES: 41 40 00 67 75 00 01 00 00 00 00
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
    May 9 14:15:41 kernel (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 08 ff 66 75 40 01 00 00 00 00 00

    is it the SSD failed?

    DiagnosticsS.M.A.R.T. StatusInformation & Tests

    smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.1-RELEASE-p7 amd64] (local build)
    Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

    === START OF INFORMATION SECTION ===
    Model Family:    SanDisk based SSDs
    Device Model:    SanDisk SSD i100 24GB
    Serial Number:    132300104320
    LU WWN Device Id: 5 001b44 9d9485e80
    Firmware Version: 11.56.00
    User Capacity:    24,015,495,168 bytes [24.0 GB]
    Sector Size:      512 bytes logical/physical
    Rotation Rate:    Solid State Device
    Form Factor:      1.8 inches
    Device is:        In smartctl database [for details use: -P show]
    ATA Version is:  ACS-2 T13/2015-D revision 3
    SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
    Local Time is:    Wed May  9 14:34:56 2018 HKT
    SMART support is: Available - device has SMART capability.
    SMART support is: Enabled

    === START OF READ SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED

    General SMART Values:
    Offline data collection status:  (0x00) Offline data collection activity
    was never started.
    Auto Offline Data Collection: Disabled.
    Self-test execution status:      (  25) The self-test routine was aborted by
    the host.
    Total time to complete Offline
    data collection: (  120) seconds.
    Offline data collection
    capabilities: (0x51) SMART execute Offline immediate.
    No Auto Offline data collection support.
    Suspend Offline collection upon new
    command.
    No Offline surface scan supported.
    Self-test supported.
    No Conveyance Self-test supported.
    Selective Self-test supported.
    SMART capabilities:            (0x0003) Saves SMART data before entering
    power-saving mode.
    Supports SMART auto save timer.
    Error logging capability:        (0x01) Error logging supported.
    General Purpose Logging supported.
    Short self-test routine
    recommended polling time: (  2) minutes.
    Extended self-test routine
    recommended polling time: (  7) minutes.

    SMART Attributes Data Structure revision number: 1
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      5 Reallocated_Sector_Ct  0x0002  100  100  000    Old_age  Always      -      0
      9 Power_On_Hours          0x0002  100  100  000    Old_age  Always      -      12565
    12 Power_Cycle_Count      0x0002  100  100  000    Old_age  Always      -      31
    171 Program_Fail_Count      0x0002  100  100  000    Old_age  Always      -      0
    172 Erase_Fail_Count        0x0002  100  100  000    Old_age  Always      -      0
    173 Avg_Write/Erase_Count  0x0002  100  100  000    Old_age  Always      -      166
    174 Unexpect_Power_Loss_Ct  0x0002  100  100  000    Old_age  Always      -      26
    187 Reported_Uncorrect      0x0002  100  100  000    Old_age  Always      -      508
    230 Perc_Write/Erase_Count  0x0002  100  100  000    Old_age  Always      -      553
    232 Perc_Avail_Resrvd_Space 0x0003  100  100  005    Pre-fail  Always      -      0
    234 Perc_Write/Erase_Ct_BC  0x0002  100  100  000    Old_age  Always      -      10000
    241 Total_LBAs_Written      0x0002  100  100  000    Old_age  Always      -      72536025643
    242 Total_LBAs_Read        0x0002  100  100  000    Old_age  Always      -      4143166

    SMART Error Log Version: 1
    No Errors Logged

    Warning! SMART Self-Test Log Structure error: invalid SMART checksum.
    SMART Self-test log structure revision number 1
    Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error

    1  Extended offline    Aborted by host              90%    12565        -

    2  Short offline      Aborted by host              80%    12565        -

    SMART Selective self-test log data structure revision number 1
    SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
        1        0        0  Not_testing
        2        0        0  Not_testing
        3        0        0  Not_testing
        4        0        0  Not_testing
        5        0        0  Not_testing
    Selective self-test flags (0x0):
      After scanning selected spans, do NOT read-scan remainder of disk.
    If Selective self-test is pending on power-up, resume after 0 minute delay.

    0
  • Rebel Alliance Developer Netgate

    Those errors are consistent with a failed disk.

    0
  • Y

    @jimp:

    Those errors are consistent with a failed disk.

    never replace OS disk

    1. backup the config file
    2. replace SSD
    3. install pfsense by USB drive
    4. restore by the config file

    is it that simple?

    0

  • It's that simple.

    0
 

Products

Applications

Support

Services

News

Resources

Company

Our Mission

As host of the pfSense open source firewall project, Netgate believes in enhancing network connectivity that maintains both security and privacy. We also believe everyone should be able to afford it.

Subscribe to our Newsletter

Product information, software announcements, and special offers. See our newsletter archive for past announcements.

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy