Помогите решить проблему пропадания сети в определенном vlan и отвал вебинтерфейса



  • pfsense
    2.4.3
    В сети 3 vlan основных, но на одном из них периодически выборочно отваливаются от сети несколько адресов (хаотично), помогает ребут pfsensa.

    Так же отваливается веб интерфейс.

    Логи выкладываю ниже при последнем пропадании вебинтерфеса:

    Feb 11 16:03:18  php-fpm  87353  /index.php: Successful login for user 'admin' from: 192.168.0.40  
    Feb 11 16:03:11  tv-router.stv   nginx: 2019/02/11 16:03:11 [error] 36354#100130: *18054 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.0.93, server: , request: "POST /widgets/widgets/interfaces.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.0.254", referrer: "http://192.168.0.254/"  
    Feb 11 16:03:00  check_reload_status   check_reload_status is starting.  
    Feb 11 16:03:00  rc.php-fpm_restart  86607  >>> Restarting php-fpm  
    Feb 11 16:01:46  sshd  30800  Accepted keyboard-interactive/pam for admin from 192.168.0.40 port 57422 ssh2  
    Feb 11 16:00:08  tv-router.stv   nginx: 2019/02/11 16:00:08 [error] 36354#100130: *18054 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.0.93, server: , request: "POST /widgets/widgets/interfaces.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.0.254", referrer: "http://192.168.0.254/"  
    Feb 11 15:58:05  check_reload_status   Linkup starting em0.30  
    Feb 11 15:58:05  check_reload_status   Linkup starting em0.45  
    Feb 11 15:58:05  check_reload_status   Linkup starting em0.10  
    Feb 11 15:58:05  check_reload_status   Linkup starting em0.24  
    Feb 11 15:58:05  check_reload_status   Linkup starting em0.20  
    Feb 11 15:58:05  check_reload_status   Linkup starting em0.1  
    Feb 11 15:58:05  check_reload_status   Linkup starting em0.50  
    Feb 11 15:58:05  kernel   em0.30: link state changed to UP  
    Feb 11 15:58:05  kernel   em0.45: link state changed to UP  
    Feb 11 15:58:05  kernel   em0.10: link state changed to UP  
    Feb 11 15:58:05  kernel   em0.24: link state changed to UP  
    Feb 11 15:58:05  kernel   em0.20: link state changed to UP  
    Feb 11 15:58:05  kernel   em0.1: link state changed to UP  
    Feb 11 15:58:05  kernel   em0.50: link state changed to UP  
    Feb 11 15:58:05  kernel   em0: link state changed to UP  
    Feb 11 15:58:05  check_reload_status   Linkup starting em0  
    Feb 11 15:58:02  check_reload_status   Linkup starting em0.30  
    Feb 11 15:58:02  check_reload_status   Linkup starting em0.45  
    Feb 11 15:58:02  check_reload_status   Linkup starting em0.10  
    Feb 11 15:58:02  check_reload_status   Linkup starting em0.24  
    Feb 11 15:58:02  check_reload_status   Linkup starting em0.20  
    Feb 11 15:58:02  check_reload_status   Linkup starting em0.1  
    Feb 11 15:58:02  check_reload_status   Linkup starting em0.50  
    Feb 11 15:58:02  kernel   em0.30: link state changed to DOWN  
    Feb 11 15:58:02  kernel   em0.45: link state changed to DOWN  
    Feb 11 15:58:02  kernel   em0.10: link state changed to DOWN  
    Feb 11 15:58:02  kernel   em0.24: link state changed to DOWN  
    Feb 11 15:58:02  kernel   em0.20: link state changed to DOWN  
    Feb 11 15:58:02  kernel   em0.1: link state changed to DOWN  
    Feb 11 15:58:02  kernel   em0.50: link state changed to DOWN  
    Feb 11 15:58:02  kernel   em0: link state changed to DOWN  
    Feb 11 15:58:02  check_reload_status   Linkup starting em0  
    Feb 11 15:57:06  tv-router.stv   nginx: 2019/02/11 15:57:06 [error] 36354#100130: *18054 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.0.93, server: , request: "POST /widgets/widgets/thermal_sensors.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.0.254", referrer: "http://192.168.0.254/"  
    Feb 11 15:54:16  check_reload_status   Linkup starting em0.30  
    Feb 11 15:54:16  check_reload_status   Linkup starting em0.45  
    Feb 11 15:54:16  check_reload_status   Linkup starting em0.10  
    Feb 11 15:54:16  check_reload_status   Linkup starting em0.24  
    Feb 11 15:54:16  check_reload_status   Linkup starting em0.20  
    Feb 11 15:54:16  check_reload_status   Linkup starting em0.1  
    Feb 11 15:54:16  check_reload_status   Linkup starting em0.50  
    Feb 11 15:54:16  kernel   em0.30: link state changed to UP  
    Feb 11 15:54:16  kernel   em0.45: link state changed to UP  
    Feb 11 15:54:16  kernel   em0.10: link state changed to UP  
    Feb 11 15:54:16  kernel   em0.24: link state changed to UP  
    Feb 11 15:54:16  kernel   em0.20: link state changed to UP  
    Feb 11 15:54:16  kernel   em0.1: link state changed to UP  
    Feb 11 15:54:16  kernel   em0.50: link state changed to UP  
    Feb 11 15:54:16  kernel   em0: link state changed to UP  
    Feb 11 15:54:16  check_reload_status   Linkup starting em0  
    Feb 11 15:54:13  check_reload_status   Linkup starting em0.30  
    Feb 11 15:54:13  check_reload_status   Linkup starting em0.45  
    Feb 11 15:54:13  check_reload_status   Linkup starting em0.10  
    Feb 11 15:54:13  check_reload_status   Linkup starting em0.24  
    Feb 11 15:54:13  check_reload_status   Linkup starting em0.20  
    Feb 11 15:54:13  check_reload_status   Linkup starting em0.1  
    Feb 11 15:54:13  check_reload_status   Linkup starting em0.50  
    Feb 11 15:54:13  kernel   em0.30: link state changed to DOWN  
    Feb 11 15:54:13  kernel   em0.45: link state changed to DOWN  
    Feb 11 15:54:13  kernel   em0.10: link state changed to DOWN  
    Feb 11 15:54:13  kernel   em0.24: link state changed to DOWN  
    Feb 11 15:54:13  kernel   em0.20: link state changed to DOWN  
    Feb 11 15:54:13  kernel   em0.1: link state changed to DOWN  
    Feb 11 15:54:13  kernel   em0.50: link state changed to DOWN  
    Feb 11 15:54:13  kernel   em0: link state changed to DOWN  
    Feb 11 15:54:13  check_reload_status   Linkup starting em0  
    Feb 11 15:54:05  tv-router.stv   nginx: 2019/02/11 15:54:05 [error] 36354#100130: *18054 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.0.93, server: , request: "POST /widgets/widgets/interfaces.widget.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.0.254", referrer: "http://192.168.0.254/"  
    Feb 11 15:54:05  tv-router.stv   nginx: 2019/02/11 15:54:05 [error] 36697#100101: *17998 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 192.168.0.93, server: , request: "POST /ifstats.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "192.168.0.254", referrer: "http://192.168.0.254/"  
    Feb 11 15:51:58  check_reload_status   Linkup starting em0.30  
    Feb 11 15:51:58  check_reload_status   Linkup starting em0.45  
    Feb 11 15:51:58  check_reload_status   Linkup starting em0.10  
    Feb 11 15:51:58  check_reload_status   Linkup starting em0.24  
    Feb 11 15:51:58  check_reload_status   Linkup starting em0.20  
    Feb 11 15:51:58  check_reload_status   Linkup starting em0.1  
    Feb 11 15:51:58  check_reload_status   Linkup starting em0.50  
    Feb 11 15:51:58  kernel   em0.30: link state changed to UP  
    Feb 11 15:51:58  kernel   em0.45: link state changed to UP  
    Feb 11 15:51:58  kernel   em0.10: link state changed to UP  
    Feb 11 15:51:58  kernel   em0.24: link state changed to UP  
    Feb 11 15:51:58  kernel   em0.20: link state changed to UP  
    Feb 11 15:51:58  kernel   em0.1: link state changed to UP  
    Feb 11 15:51:58  kernel   em0.50: link state changed to UP  
    Feb 11 15:51:58  kernel   em0: link state changed to UP  
    Feb 11 15:51:58  check_reload_status   Linkup starting em0  
    Feb 11 15:51:55  check_reload_status   Linkup starting em0.30  
    Feb 11 15:51:55  check_reload_status   Linkup starting em0.45  
    Feb 11 15:51:55  check_reload_status   Linkup starting em0.10  
    Feb 11 15:51:55  check_reload_status   Linkup starting em0.24  
    Feb 11 15:51:55  check_reload_status   Linkup starting em0.20  
    Feb 11 15:51:55  check_reload_status   Linkup starting em0.1  
    Feb 11 15:51:55  check_reload_status   Linkup starting em0.50  
    Feb 11 15:51:55  kernel   em0.30: link state changed to DOWN  
    Feb 11 15:51:55  kernel   em0.45: link state changed to DOWN  
    Feb 11 15:51:55  kernel   em0.10: link state changed to DOWN  
    Feb 11 15:51:55  kernel   em0.24: link state changed to DOWN  
    Feb 11 15:51:55  kernel   em0.20: link state changed to DOWN  
    Feb 11 15:51:55  kernel   em0.1: link state changed to DOWN  
    Feb 11 15:51:55  kernel   em0.50: link state changed to DOWN  
    Feb 11 15:51:55  kernel   em0: link state changed to DOWN  
    Feb 11 15:51:55  check_reload_status   Linkup starting em0  
    Feb 11 15:51:04  check_reload_status   rc.newwanip starting em0.20  
    Feb 11 15:51:04  php-fpm  62046  /rc.linkup: Hotplug event detected for OPT2(opt2) static IP (192.168.20.254 )  
    Feb 11 15:51:04  check_reload_status   rc.newwanip starting em0.1  
    Feb 11 15:51:04  php-fpm  62046  /rc.linkup: Hotplug event detected for OPT4(opt4) static IP (192.168.0.254 )  
    Feb 11 15:51:04  check_reload_status   Reloading filter  
    Feb 11 15:51:04  check_reload_status   rc.newwanip starting em0.50  
    Feb 11 15:51:04  php-fpm  62046  /rc.linkup: Hotplug event detected for OPT5(opt5) static IP (192.168.50.254 )  
    Feb 11 15:51:04  check_reload_status   Reloading filter  
    Feb 11 15:51:04  php-fpm  62046  /rc.linkup: Linkup detected on disabled interface...Ignoring  
    Feb 11 15:51:03  check_reload_status   Linkup starting em0.30  
    Feb 11 15:51:03  check_reload_status   Linkup starting em0.45  
    Feb 11 15:51:03  check_reload_status   Linkup starting em0.10  
    Feb 11 15:51:03  check_reload_status   Linkup starting em0.24  
    Feb 11 15:51:03  check_reload_status   Linkup starting em0.20  
    Feb 11 15:51:03  check_reload_status   Linkup starting em0.1  
    Feb 11 15:51:03  check_reload_status   Linkup starting em0.50  
    Feb 11 15:51:03  kernel   em0.30: link state changed to UP  
    Feb 11 15:51:03  kernel   em0.45: link state changed to UP  
    Feb 11 15:51:03  kernel   em0.10: link state changed to UP  
    Feb 11 15:51:03  kernel   em0.24: link state changed to UP  
    Feb 11 15:51:03  kernel   em0.20: link state changed to UP  
    Feb 11 15:51:03  kernel   em0.1: link state changed to UP  
    Feb 11 15:51:03  kernel   em0.50: link state changed to UP  
    Feb 11 15:51:03  kernel   em0: link state changed to UP  
    Feb 11 15:51:03  check_reload_status   Linkup starting em0  
    Feb 11 15:51:02  xinetd  29845  Reconfigured: new=0 old=1 dropped=0 (services)  
    Feb 11 15:51:02  xinetd  29845  readjusting service 6969-udp  
    Feb 11 15:51:02  xinetd  29845  Swapping defaults  
    Feb 11 15:51:02  xinetd  29845  Starting reconfiguration  
    Feb 11 15:51:02  xinetd  29845  Reconfigured: new=0 old=1 dropped=0 (services)  
    Feb 11 15:51:02  xinetd  29845  readjusting service 6969-udp  
    Feb 11 15:51:02  xinetd  29845  Swapping defaults  
    Feb 11 15:51:02  xinetd  29845  Starting reconfiguration  
    Feb 11 15:51:01  php-fpm  62046  /rc.linkup: Linkup detected on disabled interface...Ignoring  
    Feb 11 15:51:01  php-fpm  62046  /rc.linkup: Hotplug event detected for OPT6(opt6) static IP (192.168.45.254 )  
    Feb 11 15:51:01  php-fpm  62046  /rc.linkup: Hotplug event detected for OPT1(opt1) static IP (192.168.10.254 )  
    Feb 11 15:51:01  php-fpm  54139  /rc.linkup: Hotplug event detected for OPT2(opt2) static IP (192.168.20.254 )  
    Feb 11 15:51:01  check_reload_status   Reloading filter  
    Feb 11 15:51:01  php-fpm  62046  /rc.linkup: Hotplug event detected for OPT4(opt4) static IP (192.168.0.254 )  
    Feb 11 15:51:01  php-fpm  54139  /rc.linkup: Hotplug event detected for OPT5(opt5) static IP (192.168.50.254 )  
    Feb 11 15:51:01  check_reload_status   Reloading filter  
    Feb 11 15:51:01  php-fpm  62046  /rc.linkup: Linkup detected on disabled interface...Ignoring  
    Feb 11 15:51:00  check_reload_status   Linkup starting em0.30  
    Feb 11 15:51:00  check_reload_status   Linkup starting em0.45  
    Feb 11 15:51:00  check_reload_status   Linkup starting em0.10  
    Feb 11 15:51:00  check_reload_status   Linkup starting em0.24  
    Feb 11 15:51:00  check_reload_status   Linkup starting em0.20  
    Feb 11 15:51:00  check_reload_status   Linkup starting em0.1  
    Feb 11 15:51:00  check_reload_status   Linkup starting em0.50  
    Feb 11 15:51:00  check_reload_status   Linkup starting em0  
    Feb 11 15:51:00  kernel   em0.30: link state changed to DOWN  
    Feb 11 15:51:00  kernel   em0.45: link state changed to DOWN  
    Feb 11 15:51:00  kernel   em0.10: link state changed to DOWN  
    Feb 11 15:51:00  kernel   em0.24: link state changed to DOWN  
    Feb 11 15:51:00  kernel   em0.20: link state changed to DOWN  
    Feb 11 15:51:00  kernel   em0.1: link state changed to DOWN  
    Feb 11 15:51:00  kernel   em0.50: link state changed to DOWN  
    Feb 11 15:51:00  kernel   em0: link state changed to DOWN  
    Feb 11 15:50:05  php-fpm  54139  /rc.newwanip: rc.newwanip: on (IP address: 192.168.0.254) (interface: OPT4[opt4]) (real interface: em0.1).  
    Feb 11 15:50:05  php-fpm  54139  /rc.newwanip: rc.newwanip: Info: starting on em0.1.  
    Feb 11 15:50:05  php-fpm  54139  /rc.newwanip: rc.newwanip: on (IP address: 192.168.45.254) (interface: OPT6[opt6]) (real interface: em0.45).  
    Feb 11 15:50:05  php-fpm  54139  /rc.newwanip: rc.newwanip: Info: starting on em0.45.  
    Feb 11 15:50:05  xinetd  29845  Reconfigured: new=0 old=1 dropped=0 (services)  
    Feb 11 15:50:05  xinetd  29845  readjusting service 6969-udp  
    Feb 11 15:50:05  xinetd  29845  Swapping defaults  
    Feb 11 15:50:05  xinetd  29845  Starting reconfiguration  
    Feb 11 15:50:05  php-fpm  54139  /rc.newwanip: rc.newwanip: on (IP address: 192.168.10.254) (interface: OPT1[opt1]) (real interface: em0.10).  
    Feb 11 15:50:05  php-fpm  54139  /rc.newwanip: rc.newwanip: Info: starting on em0.10.  
    Feb 11 15:50:05  php-fpm  54139  /rc.newwanip: rc.newwanip: on (IP address: 192.168.50.254) (interface: OPT5[opt5]) (real interface: em0.50).  
    Feb 11 15:50:05  php-fpm  54139  /rc.newwanip: rc.newwanip: Info: starting on em0.50.  
    Feb 11 15:50:05  check_reload_status   Reloading filter  
    Feb 11 15:50:05  php-fpm  62046  /rc.newwanip: rc.newwanip: on (IP address: 192.168.20.254) (interface: OPT2[opt2]) (real interface: em0.20).  
    Feb 11 15:50:05  php-fpm  62046  /rc.newwanip: rc.newwanip: Info: starting on em0.20.  
    Feb 11 15:50:05  xinetd  29845  Reconfigured: new=0 old=1 dropped=0 (services)  
    Feb 11 15:50:05  xinetd  29845  readjusting service 6969-udp  
    Feb 11 15:50:05  xinetd  29845  Swapping defaults  
    Feb 11 15:50:05  xinetd  29845  Starting reconfiguration  
    Feb 11 15:50:04  check_reload_status   rc.newwanip starting em0.1  
    Feb 11 15:50:04  php-fpm  62046  /rc.linkup: Linkup detected on disabled interface...Ignoring  
    Feb 11 15:50:04  check_reload_status   rc.newwanip starting em0.45  
    Feb 11 15:50:04  php-fpm  62046  /rc.linkup: Hotplug event detected for OPT6(opt6) static IP (192.168.45.254 )  
    Feb 11 15:50:04  check_reload_status   rc.newwanip starting em0.10  
    Feb 11 15:50:04  php-fpm  62046  /rc.linkup: Hotplug event detected for OPT1(opt1) static IP (192.168.10.254 )  
    Feb 11 15:50:04  check_reload_status   rc.newwanip starting em0.50  
    Feb 11 15:50:04  check_reload_status   Reloading filter  
    Feb 11 15:50:04  check_reload_status   rc.newwanip starting em0.20  
    Feb 11 15:50:04  php-fpm  87597  /rc.linkup: Hotplug event detected for OPT2(opt2) static IP (192.168.20.254 )  
    Feb 11 15:50:04  check_reload_status   Reloading filter  
    Feb 11 15:50:04  php-fpm  87597  /rc.linkup: Linkup detected on disabled interface...Ignoring  
    Feb 11 15:50:04  php-fpm  62046  /rc.linkup: Hotplug event detected for OPT5(opt5) static IP (192.168.50.254 )  
    Feb 11 15:50:04  php-fpm  54139  /rc.linkup: Hotplug event detected for OPT4(opt4) static IP (192.168.0.254 )  
    Feb 11 15:50:03  check_reload_status   Linkup starting em0.30  
    Feb 11 15:50:03  check_reload_status   Linkup starting em0.45  
    Feb 11 15:50:03  check_reload_status   Linkup starting em0.10  
    Feb 11 15:50:03  check_reload_status   Linkup starting em0.24  
    Feb 11 15:50:03  check_reload_status   Linkup starting em0.20  
    Feb 11 15:50:03  check_reload_status   Linkup starting em0.1  
    Feb 11 15:50:03  check_reload_status   Linkup starting em0.50  
    Feb 11 15:50:03  kernel   em0.30: link state changed to UP  
    
    

    Или подскажите в каком направлении смотреть?



  • @jktu79

    Привет
    Нашел тут одну статью англоязычную , тоже у человека отваливалась карта с vlan-ми , потом поднималась ( тоже PF установлен был )
    Правда , карта у него была Realtek ( у Вас Intel)

    Ему помогло /system/advanced/networking/disable hardware checksum offloading
    и перезагрузка

    p.s. многие рекомендуют и LRO отключить и TSO тоже

    некоторые грешат на MSIX ( из доков на pf)
    это как их отключить
    0_1549917600720_b7bd8d09-4dde-48c8-b74a-a9532154d32f-image.png

    плюс к этому можно в этом же файле установить вот такую переменную
    hw.em.msix = 0

    hw.em.msix
    Enable or disable MSI-X style interrupts. Default 1 (enabled).

    https://www.freebsd.org/cgi/man.cgi?em(4)



  • @jktu79
    Еще хотел задать уточняющий вопрос - проверьте , с чего начинается падение em0 по логам ? Нет ли там таких строчек em0: watchdog timeout -- resetting ? Вы выложили лог , в котором не видно начала проблемы . Если такая строчка есть , также кабель рекомендуют проверить