Lost LAN connection



  • Hi everyone,
    I have a pfsense 2.4.4-RELEASE-p3 (amd64) installation on a proxmox v5.3-6.
    configuration is the following:
    1 network device e1000 for WAN link
    1 network device e1000 for LAN link

    Pfsense is running as:
    Firewall
    Suricata
    OpenVPN server

    32Go of storage, 2GB of RAM

    The WAN part is directly connected to Internet with a fixed adress IP
    The LAN part is composed by ~16 servers

    Since 1 month I encountered a lost LAN link issue:
    Impossible to access to web ressources from Internet to LAN
    Impossible to ping a local server from the pfesense
    Impossible to access servers through the VPN
    Possible to access the Local inteface of the pfsens and access the web gui
    The pfsense are sent to a remote syslog located on the LAN
    In /var/log/system.log not much more information, only "syslogd: sendto: Host is down"

    here the line juste befone the connection lost:
    May 28 06:40:07 fw php-cgi: suricata_check_cron_misc.inc: [Suricata] Automatic clean-up of Suricata logs completed.
    May 29 00:16:28 fw syslogd: sendto: Host is down

    It happens randomly :/
    the only solution I found is to reboot the system.
    I did some research and found the pfsense support page talking about kern.ipc.nmbclusters setting but it is already in /boot/loader.conf

    kern.cam.boot_delay=10000
    kern.ipc.nmbclusters="1000000"
    kern.ipc.nmbjumbop="524288"
    kern.ipc.nmbjumbo9="524288"
    autoboot_delay="3"
    hw.usb.no_pf="1"
    

    I ran this command line also to check the queue lenght

    sysctl net.inet.ip.intr_queue_maxlen
    net.inet.ip.intr_queue_maxlen: 1000
    

    And this to check the queue status:

    sysctl net.inet.ip.intr_queue_drops
    net.inet.ip.intr_queue_drops: 0
    

    value seems to be correct according to [https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html#](link url)

    i did some research on this forum but did not find an answer (maybe I did not well the search ?)

    When the connection is lost, there is no message in the proxmox host server (/var/log/messages) and there is no backup job running.

    Do you have an idea? I do not want to set a cron task to reboot every night the fw, I want to find why there is this issue :)
    Thank you
    Karadoc



  • @karadoc said in Lost LAN connection:

    already in /boot/loader.conf

    You are aware of the fact that that file gets over written ?
    Read the page https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html?highlight=loader local conf# again.
    The solution is mentioned : use loader.local.conf for your local settings.

    Btw : you do not have hardware issues : most of not all hardware is virtual, your using a VM.
    It would be (probably) a hardware issue if you were running pfSense outside a VM.
    But also : if it is a hardware issue, the VM wont make thing any easier to find.
    I advise you to run pfSense outside a VM, this to exclude all proxmox issues.

    edit : => loader.local.conf !



  • @Gertjan said in Lost LAN connection:

    The solution is mentioned : use loader.local.conf for your local settings.

    ....crap... I missed the .local.conf ^^
    I did the file update and I will wait if it happens again
    thank you !


  • Netgate Administrator

    That's what gets written into /boot/loader.conf autimatically. If you wanted to change those values or add others you should use /boot/loader.conf.local but there is no need to unless you do.

    Possible to access the Local inteface of the pfSense and access the web gui.

    From where? On which interface?

    Steve



  • hi Stephenw10

    I can acces the webgui from the VPN connection (from WAN).

    despite the modification done, the firewall had the same behavior I had again to reboot the firewall
    One thing I noticed, the ARP table empties when this happens...
    The limit of kern.ipc.nmbclusters was not reach when the connection on the LAN was lost.


  • Netgate Administrator

    Are those NICs configured differently in Proxmox? I assume the LAN is just internet a v-switch? Is the WAN NIC passed through to a real NIC?

    Steve



  • On the proxmox
    there is a physical NIC configure on the proxmox with public IP
    There is a virtual NIC on the proxmox with IP on this subnet 10.x.x.x/30 named WAN set as bridge
    on the proxmox an iptables config in order to forward all traffic on the NIC WAN of the pfsense
    On the pfsense, there is a NIC with internal IP on this subnet 10.x.x.x/30named WAN
    On the pfsensen there is a NIC with internal IP on this subnet 192.168.5.0/24.



  • I have just lost again LAN connection.
    I can ping 8.8.8.8 or an Internet FQDN, but I cannot ping a local server on the 192.168.5.X subnet.
    I just try to disable and re enable the LAN NIC with

    ifconfig
    

    command and everything is back to normal ...


  • Netgate Administrator

    OK, so are those two NICs configured any differently in Proxmox?

    Do they appear any differently in pfSense?

    What is if down what does ifconfig show or the LAN interface? What does proxmox show for the state of the interface?

    There must be something different between the two NICs.

    Steve



  • @stephenw10 said in Lost LAN connection:

    OK, so are those two NICs configured any differently in Proxmox?

    no they have the same type of config, Intel E1000

    @stephenw10 said in Lost LAN connection:

    Do they appear any differently in pfSense?

    no, they are seen as 1000baseT <full-duplex> NIC both

    @stephenw10 said in Lost LAN connection:

    What is if down what does ifconfig show or the LAN interface?

    hmmm good point I did not check it when I lost the connection I will try next. but pfsense seems to see the LAN NIC enable (as I can connect to the web GUI using the local IP through VPN)
    Hereafter the ifconfig output for WAN and LAN config

    em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    	options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
    	ether b2:ab:9e:15:0c:f7
    	hwaddr b2:ab:9e:15:0c:f7
    	inet6 fe80::b0ab:9eff:fe15:cf7%em0 prefixlen 64 scopeid 0x1 
    	inet 10.0.0.2 netmask 0xfffffffc broadcast 10.0.0.3 
    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    	media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
    	ether a2:b4:3c:2c:e7:d4
    	hwaddr a2:b4:3c:2c:e7:d4
    	inet6 fe80::a0b4:3cff:fe2c:e7d4%em1 prefixlen 64 scopeid 0x2 
    	inet 192.168.5.254 netmask 0xffffff00 broadcast 192.168.5.255 
    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    	media: Ethernet autoselect (1000baseT <full-duplex>)
    	status: active
    

    @stephenw10 said in Lost LAN connection:

    What does proxmox show for the state of the interface?

    Proxmox show the NIC as enabled but no traffic on the LAN


  • Netgate Administrator

    I assume em1 is LAN?

    You might try enabling promiscuous mode on em1 as a test. It shouldn't be needed but it is a difference.

    You can also try using a VirtIO NIC for LAN.

    Steve



  • Hi all

    As I lost again twice the connection on the LAN interface I tried this :

    https://docs.netgate.com/pfsense/en/latest/hardware/troubleshooting-lost-traffic-or-disappearing-packets.html

    I will see if there is any improvement


  • Netgate Administrator

    Did you try using VirtIO NICs instead?


Log in to reply