Lost LAN connection
-
Hi everyone,
I have a pfsense 2.4.4-RELEASE-p3 (amd64) installation on a proxmox v5.3-6.
configuration is the following:
1 network device e1000 for WAN link
1 network device e1000 for LAN linkPfsense is running as:
Firewall
Suricata
OpenVPN server32Go of storage, 2GB of RAM
The WAN part is directly connected to Internet with a fixed adress IP
The LAN part is composed by ~16 serversSince 1 month I encountered a lost LAN link issue:
Impossible to access to web ressources from Internet to LAN
Impossible to ping a local server from the pfesense
Impossible to access servers through the VPN
Possible to access the Local inteface of the pfsens and access the web gui
The pfsense are sent to a remote syslog located on the LAN
In /var/log/system.log not much more information, only "syslogd: sendto: Host is down"here the line juste befone the connection lost:
May 28 06:40:07 fw php-cgi: suricata_check_cron_misc.inc: [Suricata] Automatic clean-up of Suricata logs completed.
May 29 00:16:28 fw syslogd: sendto: Host is downIt happens randomly :/
the only solution I found is to reboot the system.
I did some research and found the pfsense support page talking about kern.ipc.nmbclusters setting but it is already in /boot/loader.confkern.cam.boot_delay=10000 kern.ipc.nmbclusters="1000000" kern.ipc.nmbjumbop="524288" kern.ipc.nmbjumbo9="524288" autoboot_delay="3" hw.usb.no_pf="1"
I ran this command line also to check the queue lenght
sysctl net.inet.ip.intr_queue_maxlen net.inet.ip.intr_queue_maxlen: 1000
And this to check the queue status:
sysctl net.inet.ip.intr_queue_drops net.inet.ip.intr_queue_drops: 0
value seems to be correct according to [https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html#](link url)
i did some research on this forum but did not find an answer (maybe I did not well the search ?)
When the connection is lost, there is no message in the proxmox host server (/var/log/messages) and there is no backup job running.
Do you have an idea? I do not want to set a cron task to reboot every night the fw, I want to find why there is this issue :)
Thank you
Karadoc -
@karadoc said in Lost LAN connection:
already in /boot/loader.conf
You are aware of the fact that that file gets over written ?
Read the page https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html?highlight=loader%20local%20conf# again.
The solution is mentioned : useloader.local.conffor your local settings.Btw : you do not have hardware issues : most of not all hardware is virtual, your using a VM.
It would be (probably) a hardware issue if you were running pfSense outside a VM.
But also : if it is a hardware issue, the VM wont make thing any easier to find.
I advise you to run pfSense outside a VM, this to exclude all proxmox issues.edit : => loader.local.conf !
-
@Gertjan said in Lost LAN connection:
The solution is mentioned : use loader.local.conf for your local settings.
....crap... I missed the .local.conf ^^
I did the file update and I will wait if it happens again
thank you ! -
That's what gets written into /boot/loader.conf autimatically. If you wanted to change those values or add others you should use /boot/loader.conf.local but there is no need to unless you do.
Possible to access the Local inteface of the pfSense and access the web gui.
From where? On which interface?
Steve
-
hi Stephenw10
I can acces the webgui from the VPN connection (from WAN).
despite the modification done, the firewall had the same behavior I had again to reboot the firewall
One thing I noticed, the ARP table empties when this happens...
The limit of kern.ipc.nmbclusters was not reach when the connection on the LAN was lost. -
Are those NICs configured differently in Proxmox? I assume the LAN is just internet a v-switch? Is the WAN NIC passed through to a real NIC?
Steve
-
On the proxmox
there is a physical NIC configure on the proxmox with public IP
There is a virtual NIC on the proxmox with IP on this subnet 10.x.x.x/30 named WAN set as bridge
on the proxmox an iptables config in order to forward all traffic on the NIC WAN of the pfsense
On the pfsense, there is a NIC with internal IP on this subnet 10.x.x.x/30named WAN
On the pfsensen there is a NIC with internal IP on this subnet 192.168.5.0/24. -
I have just lost again LAN connection.
I can ping 8.8.8.8 or an Internet FQDN, but I cannot ping a local server on the 192.168.5.X subnet.
I just try to disable and re enable the LAN NIC withifconfig
command and everything is back to normal ...
-
OK, so are those two NICs configured any differently in Proxmox?
Do they appear any differently in pfSense?
What is if down what does ifconfig show or the LAN interface? What does proxmox show for the state of the interface?
There must be something different between the two NICs.
Steve
-
@stephenw10 said in Lost LAN connection:
OK, so are those two NICs configured any differently in Proxmox?
no they have the same type of config, Intel E1000
@stephenw10 said in Lost LAN connection:
Do they appear any differently in pfSense?
no, they are seen as 1000baseT <full-duplex> NIC both
@stephenw10 said in Lost LAN connection:
What is if down what does ifconfig show or the LAN interface?
hmmm good point I did not check it when I lost the connection I will try next. but pfsense seems to see the LAN NIC enable (as I can connect to the web GUI using the local IP through VPN)
Hereafter the ifconfig output for WAN and LAN configem0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=2098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether b2:ab:9e:15:0c:f7 hwaddr b2:ab:9e:15:0c:f7 inet6 fe80::b0ab:9eff:fe15:cf7%em0 prefixlen 64 scopeid 0x1 inet 10.0.0.2 netmask 0xfffffffc broadcast 10.0.0.3 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC> ether a2:b4:3c:2c:e7:d4 hwaddr a2:b4:3c:2c:e7:d4 inet6 fe80::a0b4:3cff:fe2c:e7d4%em1 prefixlen 64 scopeid 0x2 inet 192.168.5.254 netmask 0xffffff00 broadcast 192.168.5.255 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> media: Ethernet autoselect (1000baseT <full-duplex>) status: active
@stephenw10 said in Lost LAN connection:
What does proxmox show for the state of the interface?
Proxmox show the NIC as enabled but no traffic on the LAN
-
I assume em1 is LAN?
You might try enabling promiscuous mode on em1 as a test. It shouldn't be needed but it is a difference.
You can also try using a VirtIO NIC for LAN.
Steve
-
Hi all
As I lost again twice the connection on the LAN interface I tried this :
https://docs.netgate.com/pfsense/en/latest/hardware/troubleshooting-lost-traffic-or-disappearing-packets.html
I will see if there is any improvement
-
Did you try using VirtIO NICs instead?
-
@stephenw10 yes and it solved my issue ! :) (https://docs.netgate.com/pfsense/en/latest/virtualization/virtio-driver-support.html)
-
Good to hear. Thanks for reporting back.