One interface loses internet access and I could get it back only after reboot the pfsense

[mokey_fraggle]

The next time it happens, can you screencap what mtr shows going to the impacted IP addresses. It will at least show what route the packets are taking. Where does the tunnel teminate? a provider? have you tried setting a static /32 route to push that traffic through the VPN tunnel?

Regards,
--Mokey

[ady2]

@mokey_fraggle
It happened again and it looks like all my interfaces have loose the internet connection.

here are the results for ping yahoo.com from LAN

and here is the results for ping yahoo.com from WAN

Running ping yahoo.com from my mac:

Looks like DNS resolver is working as DNS Lookup was given results for yahoo.com

Restarted the DNS Resolver without any change.

Finally reset all states and internet is back.

In mean time (previous week) I modified my settings trying to eliminate other possible dependences. I added a new Intel network card with 2 ports to addition to existing Intel 4 port card.
Stopped all other services like pfBlockerNG, snort, paused firewall rules to direct clients through VPN.
Now I have minimum services running:

Any suggestion will be very appreciated

Thanks

[netblues]

@ady2 It is clear its not a dns issue.
And since pf can ping, certainly not a connectivity issue too.
We do need a traceroute (or mtr) to see where the packets are going.
So clearing states solves the issue.
I suspect that clearing states forces vpn to reestablish.
Try restarting the vpn next time, it will confirm it.
Look at vpn logs when it happens

[ady2]

@netblues
In this specific case only my laptop has a firewall rule to go through VPN client on one of the interfaces, everything else and all other interfaces use the WAN gateway.
But now I have stopped all VPN clients and paused the firewall VPN rule for my laptop, now only the VPN server is on to allow me to connect remotely.

Do you mean to run Diagnostics/Traceroute on specific interface for a yahoo.com, right?

[netblues]

I mean try a traceroute from a pc/laptop etch and not from pf..

[ady2]

@netblues
Thanks a lot for your help
Will try traceroute next time.

[ady2]

Today it happened again and affected at least 2 interfaces. I was on one of it but on my work vpn and have internet till disconnected from work vpn.
The trace-route from my mac show that it reach the pfsense and nothing else. Please see the trace-route screenshot:
ping was timeout.
Verified OpenVPN and found that my clients are on, so disabled them one by one and suddenly the internet was back.
Based on this I'm thinking that my issue is related with one or all vpn client(s).
Going to keep them off and see if the internet will not disappear for a week or two, to confirm that.

Interesting why internet is blocked even on interfaces that never had VPN gateway assigned or any firewall rules for vpn. I have created a brand new interface for kids devices and today's internet outage affected that too.

One of the reason to switch to pfsense was to keep majority of devices connected to internet through vpn and that is giving me troubles now. It could be that my settings have a wrong setup, but then why everything is working fine till broke the whole home internet (only the WAN interface has internet when it happens).

[ady2]

Again same issue this evening (no internet on all interfaces.
Same symptoms, no trace-route after pfsense, ping timeout.
At this time no vpn clients on, only openVPN seerver was on.
Restarting the DNSResolver and internet is back. What the ... ?

[kiokoman]

do you have any package installed? like suricata / snort? do you have anything in dmesg ?

[ady2]

@kiokoman
I have next services active:

I have run "dmesg" on pfsense console, but don't know what to look for (I could see that lines a repeating 4 times) dmseg_07_21.txt.
Any help Welcome

[kiokoman]

do the dmesg when the problem present itself

[Asamat]

Can you

• check Status/Gateways menu
• run 'ifconfig -a'
• run 'netstat -rn'
• run Packet Capture on WAN with promiscuous mode enabled (Diagnostics/Packet Capture) to check if there is any in/out traffic
  when it happens again?

[ady2]

After 29 days of working without any issue my problem was back, no internet on a few interfaces that I have checked (same symptoms, no internet, ping timeout from interface as source). It was late night and needed internet to finish something so just restarted the pfsense and everything was back and working. Sorry, didn't have time to troubleshoot. Will hope that next time will have more to report.
Thanks to everybody for their help @Asamat, @kiokoman, @netblues, @mokey_fraggle

p.s. No vpn clients active on any interface, only vpn server was on. Nothing was changed during that time except the BandwithD packet was added to monitor the internet stats on ~ 08/18 and the problem occurred on 08/26.

[Raul Ramos]

@ady2 I have to renew the interface manually after WAN interface (DHCP mode) lose connection, just power cycle the ONT and interface tourn red and 100% packet loss.

[ady2]

Happened today again (no internet on all interfaces). ping timeout, traceroute only till pfsense ip address. Verified and I have no vpn clients active, only the vpn server so I restarted it and my internet is back.
How that is possible that vpn server is blocking internet access on all interfaces?

[Gertjan]

The VPN server is just listening on a port like 1194 on an interface like WAN.
If there are no incoming connections, then it 's doing close to nothing.

Mine is listing on WAN for month now, never had any issues.
You have quit a few interfaces : swap the WAN with some other NIC, and see if the problems moves.

[kiokoman]

sorry i didn't see the link to your dmesg
there.. apart from UP and DOWN
nothing useful

my bet.. or your network card go overload/overheat for some reason and stop working or driver are not working well anymore, especially if the network card is some chinese clone. they suddenly stop working reliable and this can happen. maybe it's time to change it.

if I were you I would reset / format pfsense after making a configuration backup. restore everything and see if the problem persist.
if it persist i would change the network card

[ady2]

@kiokoman said in One interface loses internet access and I could get it back only after reboot the pfsense:

sorry i didn't see the link to your dmesg
there.. apart from UP and DOWN
nothing useful

my bet.. or your network card go overload/overheat for some reason and stop working or driver are not working well anymore, especially if the network card is some chinese clone. they suddenly stop working reliable and this can happen. maybe it's time to change it.

if I were you I would reset / format pfsense after making a configuration backup. restore everything and see if the problem persist.
if it persist i would change the network card

@kiokoman
I was thinking that it could be the nic card that is causing those issues, so I have for some time (around 3-4 months) a second NIC card with 2 ports in addition to 4 port Chinese clone (bought as new for ~ $45 on ebay). When Installed the second nic with 2 ports, I switch a few interfaces to that card to check if there was the 4 ports card fault, and found same issue again and again. It will be quite strange that both NIC cards are overloaded at the same time or they both stop working at the same time. Could try to switch the WAN port to one of the nic cards port (my WAN is using my motherboard network port now) to check if this will block the WAN connection completely as it's happening now with my LAN interfaces.

BTW, my pfsense was dead with the "PHP Startup: Unable to load Dynamic Library." problem after 2 days I have upgraded to 2.4.4p3 version. So I have to install everything and restore from backup and even after that the problem still happens time to time, same as before that.

Is there any way to check if the NIC is the root cause ?

[kiokoman]

does it work if you force the network card of your laptop to use dnsfrom google for example? like 8.8.8.8 ?

[ady2]

@kiokoman said in One interface loses internet access and I could get it back only after reboot the pfsense:

does it work if you force the network card of your laptop to use dnsfrom google for example? like 8.8.8.8 ?

I have a firewall rule that is blocking all dns requests to other sources except pfsense, will pause that rule and try next time and report back