pfSense loses internet connection with no error

MaxPresi

@tinfoilmatt To resolve this, I restart pfSense, through the GUI itself.

The infrastructure is different now, I'm at a different company, but the ISP is the same.

The infrastructure is simpler, 4 UniFi APs and 1 Dell L2 Switch.

tinfoilmatt

@MaxPresi The best I could suggest is you gotta troubleshoot this during an outage, from pfSense before cycling anything. There are the various diagnostic/troubleshooting tools available—plus I wonder if simply unplugging and replugging the physical cable between pfSense and the Mikrotik would get traffic flowing again.

Is there anything in partcular that has you leaning toward it being a pfSense issue in particular? Or is that just kind of where you're at for the moment?

Minimally-standardized Ubiquiti hardware on the network is—woof. Many an exasperated troubleshooter has simply upgraded or otherwise replaced Ubiquiti gear after fruitlessly chasing down "broadcast storms" or whatever tf.

tinfoilmatt

@MaxPresi Also not intending to insult your clear experience whatsoever by saying something like 'you know logs are your friend.' But you know logs are your friend.

There has to be clues elsewhere either on other systems' logging, and/or by turning up the verbosity dial on the logs you're already relying on.

MaxPresi

@tinfoilmatt This is the problem; it only happens once a day, and I had to get it working again because of the live stream.

Right now, I'm trying everything. I just spoke to the ISP on the phone, and they told me they also have nothing to report from their logs, except for the LAN being active at the time I restarted the firewall.

There's also the fact that the machine it's on is a complete piece of junk, an FX 4300 with 3 Realtek LAN ports. I'm reinstalling it on an R430 (8 Broadcom) and see if that solves it.

I checked the logs from beginning to end and there were no errors. I checked the 3 days the errors occurred, but I didn't find any errors.

tinfoilmatt

@MaxPresi said in pfSense loses internet connection with no error:

Realtek LAN ports

Fairly certain I've seen nothing but bad things said about this flavor of NIC around here, so I think you're headed down a better track migrating hardware.

You haven't said if you're running CE or Plus, so it may or may not even be relevant—but be aware that the swap will probably generate a new NDI. My understanding is that this only affects a Plus install, and only until you obtain Netgate's assistance. But if you're CE, any concern here is moot.

stephenw10

Yeah check the system logs for watchdog errors from the Realtek driver. If you see them try the alternative driver or use a different NIC.

But sendto error 65 implies no route to the gateway IP. That pretty much means the WAN must have lost it's address unless you have a weird gateway setup. The system logs should show something.

MaxPresi

@stephenw10
Unfortunately not, that's the default. No error. It only shows up when I access it to restart the firewall.

Nov 13 10:00:01	kernel		---<<BOOT>>---
Nov 13 10:00:01	syslogd		kernel boot file is /boot/kernel/kernel
Nov 13 09:58:53	syslogd		exiting on signal 15
Nov 13 09:58:53	reboot	6512	rebooted by root
Nov 13 09:58:51	php-fpm	385	/diag_reboot.php: Stopping all packages.
Nov 13 09:57:01	php-fpm	53207	/index.php: Successful login for user 'x' from: x (Local Database)
Nov 13 09:57:00	sshguard	1834	Now monitoring attacks.
Nov 13 09:57:00	sshguard	16219	Exiting on signal.
Nov 13 08:22:00	sshguard	16219	Now monitoring attacks.
Nov 13 08:22:00	sshguard	9784	Exiting on signal.
Nov 13 06:47:00	sshguard	9784	Now monitoring attacks.
Nov 13 06:47:00	sshguard	83509	Exiting on signal.

Nothing has happened today, at least not yet...

stephenw10

Nothing in the routing or gateways log at the time?

Check the routing table if/when it happens again. Check the output of ifconfig directly.

To see that error it pretty much has to be the WAN losing it's address. Unless the gateway is outside the WAN subnet, as I say, but that's very unusual.

tinfoilmatt

@stephenw10 said in pfSense loses internet connection with no error:

To see that error it pretty much has to be the WAN losing it's address

Which, in fairness, still says nothing as to root cause.

tinfoilmatt

@MaxPresi What do you have selected in the Default gateway IPv4 dropdown under System / Routing / Gateways? Automatic? Or the MikroTik specifically selected? If the former, you might try out the latter.

stephenw10

Indeed. I'd expect to see something logged though.

MaxPresi

@tinfoilmatt The default gateway was specified as the ISP's gateway, GW_WAN.

I've already changed the hardware; now it's an R430 with 8 Broadcon LANs, a Xeon E5-2609 v4, 32 GB of ECC RAM, and a 480 GB SSD (a bit overkill), running version 2.8.1, the previous version was 2.7.2...

I think pfSense will be happy now.