All traffic stopped, looks a bug to me!

GTA_doum

@stephenw10 Nov 19 22:50:42 kernel re1: link state changed to DOWN
Nov 19 22:50:42 kernel re1: watchdog timeout
Nov 19 22:50:42 check_reload_status 1050 Linkup starting re1
Nov 19 22:50:13 check_reload_status 1050 Reloading filter
Nov 19 22:50:13 php-fpm 27945 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.

patient0

@GTA_doum if I read it correct the device has a RealTek (dua port?) network card. Support for them is notorious bad in FreeBSD. What RealTek NIC is it?

There is an alternative RealTek driver you can install, see
https://forum.netgate.com/post/1229634

pkg install realtek-re-kmod

And if it works better with that driver, make it permanent:

echo 'if_re_load="YES"' >> /boot/loader.conf.local
echo 'if_re_name="/boot/modules/if_re.ko"' >> /boot/loader.conf.local

But your best option is to use an Intel network card.

GTA_doum

@patient0 I tried other hardware, did not make any difference. To make it even more crazy, the first site where it is happening, they have two sites connected via IPSec. The two pfSense have exactly the same hardware, but only one of the two makes this issue! I tried replacing everything one by one, even the router itself, NIC, network cables, switch... Issue is always coming back.
I will try this driver, but with everything I tried, I would surprise it changes anything.

Few months ago, another pfSense started behaving with the same issue and it is unrelated to the first site. It was working great since its first install one or two years ago!

patient0

@GTA_doum said in All traffic stopped, looks a bug to me!:

I tried other hardware,

And the other hardware did not have a RealTek nic?

The log you posted shows that re1 has a watchdog timeout and that is a typical RealTek driver issue (run a search for 'kernel re watchdog timeout' in this forum, sort for post time).
That would indicate that in that case the network card did play a role in the issue.

GTA_doum

@patient0 I just checked and yes, the mini computer I used to test has a Realtek NIC (most onboard NIC have Realtek cards, even a lof of USB adapter have a Realtek chipset in them). On the first site mentioned, I can install another NIC and will try an Intel one. For mini-computers, I cannot unfortunetely install another NIC, there is no space in those box to install a PCIe card.
What I fail to understand is why a pfSense was working for over a year and suddenly starts acting up...

GTA_doum

I just recalled what we did at that place few months ago, we upgraded pfSense to 2.8.0 and lately to 2.8.1. Which means the Realtek driver in 2.8 is indeed misbehaving, if the driver is the cause.
The first site I mentioned was always acting up, even with version 2.7 of pfSense.

tinfoilmatt

@GTA_doum Sooooooo, not a bug then.

GTA_doum

@tinfoilmatt Well, if the driver has a bug, that is included in pfSense/BSD, it is still a bug...

tinfoilmatt

@GTA_doum If Netgate has provided you with a modified driver that resolves your issue, then perhaps FreeBSD's issue tracker will be receptive to your report.

GTA_doum

@tinfoilmatt I'll gladly do it once confirmed. It will take few months to confirm, as the issue sometimes could happen after few weeks. It's been few times I taught it was fixed, but then reappeared. I am still unsure what triggers it, maybe the amount of traffic, but really, I could not make a link of cause to effect.

stephenw10

I mean there is a bug there but it's known. The Realtek hardware is such that there is little motivation for devs to fix it. And the newer Realtek NICs are better anyway.

But, yes, if you see that watchdog error from the re(4) driver then definitely try the alternative driver.