All traffic stopped, looks a bug to me!

GTA_doum

Hello,
I have now two pfSense from two different environment that has this strange buggy behaviour. The first one has been doing it since a long time, all LAN traffic is blocked by pfSense, no warning, no reason that could be found, no hardware issues, traffic just stops, WAN to LAN or LAN to WAN. I tried everything (yes, every detail possible, modifying or replacing) and the issue always comes back, traffic is blocked at large, the only to restart it is to restart the router.
I taught it was only because of the environment where the pfSense was and I could not figure out what was specific to it. But since a month or two, there is now a second environment where pfSense is starting that behaviour, all LAN traffic is blocked for no reason. In System logs, it shows that the LAN NIC has been stopped and restarted, so I guess pf is not reloaded properly, which results in a complete blockage. So far, it is the best explanation I have, but I cannot find why traffic is stopped. When it happens could be at any moment of the day and the next time it would do it could be few days or many weeks later.

Any one have seen this and what is the resolution so that it stops doing that?

stephenw10

@GTA_doum said in All traffic stopped, looks a bug to me!:

In System logs, it shows that the LAN NIC has been stopped and restarted,

What exactly is shown in the logs?

GTA_doum

@stephenw10 Nov 19 22:50:42 kernel re1: link state changed to DOWN
Nov 19 22:50:42 kernel re1: watchdog timeout
Nov 19 22:50:42 check_reload_status 1050 Linkup starting re1
Nov 19 22:50:13 check_reload_status 1050 Reloading filter
Nov 19 22:50:13 php-fpm 27945 /rc.newipsecdns: IPSEC: One or more IPsec tunnel endpoints has changed its IP. Refreshing.

patient0

@GTA_doum if I read it correct the device has a RealTek (dua port?) network card. Support for them is notorious bad in FreeBSD. What RealTek NIC is it?

There is an alternative RealTek driver you can install, see
https://forum.netgate.com/post/1229634

pkg install realtek-re-kmod

And if it works better with that driver, make it permanent:

echo 'if_re_load="YES"' >> /boot/loader.conf.local
echo 'if_re_name="/boot/modules/if_re.ko"' >> /boot/loader.conf.local

But your best option is to use an Intel network card.

GTA_doum

@patient0 I tried other hardware, did not make any difference. To make it even more crazy, the first site where it is happening, they have two sites connected via IPSec. The two pfSense have exactly the same hardware, but only one of the two makes this issue! I tried replacing everything one by one, even the router itself, NIC, network cables, switch... Issue is always coming back.
I will try this driver, but with everything I tried, I would surprise it changes anything.

Few months ago, another pfSense started behaving with the same issue and it is unrelated to the first site. It was working great since its first install one or two years ago!

patient0

@GTA_doum said in All traffic stopped, looks a bug to me!:

I tried other hardware,

And the other hardware did not have a RealTek nic?

The log you posted shows that re1 has a watchdog timeout and that is a typical RealTek driver issue (run a search for 'kernel re watchdog timeout' in this forum, sort for post time).
That would indicate that in that case the network card did play a role in the issue.

GTA_doum

@patient0 I just checked and yes, the mini computer I used to test has a Realtek NIC (most onboard NIC have Realtek cards, even a lof of USB adapter have a Realtek chipset in them). On the first site mentioned, I can install another NIC and will try an Intel one. For mini-computers, I cannot unfortunetely install another NIC, there is no space in those box to install a PCIe card.
What I fail to understand is why a pfSense was working for over a year and suddenly starts acting up...

GTA_doum

I just recalled what we did at that place few months ago, we upgraded pfSense to 2.8.0 and lately to 2.8.1. Which means the Realtek driver in 2.8 is indeed misbehaving, if the driver is the cause.
The first site I mentioned was always acting up, even with version 2.7 of pfSense.

tinfoilmatt

@GTA_doum Sooooooo, not a bug then.

GTA_doum

@tinfoilmatt Well, if the driver has a bug, that is included in pfSense/BSD, it is still a bug...

tinfoilmatt

@GTA_doum If Netgate has provided you with a modified driver that resolves your issue, then perhaps FreeBSD's issue tracker will be receptive to your report.

GTA_doum

@tinfoilmatt I'll gladly do it once confirmed. It will take few months to confirm, as the issue sometimes could happen after few weeks. It's been few times I taught it was fixed, but then reappeared. I am still unsure what triggers it, maybe the amount of traffic, but really, I could not make a link of cause to effect.