Official Realtek Driver Binary 1.95 For 2.4.4 Release

gordon

[2.4.4-RELEASE][root@firewall.test.net]/root: grep re1 /var/run/dmesg.boot
re1: <Realtek PCIe GBE Family Controller> port 0xd000-0xd0ff mem 0xa1104000-0xa1104fff,0xa1100000-0xa1103fff at device 0.0 on pci3
re1: Using Memory Mapping!
re1: Using 1 MSI-X message
re1: ASPM disabled
re1: version:1.95.00
re1: Ethernet address: 00:1e:06:45:02:83
re1: Ethernet address: 00:1e:06:45:02:83

[2.4.4-RELEASE][root@firewall.test.net]/root: route -n
route: usage: route [-46dnqtv] command [[modifiers] args]

[2.4.4-RELEASE][root@firewall.test.net]/root: arp -an | grep re1
? (92.221.80.253) at 00:1e:06:45:02:83 on re1 permanent [ethernet]
? (92.221.80.1) at 00:02:00:01:00:01 on re1 expires in 97 seconds [ethernet]

[2.4.4-RELEASE][root@firewall.test.net]/root: tcpdump -c 20 -n -i re1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on re1, link-type EN10MB (Ethernet), capture size 262144 bytes
03:49:11.244621 IP 92.221.80.253.49228 > XX.XX.XX.XX.XXX: 26302+ XXXXXXX (91)
03:49:11.244626 IP 92.221.80.253.49228 > XX.XX.XX.XX.XXX: 26302+ XXXXXXX. (91)
03:49:11.246135 IP 92.221.80.253.25525 > XX.XX.XX.XX.XXX: 28630+ XXXXXXX. (35)
03:49:11.246139 IP 92.221.80.253.25525 > XX.XX.XX.XX.XXX: 28630+ XXXXXXX. (35)
03:49:11.262982 IP 92.221.80.253.25525 > XX.XX.XX.XX.XXX: 42079+ XXXXXXX. (57)
03:49:11.262985 IP 92.221.80.253.25525 > XX.XX.XX.XX.XXX: 42079+ XXXXXXX. (57)
03:49:11.263013 IP 92.221.80.253.25525 > XX.XX.XX.XX.XXX: 23104+ XXXXXXX. (53)
03:49:11.263016 IP 92.221.80.253.25525 > XX.XX.XX.XX.XXX: 23104+ XXXXXXX. (53)
03:49:11.413788 IP 92.221.80.253.25525 > 109.247.114.4.53: 46408+ XXXXXXX. (41)
03:49:11.413793 IP 92.221.80.253.25525 > 92.220.228.70.53: 46408+ XXXXXXX. (41)
03:49:11.413823 IP 92.221.80.253.25525 > 109.247.114.4.53: 30408+ XXXXXXX. (41)
03:49:11.413826 IP 92.221.80.253.25525 > 92.220.228.70.53: 30408+ XXXXXXX. (41)
03:49:11.414619 IP 92.221.80.253.25525 > XX.XX.XX.XX.XXX: 9197+ XXXXXXX. (31)
03:49:11.414622 IP 92.221.80.253.25525 > XX.XX.XX.XX.XXX: 9197+ XXXXXXX. (31)
03:49:11.414695 IP 92.221.80.253.36009 > XX.XX.XX.XX.XXX: 247+ XXXXXXX. (31)
03:49:11.414699 IP 92.221.80.253.36009 > XX.XX.XX.XX.XXX: 247+ XXXXXXX. (31)
03:49:11.490012 IP 92.221.80.253.49228 > XX.XX.XX.XX.XXX: 26302+ XXXXXXX. (91)
03:49:11.490015 IP 92.221.80.253.49228 > XX.XX.XX.XX.XXX: 26302+ XXXXXXX. (91)
03:49:11.491250 IP 92.221.80.253.25525 > XX.XX.XX.XX.XXX: 28630+ XXXXXXX. (35)
03:49:11.491253 IP 92.221.80.253.25525 > XX.XX.XX.XX.XXX: 28630+ XXXXXXX. (35)
20 packets captured
20 packets received by filter
0 packets dropped by kernel

gordon

@kiokoman are you sure the driver has been loaded ?
Just that I facing the same problem after also....

stephenw10

@gordon said in Official Realtek Driver Binary 1.95 For 2.4.4 Release:

re1: version:1.95.00

It shows it's loaded there. The FreeBSD in kernel driver doesn't show the version as far as I know. I only have 2.5 dev box up right now but:

[2.5.0-DEVELOPMENT][root@apu.stevew.lan]/root: grep re1 /var/run/dmesg.boot
re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0x2000-0x20ff mem 0xf7c00000-0xf7c00fff,0xf7b00000-0xf7b03fff irq 17 at device 0.0 on pci2
re1: Using 1 MSI-X message
re1: ASPM disabled
re1: Chip rev. 0x2c000000
re1: MAC rev. 0x00200000
miibus1: <MII bus> on re1
re1: Using defaults for TSO: 65518/35/2048
re1: Ethernet address: 00:0d:b9:89:3f:11
re1: netmap queues/slots: TX 1/256, RX 1/256

Try commenting out the line in loader.conf.local and rebooting.

Steve

kiokoman

@gordon said in Official Realtek Driver Binary 1.95 For 2.4.4 Release:

[2.4.4-RELEASE][root@firewall.test.net]/root: grep re1 /var/run/dmesg.boot
re1: <Realtek PCIe GBE Family Controller> port 0xd000-0xd0ff mem 0xa1104000-0xa1104fff,0xa1100000-0xa1103fff at device 0.0 on pci3
re1: Using Memory Mapping!
re1: Using 1 MSI-X message
re1: ASPM disabled
> re1: version:1.95.00
re1: Ethernet address: 00:1e:06:45:02:83
re1: Ethernet address: 00:1e:06:45:02:83

yes it is like stephenw10 say,
version 1.95.00 is the official driver, the one shipped with freebsd does not output the version when it load

gordon

@kiokoman said in Official Realtek Driver Binary 1.95 For 2.4.4 Release:

@gordon said in Official Realtek Driver Binary 1.95 For 2.4.4 Release:

[2.4.4-RELEASE][root@firewall.test.net]/root: grep re1 /var/run/dmesg.boot
re1: <Realtek PCIe GBE Family Controller> port 0xd000-0xd0ff mem 0xa1104000-0xa1104fff,0xa1100000-0xa1103fff at device 0.0 on pci3
re1: Using Memory Mapping!
re1: Using 1 MSI-X message
re1: ASPM disabled
> re1: version:1.95.00
re1: Ethernet address: 00:1e:06:45:02:83
re1: Ethernet address: 00:1e:06:45:02:83

yes it is like stephenw10 say,
version 1.95.00 is the official driver, the one shipped with freebsd does not output the version when it load

Thank you,

I reinstalled pfsense today and are using WAN on re0 instead of re1 and LAN on re1, so fare so fare, new record in uptime... cross my fingers....

But I cannot understand these shall make any different ?

stephenw10

The driver provided by Realtek can commonly help if you're seeing watchdog timeout errors. But it sounds like that's not what you're seeing so it may not make any difference here. You have nothing to lose by trying it though.

Steve

gordon

@stephenw10
Problem is still present, card stop working and report DOWN in status under

Remember both card is identical network card... strange it’s only WAN card who goes down.

Have now tried disabling «Disable hardware checksum offload»

Is it possible to make the card recycle DISABLE and then ENABLE when these happens, way around the problem until I find a fix?

gordon

«Disable hardware checksum
Was not fixing the problem....

Does anybody know how I can auto disable and enable the card when it goes down?

kiokoman

try with
/etc/rc.d/netif restart re0

gordon

@kiokoman said in Official Realtek Driver Binary 1.95 For 2.4.4 Release:

try with
/etc/rc.d/netif restart re0

Thank you, I need some script to detect when it shall happens also...

I found some things to try another place on these forum, so I have added these lines to the boot config:
net.inet.tcp.tso=0
hw.pci.enable_msix=0
hw.pci.enable_msi=0
hw.re.tso_enable=0

to the boot loader config, and for now I haven`t seen the issue again, need more hours... will report back later.

stephenw10

Yes, definitely disable all hardware offloading features. Check what is still enabled using ifconfig re1.

Did you try re-assigning the NICs? Does the fault follow re1 or remain on WAN?
It's likely you are downloading far more traffic that uploading. Whichever NIC is WAN is doing the receiving of that traffic and a number of those hardware offloading features are receive only.

Steve

gordon

@stephenw10 said in Official Realtek Driver Binary 1.95 For 2.4.4 Release:

Yes, definitely disable all hardware offloading features. Check what is still enabled using ifconfig re1.

Did you try re-assigning the NICs? Does the fault follow re1 or remain on WAN?
It's likely you are downloading far more traffic that uploading. Whichever NIC is WAN is doing the receiving of that traffic and a number of those hardware offloading features are receive only.

Steve

I reinstalled the whole pfsense yesterday and then I swapped so re0 is WAN and re1 is LAN.
The problem have always followed the card I have assigned to be WAN.

The problem has not accour after I disabled al the things mention above from the HW i the boot loader. But there is a strange thing the uptime on the box is never 1 hour and 47 minutes, but I have not restarted the box today..... I think there are something wrong with the clock... its not going correct... it`s deviate more and more from the "real time"....

Maybe there is a releation here ? lead time on the DHCP maybe got problem...

I will look into getting the clock correct first.

stephenw10

Make sure NTP is configured and working. All APUs are old now, the CMOS battery may be dead.

Edit: Is this an APU? Not sure where I pulled that from..

Steve

gandalfk7

This post is deleted!

gandalfk7

@TheNarc

I just subscribed to thank you for your work,
I was going mad trying to understand the issue and finally this morning I found a post that directed me here.

The issues I had on my ci323 nano seems all solved now.

thanks again

matrix200 · Current network "hardware" : Running 2.2RC in Virtualbox 4.2.16.

@TheNarc, just wanted to thank you and post an interesting update.
Apparently your fix also solves an issue I had been experiencing with passing the realtek card through to a VM running Pfsense on a Centos 8 host.
Running on J4105-ITX motherboard.
The error occured with pretty much any realtek card I tried including the onboard one.
Until now, it would give an nasty DMAR error and the card would always stop working after pfsense VM reboot.
Sometimes, it would show a watchdog timeout, sometimes would just quetly stop responding.
With this driver, the card is stable and there is no DMAR error in Host's dmesg.

knight-of-ni

I've had three Zotac CI32x boxes deployed, running pfsense for a few years now, and I had not run into the dreaded watchdog timeout error until after upgrading to pfSense 2.4.4. Now the lan interface of two out of three boxes goes silent about once a month. Fortunately, since it it is the lan interface, I just reboot them remotely.

Thanks for doing the work building this driver. This saved me quite a bit of time.

Question:
With this new driver, can/should I re-enable hardware offloading under Advanced-> Networking?

I've also got the following set in loader.conf.local. These were suggested as workarounds in another thread. Is it safe to remove these?
hw.re.msi_disable=1
hw.pci.enable_msix=0
hw.pci.enable_msi=0

TheNarc

@knnniggett I'm glad to hear the driver has been useful. I'm sorry to hear about the LAN interface flakiness; that's not something I've seen (yet) in on the two CI323s I run. With regard to your questions:

I would not recommend re-enabling hardware offloading. I believe that Realtek claims to support it, but from what I understand it can be prone to bugginess and you're unlikely to see any tangible benefit from it. Also, it's the sort of thing that (I think) you wouldn't necessarily know right away if it works. So you might try it and it would seem fine, but some time down the line you'd have a problem and, if you're anything like me, probably forget that you'd changed that setting a while back and not immediately realize that it's related.

As for the loader.conf.local entries, at least based on my experience it should be fine to get rid of them. I don't have them set on either of my boxes right now and haven't had any issues.

Napsterbater

@knnniggett said in Official Realtek Driver Binary 1.95 For 2.4.4 Release:

I've had three Zotac CI32x boxes deployed, running pfsense for a few years now, and I had not run into the dreaded watchdog timeout error until after upgrading to pfSense 2.4.4. Now the lan interface of two out of three boxes goes silent about once a month. Fortunately, since it it is the lan interface, I just reboot them remotely.

Thanks for doing the work building this driver. This saved me quite a bit of time.

Question:
With this new driver, can/should I re-enable hardware offloading under Advanced-> Networking?

I've also got the following set in loader.conf.local. These were suggested as workarounds in another thread. Is it safe to remove these?
hw.re.msi_disable=1
hw.pci.enable_msix=0
hw.pci.enable_msi=0

I do not have these anymore either with the new driver.

Should be good to remove.

fromage9747

I made myself a new router after my previous one failed(power supply fried).

I got myself a GIGABYTE GA-N3160N-D3V motherboard with an embedded CPU and DUAL NIC's perfect for making a nice small compact router with pfsense. Everything was hunky-dory for a few months with no issues until one day I started encountering this "watchdog timeout" issue.

I thought that the CPU had overheated and bent the solder like how the Xbox 360 did with the RROD. The CPU was only at 65 degrees Celcius...

I installed fans in the case and ensured that the temp never grew past 40 degrees Celcius. The issue still persisted and I thought I would now need to RMA the board.

As I cannot go without internet for longer periods of time due to working from home I needed to get my original pfsense router up and running before sending this board in.

Whilst in the process of doing so I was doing some uploading of work to Jira when one gif file would repeatedly kill the router. I was stunned, how on earth could a 2mb file kill the router?!?!?!

I thought I would do one last search online to see if I could find anything software related that might be the issue and I stumbled upon this thread.

I can confirm that applying the Realtek driver has now solved my issue!

Thank you!