Package realtek-re-kmod198 for pfSense 2.8.0 (amd64)

SpunkThing

First of all thank you for your quick reply and additional clarification!

Another funny thing is that you mentioned disabling IPv6 system-wide might help some. Haven't seen that suggestion anywhere else so far so of course I went ahead and disabled it.
Up until about 2 hours ago it was enabled even though I'm not using IPv6, possibly something to do with too much log-spam when that checkbox was ticked.
I had/have per-interface no-log IPv6 block rules to avoid flooding the logs.

Maybe that's a backwards way of doing things and feel free to tell me if so.

Anyway, disabling IPv6 seemed like a sensible and harmless way to test if things improve so I fired up my BT client. It's been running for a little over two hours at the time of writing, meanwhile YouTube is playing continually and there's no obvious signs of anything going awry so far. Granted, I did tweak a few settings in my BT client such as maximum number of total and per-torrent connections and I haven't hit speeds over 200Mbps yet. I've seen my poor old scrap machine choke on less though, so you may just be on to something here! (I say that with a moderate amount of caution, hehe, don't want to jinx it already)

As for the more technical details you mentioned, I'd love to get into that but my knowledge of networking is limited at best. But the more I read the more I learn, and that is never a bad thing!

I did also read that the 1.98 was pretty solid on 2.7.2 so that's where my hunt started. Just couldn't find any pre-compiled packages and assumed there wouldn't be any until this crossed my path. And I have to say, I'm rather curious now.

You also mentioned the latest driver (1100.00) performs poorly, what do you mean by that? Lower throughput, higher latency, packet loss?
But it did "work" for you, right? For me it made the entire system unusable as the WAN interface just wouldn't stay up for more than a couple of minutes at a time.
Possibly a driver-hardware mismatch? I assumed these Realtek drivers were a one fits all package but that's based on absolutely nothing.

Besides that my knowledge of the inner workings of BSD is laughable at best and far, far worse than that of Linux. And I manage to regularly break things there even afters years of using it.
So it could have easily been an error on my part that caused the driver not to work at all.
I did have several attempts at it and may have even tried to install the 1.97 or 1.98 driver by using links to older FreeBSD repos. Can't quite remember everything I've tried, but the end result after hours and hours of monkying around was a system that was unrecoverable by conventional means.

Ended up just biting the bullet, did a full reset and and used the configuration backup from just before this carnage started.

Pretty impressive speeds btw! If that doesn't do one of those Realtek NICs in it'll probably work just fine.
Out of curiosity - have you tried testing with something that creates a large number of states / simultaneous connections?
I had a feeling / suspicion that this may contribute even more to interface flapping (forgive me if I’m using that terminology the wrong way…) and/or gateways downright flat-lining.

And… Ha! Ha! Ha! Just as I was writing that last sentence YouTube stopped playing and a message on the forum popped up saying something like ‘it looks like you lost your connection’. ...I did indeed!

Sure enough the same thing happened again. It did take a lot longer than what I was used to, but at the same time it happened while my BitTorrent client hadn’t been doing much of anything for some time by the time I was writing that. About 30-35 active connections with a total upload speed of less than 100Mbps. That's nothing, so I'm not sure what caused this gateway slapper.

I had a quick peek in the logs, the States Summary page and checked states per interface.
No idea if there’s any relation but the computer I’m currently using looked as if it was bombarding the firewall with DNS queries. A few other connected devices also showed more states than I’d expect to see considering they weren’t actively using the internet.

Could just be a coincidence, perhaps the result of the gateway suddenly crashing instead of the cause, but it stood out.
I’ll see if I can attach some screenshots I took just after it happened. Maybe you can see something I missed or just don’t understand. If you feel like it, of course. :-)

In any case, the idea of trying your creation is starting to appeal to me more and more.
I'm really curious to see if there's any improvement to be had for my system after reading about your own successful testing.

One more thing, perhaps a little off-topic but hey... How do you use iperf to perform such tests? I was thinking about trying that as well to test throughput and stability but after reading the following I'm not sure how to go about it and if there's even any value in it when you're limited to your own home network.

https://docs.netgate.com/pfsense/en/latest/packages/iperf.html

Guess I may be having a closer look at that shiny new driver in a little while. Seems more interesting than sleep anyway. 🤫

SpunkThing

@zeroepoch

Added a few snippets. Maybe you or somebody else can make sense of what happened there.
As you can tell from the screenshots, just after 4 AM things went pear-shaped.
For me looking at logs like these feels a bit like trying to read an ancient Hindu manuscript or something. Chances are I'd understand more of the latter one, but maybe there's a clue in there for somebody with a better understanding of these things.

zeroepoch

@SpunkThing

I didn't do extensive testing after disabling IPv6 since the end goal was to keep IPv6 on long-term, so looked for another solution. I did notice more packet loss than expected and it appeared (my perception) at least to stall the connection more often.

I haven't tried anything with a high number of high bandwidth connections like a large torrent. Probably at some point with 1.98.00 on pfSense 2.7.2. We have 4 people in the house all watching movies and things at the same time, and a few dozen other servers, devices, etc. According to the states status page I currently have around 1500 open connections, which to be honest surprised me quite a bit.

One more thing, perhaps a little off-topic but hey... How do you use iperf to perform such tests? I was thinking about trying that as well to test throughput and stability but after reading the following I'm not sure how to go about it and if there's even any value in it when you're limited to your own home network.

I installed the iperf3 package directly on the router over SSH. Used pkg install iperf3. Then ran iperf3 -s (for the server) on the router and iperf3 -c router (for the client) on my Linux desktop (there are Windows binaries as well). You can append -P #, which I typically use 4 to get the max connection speed, but it could be much higher if you want. The nice thing about iperf being peer-to-peer on your local network is you can see what your links/connections are capable of handling.

SpunkThing

@zeroepoch

Yeah, that's understandable. If you're going to use IPv6 your time would be much better spent looking for a solution wherein it it doesn't need disabling to have a functioning internet connection. That'd just be silly, hehe.

And around 1500 open connections without something like BitTorrent running is indeed quite something. I'm a bit surprised myself to be honest.
That's more or less the number of connections I typically see when I have a few active torrents, and quite possibly the absolute maximum that my pfSense machine will handle in it's current state.
The WAN interface has gone down plenty of times when dealing with smaller numbers, so maybe it's a combination of the total bandwidth used and the number of connections / states (not quite sure if they're the exact same thing, going to look that up now...)

Thanks for the quick iperf tutorial by the way! After reading that small bit of text I linked before it seemed rather pointless to perform a test like that within the confines of my own home network.
Perhaps I just didn't quite understand what it really said, or the person who wrote it didn't really know what he was talking about. Who knows.
I will definitely have a go at it now. Now as in - some time today.
And since it's a lazy day anyway, might as well try out that driver you managed to put together.

Hope it works - but even if it doesn't I appreciate people taking the time to try and come up with solutions and sharing them.
Maybe I can do my small part by being a pfTestDummy and report back my findings.
The worst that could happen is another irreparable pfSense installation, and if that happens I will probably just take it apart and finally give it the little upgrade that I've been postponing for months.

I'll keep you updated on how things work out. As best I can anyway.
Could be a couple of hours, could take more than a day if things go as did last night... 🥴

Have a great day! Or evening, night, afternoon... whichever. Have a good one!

zeroepoch

@SpunkThing

My first attempt was loading the package from freshports. Finding the extract URL was a challenge since you can't browse the package files. That reported some missing symbol I believe. Don't exactly remember now what the error was. The second attempt I built the driver using the latest commit from the FreeBSD-src repo. That complained that version didn't match, which I mentioned I had to find a close commit with matching version to fix it. In both those failed cases rather than crash it just didn't load the model and hence those network interfaces were missing.

stephenw10

From the bug it looks like it should be sufficient to simply disable hardware checksum offloading. Are you saying setting that in the webgui does not disable it for the driver?

TSO/LRO are disabled by default but are separate from Checksum.

Edit: Hmm in 25.03 at least it does appear to fail to disable RXCSUM_IPV6....

stephenw10

I will say I've been running that driver on an APU in 25.03 without issue for months. That's using 1G NICs though so maybe something specific to RTL8125.

zeroepoch

@stephenw10

Correct, it doesn't disable checksumming even when checked in the GUI. It's not a bug in the GUI because I can't even disable it manually using ifconfig. It must work somewhere, maybe FreeBSD 14, because that's what's recommended on the bug.

I think it's specific to the 2.5g NICs. I had an older box with 1g Realtek NICs and it didn't need this package to function. I just had it installed for improved stability. These 2.5g NICs don't show up at all until you install this package.

stephenw10

You see the same thing? Only RX checksum for IPv6 is still enabled?

[25.03-BETA][admin@apu.stevew.lan]/root: ifconfig -vvm re0
re0: flags=1008803<UP,BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=202018<VLAN_MTU,VLAN_HWTAGGING,WOL_MAGIC,RXCSUM_IPV6>
	capabilities=403c1a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,TXCSUM_IPV6>
	ether 00:0d:b9:37:30:10
	inet6 fe80::20d:b9ff:fe37:3011%re0 prefixlen 64 tentative scopeid 0x1
	media: Ethernet autoselect
	status: no carrier
	supported media:
		media autoselect
		media 1000baseT mediaopt full-duplex
		media 100baseTX mediaopt full-duplex
		media 100baseTX
		media 10baseT/UTP mediaopt full-duplex
		media 10baseT/UTP
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	drivername: re0

Looking again though that NIC only supports TX checksum.

zeroepoch

Hi @stephenw10,

I don't remember exactly now and it's different with the older driver I'm currently running, so I can't look it up, but I believe only TXCSUM_IPV6 was removed. For sure TXCSUM and RXCSUM were still there on 1.100.00 after running ifconfig re0 -rxcsum -txcsum -rxcsum6 -txcsum6.

What I get now with disable checksum offloading checked on the 1.98.00 driver is:

[2.8.0-RELEASE][admin@router]/root: ifconfig re0
re0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: LAN
	options=2018<VLAN_MTU,VLAN_HWTAGGING,WOL_MAGIC>
	ether xx:xx:xx:xx:xx:xx
	inet x.x.x.x netmask 0xffffff00 broadcast x.x.x.x
	inet6 x::x prefixlen 64 scopeid 0x1
	inet6 x::x prefixlen 64 scopeid 0x1
	inet6 x::x prefixlen 64
	media: Ethernet autoselect (2500Base-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

In this case everything checksum related was disabled as expected. The 1.100.00 driver as you are seeing as well doesn't allow clearing all the flags. The behavior appears to vary I guess depending on the particular chipset?

stephenw10

Yeah if you run ifconfig -vm re0 it should show you what the driver thinks the hardware is capable of too.

SpunkThing

@zeroepoch

Took me a while but I finally have an update.

Unfortunately it did exactly the same as the 1.100.00 driver.
And to make things even stranger, both these drivers left my pfSense installation inoperable even after uninstalling them.

I've not got the faintest idea what kind of voodoo witchcraft sorcery stuff is at work here.
The first time after installing your version of the 1.98 driver the WAN interface immediately came online after the system rebooted, so I thought bingo!
Less than 3 minutes later it collapsed just like before.

The only difference is that this time it was possible to bring it back online by disabling the interface in the GUI and then enabling it again. Not every time though.

I noticed that in the Gateways widget there sometimes was a downward facing red arrow next to the WAN interface. When this happened it could only be brought up again by first doing ifconfig re0 up via SSH and then the GUI steps.

So after fiddling around for a while I decided I had enough and removed the driver, rebooted and to my shock and horror the same strange thing occurred. The WAN interface was down after booting like with the 1.100.00 driver. It could be brought up either through the GUI or ifconfig + GUI, but like before it always went down within a matter of minutes.

I dug up some other, more recent unused computer parts including a different NIC. Still a Realtek based one, but who knows.
Now putting it together and I'll do a fresh install of pfSense 2.7.2 to see how that works out.

Attached a few screenshots again. Maybe, hopefully there is something useful to be seen in them.
If you'd like any other info about my latest adventure, feel free to ask.

stephenw10

Are you sure it's actually loading either of those drivers and not just hitting a timeout on the in-kernel driver?

You should see the version reported in the boot log if a module is loaded correctly.

zeroepoch

@stephenw10

Thanks for sharing that command! I was reading online 1.99+ added support for IPv6 checksum offloading, which explains why the suggestion to disable IPv6 (set to None) on those interfaces helps somewhat.

For 1.98.00 I get:

capabilities=381b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,WOL_UCAST,WOL_MCAST,WOL_MAGIC>

zeroepoch

@SpunkThing

If I'm reading your output correctly (based on kern.ipc.nmbjumbo9?) you enabled jumbo frames? If not there is an additional setting you can add to /boot/loader.conf.local to fix a possible hang (if not using jumbo frames). I did the following on my system.

By default, the size of allocated mbufs is enough
to receive the largest Ethernet frame supported
by the card.  If your memory is highly fragmented,
trying to allocate contiguous pages (more than
4096 bytes) may result in driver hangs.
For this reason the value is tunable at boot time,
e.g. if you don't need Jumbo frames you can lower
the memory requirements and avoid this issue with:

hw.re.max_rx_mbuf_sz="2048"

My /boot/loader.conf.local looks like:

if_re_load="YES"
if_re_name="/boot/modules/if_re.ko"
hw.re.max_rx_mbuf_sz="2048"

As @stephenw10 suggested you should check dmesg to see what version was loaded.

# dmesg | grep "re[0-9]: version:"
...
re0: version:1.98.00
re1: version:1.98.00

SpunkThing

Hi @stephenw10

Seems to be the case. This screenshot was taken after installing the driver we're discussing here.

Another indication, if I saw that correctly, is that there is no corresponding module in the /boot/modules directory. That's placed there only when installing an alternative driver such as this one if my observations were correct.

Can't verify now as I'm in the process of installing pfSense on a 'new' system.

SpunkThing

@zeroepoch

That's not something I would deliberately enable. At least not that I remember.
Actually I think that's one of those things I used to turn off on network adapters in Windows in the old days because it caused trouble.

But on the other hand I am the type that messes around with settings often while not completely understanding them. So don't rule it out completely.

Anyway, as I said I'm currently working on a fresh installation of 2.7.2 and that version has always worked nicely for me, except for that one problem with the WAN going down under "heavy" loads. Suppose I'll be looking out for better NIC's if it happens again.

I'll restore a recent-ish configuration backup and leave it at that for the time being.

I'll keep following this though if you decide to dive further into it.
Very curious why both drivers were acting up on my system while they work fine for you.
Could be anything, my messing in settings not being the least likely of causes... Hehehe

zeroepoch

@SpunkThing

That's right. There is no /boot/modules/if_re.ko without either of these packages being installed. It wouldn't print this message if you didn't have the driver I provided loaded, so indeed you're using the same driver as me.

zeroepoch

@stephenw10 any chance we can get realtek-re-kmod198 to be part of the official pfSense repo? FreeBSD made the decision to package these two versions separately because both 1.98 and 1.100 both have different bugs but each work best for different NICs.

stephenw10

I'll see what I can do.