Upgrade from 23.09.1 to 24.03 Completes Successfully, But NIC Will No Longer Pass Traffic

stephenw10

Hmm, I agree I wouldn't expect the actual link media reported to matter as long as the speed is correct.

jsylvia007

H'okay... Got a chance to reboot and test. Here is the screenshot (the only way I could get the information because I didn't have SSH and had to use IPMI).

Screenshot 2024-04-30 202837.png

Interesting here, BUT, the inet line is completely missing... Yet the main page says that the IP is assigned. I also selected the command line option to assign an IP to the interface to re-assign the LAN IP, and it made no difference. The output of the command was identical.

I then manually tried to add an IP to the interface, and it didn't like that either:

Screenshot 2024-04-30 204000.png

Took 2 reboots back into 23.09.1 to get the interface back. Interestingly enough, that FIRST reboot back, the output of the command is identical to what is shown in the above screenshot...

Weird.

Another thing I noticed, somehow my ntop-ng got hosed and even in the old boot environment it won't work. This is a different issue, so I will wait to fix that one until we get an idea on what might be going on here.

stephenw10

Hmm, curious. I assume specifying to subnet using CIDR notation also fails?

What firmware version does that NIC have? It could be the newer driver trying to use some API update perhaps.

sysctl dev.ixl.0.fw_version

jsylvia007

@stephenw10 -- Interesting point... I never upgraded the firmware on this NIC. I've had real bad luck with NIC firmwares on some Intel Atom chips, so I avoided it.

I've never updated firmware on BSD... Guess I could crack the case open and see where the NIC came from to get updated firmware.

Output from the command:

dev.ixl.0.fw_version: fw 5.50.47059 api 1.5 nvm 5.51 etid 80002bca oem 1.262.0

Edit: Apparently I got the card on eBay... This is the card:

Intel XXV710-DA2 25GbE Dual-Port Ethernet Network Adapter XXV710DA2BLK

stephenw10

Oh that is an old firmware version. Importantly an old API version. I would try upgrading it. You'll probably have to do that from Windows or Linux though.

jsylvia007

@stephenw10 - H'okay. Believe it or not they have a BSD & EFI version for the latest firmware. I'm going to try the EFI version tonight... I will let you know how it goes.

jsylvia007

@stephenw10 - Ok... So... Bear with me... I just spent about 6 straight hours troubleshooting and didn't really accomplish much.

Upgraded the firmware on the card, that was a breeze. Took about 15 min through UEFI. Here is the new firmware information:

dev.ixl.0.fw_version: fw 9.140.76856 api 1.15 nvm 9.40 etid 8000ed12 oem 1.269.0

Appears to be much newer, and BONUS, it STILL works with 23.09.1.

Long story short, SAME exact symptoms with 24.03.

So, I decided to factory reset the configuration. After the reboot, I manually reassigned the interfaces to be the correct ones for at least my LAN and WAN, manually set the IP address for the LAN and.... NOTHING. I performed a reboot just for giggles, and, wouldn't you know it, it WORKED. And it was repeatable. 3 reboots later and I was confident that it was 'stable'.

I took my backup config (downloaded from the working 23.09.1), loaded it on the GUI, and it... kinda worked after a reboot. The LAN interface came back, and all the other settings came back, but I got an effort for EVERY package that basically said, "Package ABCDEF does not exist in current Netgate pfSense Plus version and it has been removed.", for all 22 of my packages.

I rebooted a couple times, and it again seemed 'stable' on the LAN interface.

I started adding my packages, they all came up and worked no problem... Then I rebooted again... LAN interface was dead again. Reboot 3 more times... still dead. Reboot into the 23.09.1 boot environment, everything is hunky dory again.

So... Maybe it's related to SOMETHING in my configuration related to the packages I have installed?

Here is the list of all 22 packages that I use:

mailreport
iperf
nmap
mtr-nox11
openvpn-client-export
acme
bandwidthd
Cron
Status_Traffic_Totals
syslog-ng
Service_Watchdog
System_Patches
avahi-daemon
arpwatch
pimd
pfBlockerNG
zabbix-agent64
nut
WireGuard
suricata
ntopng

If I had to make a guess, I would suspect that MAYBE it has something to do with either bandwidthd, Status_Traffic_Control, avahi-daemon, or pimd because I believe those actually have the ability to muck with the interfaces at a more substantial level than the rest of the packages. I'm reasonably certain that I never actually got media casting across VLANs to work successfully. so I think I can ditch avahi-daemon and pimd. Might be able to nix the others too, but I'm not actually sure if that's really needed.

I'm willing to share my config privately with support if you think there's something in there that might help.

Back on 23.09.1 for now...

stephenw10

Ah, well some progress at least. And always good to prove a theory incorrect. New firmware doesn't hurt also good to know.

I'd guess bandwidthd or, more likely, Suricata if it's running in in-line mode which uses the NIC in netmap mode and can break everything!

jsylvia007

@stephenw10 said in Upgrade from 23.09.1 to 24.03 Completes Successfully, But NIC Will No Longer Pass Traffic:

I'd guess bandwidthd or, more likely, Suricata if it's running in in-line mode which uses the NIC in netmap mode and can break everything!

Good to know. My suricata is IDS only, so it shouldn't be mucking with the interface. Tonight I'm hoping to go through this again, reload my config (hoping that it also 'fails' to load the packages), and then I will install one and reboot, rinse and repeat until I find the cranky package.

jsylvia007

@stephenw10 - Ok... So, I'm at a loss. It HAS to be something with my config, but it's somewhat complex, and I really don't want to create everything by hand.

I reset 24.03 back to factory defaults, configured WAN and LAN, set the IPs, rebooted (working). Rebooted again (working)...

I installed the acme, zabbix, and Wireguard packages... Really low impact, right, and should be completely unrelated to the LAN interface. Install works, reboot... Dead. Reboot. Still dead.

Back to 23.09 I go...

I'm not above getting another NIC with another chipset entirely to try it, BUT this SHOULD work without an issue, and swapping out a NIC is going to kill my Netgate ID, which will kill my paid plus subscription, and to be honest, that whole implementation seems flakey to me, so I don't want to introduce yet another wrinkle.

Kinda at a loss... Really want to upgrade, but I now have NO idea what it could be, without manually recreating my config (consisting of almost a dozen interfaces, 6 VLANs, countless rules, and a ton of Suricata & pfBlocker-NG configurations). That would take a SIGNIFICANT amount of time to re-create and the risk of screwing something up in the details is REALLY a possibility.

Thoughts? I mean... This should work. So what else can I do?

stephenw10

Hmm well of those 3 I'd have to suspect Wireguard. That can at least add an interface. Zabbix and ACME really could not prevent traffic.

jsylvia007

@stephenw10 - I will try just wire guard and see what happens. It worked on one of my previous attempts and reboots. So I figured it was safe.

It still leaves me in a pretty crappy situation. I can't swap hardware, because I lose my Plus (different MAC), I can't actually upgrade because, well, it doesn't work.

Anyone else there at Netgate have any ideas? This one happens to be my main router in my home lab, so it's kinda the lynchpin in everything. I DEFINITELY need wire guard to work.

I guess I can wait until there's another release, but that leaves me in 23.09.1 for a long time without any security enhancements.

I really think it might be something latent in my config. Is there anyone at Netgate who would take a look at the XML? Perhaps there's something I'm not seeing? Maybe you guys have better debug tools?

I'll try to do more testing tomorrow...

stephenw10

It does seem like something in you config I agree. If it's not some package putting the NIC in an odd mode it could be a system tunable you have added.

Are you able to upload the config for us to review here: https://nc.netgate.com/nextcloud/s/fcTw2Dy3FKD7bCK

Steve

jsylvia007

@stephenw10 - Config uploaded.

Note, specifically about tunables. I've never actually added any, and there are likely some in there from considerably different hardware, IF, that stuff carries forward. I'm not sure what should be there from default, or how to "safely" reset them back to "default", but I'd definitely be willing to try that too.

jsylvia007

@stephenw10 - HOLY CRAP I think I figured it out. Performing more testing. Will know in a few more reboots once I get the rest of the packages installed.

It looks like it WAS wireguard in a, "this should never have worked" type of scenario...

Will edit post shortly...

Edit: YES!

The issue was with a WireGuard Gateway Monitor IP. It just so happens that the LAN IP of my router and the LAN IP of the router on the other side of the WG Gateway are flipped, (think 192.168.1.1 and 192.1.168.1). Apparently, 23.09.1 didn't care that I had the LAN IP entered in there and was happy to just status something that was always up... 24.03 was none-too-happy with that config, however, and broke the LAN interface because of it.

Troubleshooting:
I enabled access to the Webconfigurator through another interface so that I could actually see what was going on, and noticed that there was an issue with that ONE WireGuard gateway and when I looked why, I saw it immediately.

Problem. SOLVED. Awesome news on a Friday night, and dare I say it, this one was kinda fun!

stephenw10

Wow nice catch! Interesting that worked in 23.09.1. Hmm.

jsylvia007

@stephenw10 - Right?

Thanks for all the help!!