Upgrade from 23.09.1 to 24.03 Completes Successfully, But NIC Will No Longer Pass Traffic
-
Ah, well some progress at least. And always good to prove a theory incorrect. New firmware doesn't hurt also good to know.
I'd guess bandwidthd or, more likely, Suricata if it's running in in-line mode which uses the NIC in netmap mode and can break everything!
-
@stephenw10 said in Upgrade from 23.09.1 to 24.03 Completes Successfully, But NIC Will No Longer Pass Traffic:
I'd guess bandwidthd or, more likely, Suricata if it's running in in-line mode which uses the NIC in netmap mode and can break everything!
Good to know. My suricata is IDS only, so it shouldn't be mucking with the interface. Tonight I'm hoping to go through this again, reload my config (hoping that it also 'fails' to load the packages), and then I will install one and reboot, rinse and repeat until I find the cranky package.
-
@stephenw10 - Ok... So, I'm at a loss. It HAS to be something with my config, but it's somewhat complex, and I really don't want to create everything by hand.
I reset 24.03 back to factory defaults, configured WAN and LAN, set the IPs, rebooted (working). Rebooted again (working)...
I installed the acme, zabbix, and Wireguard packages... Really low impact, right, and should be completely unrelated to the LAN interface. Install works, reboot... Dead. Reboot. Still dead.
Back to 23.09 I go...
I'm not above getting another NIC with another chipset entirely to try it, BUT this SHOULD work without an issue, and swapping out a NIC is going to kill my Netgate ID, which will kill my paid plus subscription, and to be honest, that whole implementation seems flakey to me, so I don't want to introduce yet another wrinkle.
Kinda at a loss... Really want to upgrade, but I now have NO idea what it could be, without manually recreating my config (consisting of almost a dozen interfaces, 6 VLANs, countless rules, and a ton of Suricata & pfBlocker-NG configurations). That would take a SIGNIFICANT amount of time to re-create and the risk of screwing something up in the details is REALLY a possibility.
Thoughts? I mean... This should work. So what else can I do?
-
Hmm well of those 3 I'd have to suspect Wireguard. That can at least add an interface. Zabbix and ACME really could not prevent traffic.
-
@stephenw10 - I will try just wire guard and see what happens. It worked on one of my previous attempts and reboots. So I figured it was safe.
It still leaves me in a pretty crappy situation. I can't swap hardware, because I lose my Plus (different MAC), I can't actually upgrade because, well, it doesn't work.
Anyone else there at Netgate have any ideas? This one happens to be my main router in my home lab, so it's kinda the lynchpin in everything. I DEFINITELY need wire guard to work.
I guess I can wait until there's another release, but that leaves me in 23.09.1 for a long time without any security enhancements.
I really think it might be something latent in my config. Is there anyone at Netgate who would take a look at the XML? Perhaps there's something I'm not seeing? Maybe you guys have better debug tools?
I'll try to do more testing tomorrow...
-
It does seem like something in you config I agree. If it's not some package putting the NIC in an odd mode it could be a system tunable you have added.
Are you able to upload the config for us to review here: https://nc.netgate.com/nextcloud/s/fcTw2Dy3FKD7bCK
Steve
-
@stephenw10 - Config uploaded.
Note, specifically about tunables. I've never actually added any, and there are likely some in there from considerably different hardware, IF, that stuff carries forward. I'm not sure what should be there from default, or how to "safely" reset them back to "default", but I'd definitely be willing to try that too.
-
@stephenw10 - HOLY CRAP I think I figured it out. Performing more testing. Will know in a few more reboots once I get the rest of the packages installed.
It looks like it WAS wireguard in a, "this should never have worked" type of scenario...
Will edit post shortly...
Edit: YES!
The issue was with a WireGuard Gateway Monitor IP. It just so happens that the LAN IP of my router and the LAN IP of the router on the other side of the WG Gateway are flipped, (think 192.168.1.1 and 192.1.168.1). Apparently, 23.09.1 didn't care that I had the LAN IP entered in there and was happy to just status something that was always up... 24.03 was none-too-happy with that config, however, and broke the LAN interface because of it.
Troubleshooting:
I enabled access to the Webconfigurator through another interface so that I could actually see what was going on, and noticed that there was an issue with that ONE WireGuard gateway and when I looked why, I saw it immediately.Problem. SOLVED. Awesome news on a Friday night, and dare I say it, this one was kinda fun!
-
Wow nice catch! Interesting that worked in 23.09.1. Hmm.
-
@stephenw10 - Right?
Thanks for all the help!!
