22.02 Upgrade Experience

[khegel]

Upgrading to the new 22.02 version was very painful. The router I was upgrading was routing for my network at home. The install fails if you specify a host interface in the beginning of the install but have no internet connection which I don't because I'm upgrading my router. After figuring this out I specified no networking which got me through the installer successfully.

All interfaces were on the dataplane at this point so I read the documentation for how to remove NIC for host use https://docs.netgate.com/tnsr/en/latest/setup/return-host-interface.html it says to run

no interface GigabitEthernet6/0/1
no dataplane dpdk dev 0000:06:00.1

but these both return invalid command. Tried just putting the interface in netplan in the host os and rebooting and that fixed it.

Restored my backup.xml to /var/tnsr/running_db and /var/tnsr/startup_db . When I restarted with tnsrctl start the clixon-backend service failed to start. After reading through logs with lots of errors I tried just rebooting and luckily it came up with the new config.

Only problem is now one of the other interfaces is taken by the host os and I can't remove it and get it back in the dataplane. Tried https://docs.netgate.com/tnsr/en/latest/setup/setup-host-interfaces.html but it is already not in the netplan config. To make matters worse when I restart tnsr with tnsrctl restart, all routing breaks and I have to reboot the host for it to start working again.

[jimp]

Adding/removing a NIC in the dataplane or netplan only makes it attach to those when it's possible. If an interface is not in either place then it's sort of in a limbo state but it can be nudged in other ways.

If an interface is not configured in the dataplane and you want it in the host, but it doesn't have a Linux interface name, then the host driver probably isn't attached. Adding it to Netplan and rebooting should fix this.

If an interface was removed from Netplan the driver may still be attached, but doing ip link set <blah> down should stop the host OS from seeing the NIC and then it can be added to the dataplane after.

As for why the interface was grabbed by the dataplane in that last part, it's probably because the driver whitelisting allowed that to happen. We're already planning on changing that behavior for 22.06 so the dataplane will not automatically grab interfaces, they'll need to be manually set in the dataplane.

[khegel]

@jimp I did previously try ip link set eno1 down on the host and then restarted tnsr as well as tried rebooting and neither brought the interface back into the dataplane. Luckily it is an interface that I am not currently using.