@stephenw10 thank you, didn't know about bt, patient0's tip about disabling the hardware via the loader.conf.local file did the trick. I was able to upgrade, it went super fast.
For whatever reason the pkg repo was not available when it tried to fetch them during the config install. Usually that's because the WAN was reconfigured. In that situation you may need to restore a config twice. However since it was only those two packages and you have manually reinstalled them it's not a big issue.
The LAN settings for the installer are used by the installer, but it's not fully implemented yet, and optional. For example, there could be a web-based installer, or you can SSH in and perform some pre-install configuration if needed.
As for the WAN/LAN settings being exported to the fresh install, that's in progress and will be in the next installer release, it's already being tested internally.
It was weird the only thing I have changed in months was adding DHCPv6 which is why I set it back to none. I guess it could have just been unrelated and fixed when it rebooted but I thought i should mention it as thats what I did.