SG-1100 Won’t Reboot on Upgrade - no internet access!

Gertjan

@TangoOversway said in SG-1100 Won’t Reboot on Upgrade - no internet access!:

I did run usbboot

The installer is also activating the NICs - so all the NICs become activate in their default state : a switch, without any further firewall rules etc. As long as you stay in the installer, which is a stripped down FreeBSD OS, this situation is valid. That explains what you saw.
Very IHMO of course. I actually hope to be wrong.

stephenw10

If you interrupt the boot again to each the Marvell>> prompt where you previously ran usbrecovery and instead run printenv you will see all the current uboot envs.

They should include:
switch_disable=switch phy_write 1 0 0 0xffff; switch phy_write 2 0 0 0xffff; echo "Switch Ports Disabled";
and
preboot=run switch_disable;

That is run very early during the boot to isolate the switch ports. If you really try hard, like running a ping flood, you might get some packets through but it should not be connected long enough for dhcp.

Gertjan

@stephenw10
Thanks for that info

TangoOversway

@stephenw10

@stephenw10 said in SG-1100 Won’t Reboot on Upgrade - no internet access!:

They should include:
switch_disable=switch phy_write 1 0 0 0xffff; switch phy_write 2 0 0 0xffff; echo "Switch Ports Disabled";
and
preboot=run switch_disable;

That is run very early during the boot to isolate the switch ports. If you really try hard, like running a ping flood, you might get some packets through but it should not be connected long enough for dhcp.

The way I read that, if you run preboot=run switch_disable it will block the connection, but I did have and could see a connection. So shouldn't that be run by default or maybe the installer should do it automatically?

stephenw10

Nope the preboot env is evaluated automatically by uboot before it runs the boot env. You shouldn't need to do anything.

TangoOversway

@stephenw10

@stephenw10 said in SG-1100 Won’t Reboot on Upgrade - no internet access!:

Nope the preboot env is evaluated automatically by uboot before it runs the boot env. You shouldn't need to do anything.

But shouldn't that have prevented my Starlink router on the WAN from seeing everything on the LAN? I thought I got how that happened, since the switch was active, but rules were not. Apparently that's not what's going on.

stephenw10

Yes it should. Which implies the uboot envs may never have been updated. That was not set in the 1100 initially, it was added specifically to address this issue and should be applied automatically at pfSense upgrade.

TangoOversway

@stephenw10

@stephenw10 said in SG-1100 Won’t Reboot on Upgrade - no internet access!:

Yes it should. Which implies the uboot envs may never have been updated. That was not set in the 1100 initially, it was added specifically to address this issue and should be applied automatically at pfSense upgrade.

So I did see the devices, and it wasn't something misleading and this does indicate there is a problem with the 1100 providing pass-through?

I'm not saying that as an, "I was right!" kind of thing - just being sure I am following what is going on. I'm sure this is the kind of thing Netgate would resolve quickly, so I'm not blaming or accusing. Just trying to verify that either I messed up or that it's an issue that's going to be handled.

stephenw10

Yeah I'm suggesting you almost certainly don't have those uboot envs for some reason.

If you need to you can force it to rewrite uboot and update the envs from pfSense like so:

[root@1100-3.stevew.lan]/root: /usr/local/share/u-boot/1100/u-boot-update.sh -f
=> U-Boot is already at the latest version.  Continuing with the installation...
=> Updating the Netgate 1100 U-boot
==> Reading current settings
==> Updating the U-boot image (this may take a few minutes)
64+0 records in
64+0 records out
4194304 bytes transferred in 53.925072 secs (77780 bytes/sec)
==> Updating settings
==> Restoring settings
writing u-boot env(1)... done

TangoOversway

@stephenw10

I have a replacement that was supposed to arrive today - but FedEx delivered it to the wrong address. (We have continual issues with FedEx making proper deliveries.) Once I get the replacement, my only plan for this unit was to keep it stored as an emergency replacement. Since I can keep the OS on the USB stick and always have that, I might try to re-install on the built-in storage.

If I do that, or anything else with this unit, I will not have the LAN and WAN plugged in at the same time until I'm sure the configuration I upload is working.

Does the Netgate symbol by your name mean you're with Netgate? If so, are they looking into this or do I need to file a bug or incident report?

@stephenw10 said in SG-1100 Won’t Reboot on Upgrade - no internet access!:

Yeah I'm suggesting you almost certainly don't have those uboot envs for some reason.

I bought this in 2020, almost exactly 5 years ago. Is that long enough ago that things could be different?

stephenw10

Yes I work at Netgate.

Yes 5 years ago is long enough that it may have shipped without that fix. Running the above command will add the appropriate uboot envs.

The new 1100 should already have them if you just ordered it but since it will be available for testing I'd encourage you to check it to see how to access it etc when not against the clock!

TangoOversway

@stephenw10 said in SG-1100 Won’t Reboot on Upgrade - no internet access!:

The new 1100 should already have them if you just ordered it but since it will be available for testing I'd encourage you to check it to see how to access it etc when not against the clock!

So I can check printenv, as you mentioned and see if it includes:

switch_disable=switch phy_write 1 0 0 0xffff; switch phy_write 2 0 0 0xffff; echo "Switch Ports Disabled";

and

preboot=run switch_disable;

Right?

stephenw10

Yes, run printenv from uboot, the Marvell>> prompt, to see the current configured envs.

TangoOversway

@stephenw10

Another thought - since you've been interested in the issue of the same address space on both sides and since we both thought that's why it could not reach the servers.

First I had it in the usual setup place, one CAT5 going to Starlink, one going to my LAN. That's when I was trying, over and over, to get run usbboot to work. I even tried disabling the LAN interface. I could not get through no matter what. And, as we've discussed, with the 1100 setup that way, Starlink was still seeing my entire LAN.

I figured if there are issues with Starlink getting to my LAN, that had happened. So I connected Starlink to my switch and took my 1100 out.

I keep asking, "What was different between that setup and the one I used where it finally got through to the servers?" Well, first, it tried multiple times and did not get through to them every time. So maybe there's some randomness involved.

The second setup, when it did work, as after Starlink was acting as DNS for my entire LAN, not just the 1100's WAN interface. I took the 1100 upstairs (I know physical position isn't an issue), so I could work on it near my desktop. I hooked up the WAN side to my LAN (which, to me, seems the same as hooking it up to the Starlink router, just with more systems on the same connection). But this time I did NOT hook up anything to the LAN side. As best I can remember, that's the big difference. Also, I found I could get the connection to Netgate servers with the LAN NIC up, but NOT with it down. (And it was never plugged in.)

So I'm wondering if what made the difference in connecting to Netgate was that both LAN and WAN had the same address space, but that, somehow, having nothing on the LAN side to give an address to could have made the difference.

stephenw10

Having two interfaces in the same subnet is a conflict but the result of doing so can be unpredictable. When pfSense is trying to connect to something in that subnet, like it's gateway, there is no unique path to it. Both WAN and LAN NICs are in that subnet so which one it uses can be determined simply by which was last brought up. So having LAN connected or not connected could certainly make a difference.
But the correct way to do this is to set the LAN to a different subnet or set it to 'none' during the install because that removes any routing confusion.

TangoOversway

@stephenw10

@stephenw10 said in SG-1100 Won’t Reboot on Upgrade - no internet access!:

But the correct way to do this is to set the LAN to a different subnet or set it to 'none' during the install because that removes any routing confusion.

Since the user has no control of the LAN subnet at that point, that can be ruled out - unless there's a way, from the Marvell prompt, to do that. Is there?

@stephenw10 said in SG-1100 Won’t Reboot on Upgrade - no internet access!:

Both WAN and LAN NICs are in that subnet so which one it uses can be determined simply by which was last brought up.

At this point, is pfSense using multithreading? It's possible that the times it worked were when there was nothing on the LAN side AND the WAN side came up first. If that's the case, then having the LAN disconnected might not be the important factor. I'm thinking if the LAN comes up first, it'll take 192.168.1.1 for itself, which forces a conflict with a router doing the same. So I think it would only work if the WAN comes up first and gets an address from the DHCP before the LAN comes up and pfSense gives it an address.

If this is the case, I can think of several ways to fix it - some simpler than others. (And I do think this needs attention, since Starlink is growing and isn't just a regional ISP, it's worldwide. It's even used in Antarctica now.)

Since there is the problem with pass-through anyway, I think the docs could be modified to suggest disconnecting the LAN and WAN during setup, and there could also be a prompt for that before the Marvell prompt comes up.

I don't remember if I saw devices from Starlink before I ran usbboot or not, so I don't remember if the NICs are brought up when the Marvell prompt is active or after run usbboot. When they come up changes what would be easy to add to prevent the subnet conflict:

Add a delay so whenever the Marvell prompt is used, the LAN interface is not brought up until after the WAN receives an IP address. This could also be done with a command at the prompt, or with an option the user responds to before the Marvell prompt. (I think that would be the simplest way to handle it.)
Add a command at the Marvell prompt to change the subnet on the LAN (and that could include an option to change it only during setup or long term).
During install, check the WAN interface. If it's in that subnet, then change the LAN subnet and let it revert on reboot (or change it back before rebooting).

stephenw10

The subnet conflict is only an issue once you've booted into the installer where a cut-down pfSense is running. There a conflict might prevent the installer being able to contact the servers to check the available versions and pull in the required pkgs.

At the Marvell>> prompt (uboot) the LAN and OPT ports should be disabled. Whilst uboot can try to connect out it doesn't so any conflict that existed there wouldn't matter. Even if the ports are not disabled.
One you run anything to boot from USB or eMMC pfSense loads and reconfigured the NIC.

TangoOversway

@stephenw10

@stephenw10 said in SG-1100 Won’t Reboot on Upgrade - no internet access!:

The subnet conflict is only an issue once you've booted into the installer where a cut-down pfSense is running. There a conflict might prevent the installer being able to contact the servers to check the available versions and pull in the required pkgs.

Am I missing something, or doesn't that still indicate the installer needs a way to resolve possible conflicting address space?

@stephenw10 said in SG-1100 Won’t Reboot on Upgrade - no internet access!:

One you run anything to boot from USB or eMMC pfSense loads and reconfigured the NIC.

Wouldn't that be a good point where the LAN interface is either delayed so the WAN can get its address and address space first or to allow for the user to specify a different address space?

stephenw10

You can set the LAN subnet in the installer or set it as none:
https://docs.netgate.com/pfsense/en/latest/install/install-walkthrough.html#configure-lan-interface

There's no way setup the LAN from uboot. Potentially a user could enter something there and the imstaller could inherit it but the uboot CLI interface is not intended for that. And it's not necessary because you can just set it n the installer.

TangoOversway

@stephenw10

@stephenw10 said in SG-1100 Won’t Reboot on Upgrade - no internet access!:

There's no way setup the LAN from uboot. Potentially a user could enter something there and the imstaller could inherit it but the uboot CLI interface is not intended for that. And it's not necessary because you can just set it n the installer.

I never saw the option to configure the LAN interface and, from the text in the page above your link, I tried to see how I'd get to that point. It's from the same selection box I ran into multiple times and tried the different choices, but never had the chance to configure the LAN.