Netgate 1100 bridges LAN and WAN port during boot?
-
Hi everyone!
We recently start using Netgate 1100 devices on remote locations to secure a part of the remote network.
Every time the Netgate reboots the devices in the secure LAN receive new IP addresses (192.168.1.xyz) from the public LAN and a public LAN DCHP lease from the modem/router.
When the Netgate is finished booting we need to reboot the devices to have them get the correct IP (10.6.4.xyz) once again.We suspect the Netgate to bridge or being a switch between the public and secure LAN during boot.
We tried to prevent this but with no positive result.Does anyone know how to prevent the 1100 from being a switch between the 2 networks during boot?
Thanks!
-
@bm-1 The 1100 is a switch that uses VLANs to isolate the ports. I don't know much about the boot sequence but what you're describing might be possible before the VLANs are configured (or maybe misconfigured). I can't imagine that is expected though. I would think it would take more than a few milliseconds for the devices to get IPs?
Does it happen if you [back up the config and] reset the config to factory defaults?
-
@bm-1 said in Netgate 1100 bridges LAN and WAN port during boot?:
Does anyone know how to prevent the 1100 from being a switch between the 2 networks during boot?
The recent releases of pfSense include a BIOS update that limit the time it is in this mode to a second or two. Not nearly enough time to do anything and not really enough time to get a DHCP address, either.
Are you running latest on these? If you are and the gap is more than 1-2 seconds I would suggest requesting the latest firmware at https://go.netgate.com/ and reinstall to be certain you limit the time it is in that mode.
-
Thank you for the answers. This is much appreciated.
We see this behavior happening on all 1100 devices. My guess is it is designed by default seeing the device uses a Marvel switch chip for all interfaces instead of real network interface chips.
It sounds plausible that interfaces are isolated by vlans on the switch chip. And initializing the vlans takes time after the boot sequence. Therefor until the boot has finished the firewall acts as a regular switch....
I will see if I can reproduce this.Also I will request a BIOS update and see if reducing the time it takes reduces the impact of this behavior.
-
The update disables the switch ports in uboot and they are then re-enabled when the OS boots and configures the switch. So the only time they are connected is when the switch chip is powered up and before uboot runs which is a very short time span.
Steve
-
Ok, we are done testing.
The test went as follows. A device in the secure LAN got a IP address from the public LAN (192.168.1.xyz). And we setup a ping from the modem/router to that device to see if was reachable through the Netgate. If not, the Netgate secures the LAN. If so, the Netgate allows traffic from public to secure LAN, which is a breach.
At the start it is unreachable. During the (cold) reboot it starts pinging. When BSD loads the etherswitch driveris the destination becomes unreachable once again.From 172.24.254.20 icmp_seq=298 Destination Host Unreachable
64 bytes from 172.24.254.200: icmp_seq=299 ttl=64 time=1002 ms
64 bytes from 172.24.254.200: icmp_seq=300 ttl=64 time=2.50 ms
64 bytes from 172.24.254.200: icmp_seq=301 ttl=64 time=0.358 ms
64 bytes from 172.24.254.200: icmp_seq=303 ttl=64 time=0.290 ms
64 bytes from 172.24.254.200: icmp_seq=304 ttl=64 time=0.363 ms
64 bytes from 172.24.254.200: icmp_seq=305 ttl=64 time=0.400 ms
64 bytes from 172.24.254.200: icmp_seq=306 ttl=64 time=0.414 ms
64 bytes from 172.24.254.200: icmp_seq=307 ttl=64 time=0.628 ms
64 bytes from 172.24.254.200: icmp_seq=308 ttl=64 time=0.296 ms
64 bytes from 172.24.254.200: icmp_seq=309 ttl=64 time=0.341 ms
64 bytes from 172.24.254.200: icmp_seq=310 ttl=64 time=0.395 ms
64 bytes from 172.24.254.200: icmp_seq=311 ttl=64 time=0.656 ms
64 bytes from 172.24.254.200: icmp_seq=312 ttl=64 time=0.420 ms
64 bytes from 172.24.254.200: icmp_seq=313 ttl=64 time=0.400 ms
64 bytes from 172.24.254.200: icmp_seq=314 ttl=64 time=0.492 ms
64 bytes from 172.24.254.200: icmp_seq=315 ttl=64 time=0.431 ms
64 bytes from 172.24.254.200: icmp_seq=316 ttl=64 time=0.422 ms
64 bytes from 172.24.254.200: icmp_seq=317 ttl=64 time=0.396 ms
From 172.24.254.20 icmp_seq=342 Destination Host Unreachable
From 172.24.254.20 icmp_seq=343 Destination Host Unreachable
From 172.24.254.20 icmp_seq=344 Destination Host UnreachableBoth with the old and new firmware the switch ports are open during a cold boot. The time they allow traffic from public LAN to secure LAN is between 15-20 seconds...
After the boot is complete and the configuration is loaded the ports are shut or it is no longer to pass traffic.
It is save to say that during a cold reboot by default the switch ports are open and there is no security.
With a power failure and all devices are powering up 15-20 seconds is enough to get a DCHP lease.
In my opinion the switch ports should be down by default and after booting and the configuration is loaded they should be open or allow traffic.
For a security device this is bit off.During a warm boot or reboot the switch holds it's configuration and there are no issues.
I have also taken a look at the vlan separation suggestion. I am not sure want to change and how how to fix the current issue with the current hardware.
Below you find the switch config.etherswitchcfg
etherswitch0: VLAN mode: DOT1Q
port0:
pvid: 1
state=8<FORWARDING>
flags=1<CPUPORT>
media: Ethernet 1000baseT <full-duplex>
status: active
port1:
pvid: 4092
state=8<FORWARDING>
flags=0<>
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
port2:
pvid: 4091
state=8<FORWARDING>
flags=0<>
media: Ethernet autoselect (none)
status: no carrier
port3:
pvid: 4090
state=8<FORWARDING>
flags=0<>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vlangroup0:
vlan: 1
members 0
vlangroup1:
vlan: 4090
members 0t,3
vlangroup2:
vlan: 4091
members 0t,2
vlangroup3:
vlan: 4092
members 0t,1This is not a issue on one device we have several. Also it was tested on and is repreproducible on several devices.
-
@stephenw10 Only during warm boots. During cold reboots the security device acts as a regular switch. And 15-20 seconds is not a very short time span.
I hope my information helps.
-
If you're seeing 15-20s then it doesn't have the new uboot values. They should be added at upgrade.
You can check manually if you runprintenvat the uboot prompt. You should see the 'switch_disable' entry.Let me test that....
-
Workaround:
If you use a managed Switch behind, use Tagged only VLANs between.
There is no tagged Frame in the Switch boot and brided time. -
You can also check this from the pfSense command line like:
[23.01-BETA][root@1100.stevew.lan]/root: u-boot-env-read /tmp/envs [23.01-BETA][root@1100.stevew.lan]/root: cat /tmp/envs �~Narch=armbaudrate=115200board=mvebu_armada-37xxboard_name=mvebu_armada-37xxbootargs=root=PARTUUID=15896539-02 rw rootwait console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000bootdelay=2console=console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000cpu=armv8eth1addr=00:51:82:11:22:01eth2addr=00:51:82:11:22:02eth3addr=00:51:82:11:22:03ethact=neta@30000ethaddr=f0:ad:4e:08:6b:dfethprime=eth0extra_params=pci=pcie_bus_safefdt_high=0xfffffffffffffffffdt_name=fdt.dtbfdtcontroladdr=3f62d490fileaddr=6f00000filesize=2c88gatewayip=10.4.50.254get_images=tftpboot $kernel_addr_r $image_name; tftpboot $fdt_addr_r $fdt_name; run get_ramfsget_ramfs=if test "${ramfs_name}" != "-"; then setenv ramdisk_addr_r 0x8000000; tftpboot $ramdisk_addr_r $ramfs_name; else setenv ramdisk_addr_r -;fihostname=marvellimage_name=Imageinitrd_addr=0xa00000initrd_size=0x2000000ipaddr=0.0.0.0memtest86=usb reset; load usb 0:1 0x7000000 efi/boot/bootaa64.efi; load usb 0:1 0x6f00000 armada-3720-netgate-1100.dtb; fdt addr 0x6f00000; bootefi 0x7000000 0x6f00000;netdev=eth0netmask=255.255.255.0openwrt=usb start; setenv bootargs "root=/dev/sda2 rw rootwait ${console}"; load usb 0:1 ${fdt_addr_r} armada-3720-espressobin-v7-emmc.dtb; load usb 0:1 ${kernel_addr} Image; booti ${kernel_addr} - ${fdt_addr_r}ramdisk_addr_r=0x8000000ramfs_name=-root=root=/dev/nfs rwrootpath=/srv/nfs/serial=NTG1846000004serverip=0.0.0.0set_bootargs=setenv bootargs $console $root ip=$ipaddr:$serverip:$gatewayip:$netmask:$hostname:$netdev:none nfsroot=$serverip:$rootpath,tcp,v3 $extra_params $cpuidlesoc=mvebustderr=serial@12000stdin=serial@12000stdout=serial@12000vendor=Marvellbootcmd=run emmcboot; run scsiboot; run net;boot_file=efi/boot/bootaa64.efidtb_name=armada-3720-netgate-1100.dtbdtb_name_old=armada-3720-sg1100.dtbemmcboot=mmc rescan; setenv loaddev mmc; setenv loadunit 1; if fatls $loaddev $loadunit:2 /; then run pfsenseboot; fi;fdt_addr_r=0x6f00000kernel_addr=0x7000000kernel_addr_r=0x7000000loadaddr=0x7000000net=dhcp; tftp $fdt_addr_r $dtb_name; tftp $kernel_addr_r loader.efi; fdt addr $fdt_addr_r; fdt set / sn $serial; fdt set /soc/internal-regs/ethernet local-mac-address $ethaddr; bootefi $kernel_addr_r $fdt_addr_r;pfsenseboot=if fatls $loaddev $loadunit:2 /armada-3720-netgate-1100.dtb ; then setenv dtb $dtb_name; else setenv dtb $dtb_name_old; fi; load $loaddev $loadunit:1 $kernel_addr_r $boot_file; load $loaddev $loadunit:2 $fdt_addr_r $dtb; fdt addr $fdt_addr_r; fdt set / sn $serial; fdt set /soc/internal-regs/ethernet local-mac-address $ethaddr; bootefi $kernel_addr_r $fdt_addr_r;preboot=run switch_disable;recovery=run usbboot;scsiboot=scsi reset; setenv loaddev scsi; setenv loadunit 0; if fatls $loaddev $loadunit:2 /; then run pfsenseboot; fi;switch_disable=switch phy_write 1 0 0 0xffff; switch phy_write 2 0 0 0xffff; echo "Switch Ports Disabled";usbboot=usb reset; setenv loaddev usb; setenv loadunit 0; if fatls $loaddev $loadunit:2 /; then run pfsenseboot; fi;usbrecovery=mmc dev 1; mmc erase 0 400000; run usbboot;You are looking for the envs:
preboot=run switch_disable switch_disable=switch phy_write 1 0 0 0xffff; switch phy_write 2 0 0 0xffff; echo "Switch Ports Disabled"Steve
-
@stephenw10 said in Netgate 1100 bridges LAN and WAN port during boot?:
u-boot-env-read /tmp/envs
cat /tmp/envsWait, what?!
Learn something new every day...
-
@stephenw10 Hi I have checked but indeed its missing the switch_disable env. How can I add these to u-boot environment?
-
There are several ways you can add them. They are added automatically when uboot is updated but if you are already running the latest uboot version that would not have happened. The easiest way is to force uboot to be rewritten which will then also run the env updates:
[23.01-BETA][root@1100.stevew.lan]/root: /usr/local/share/u-boot/1100/u-boot-update.sh -f => U-Boot is already at the latest version. Continuing with the installation... => Updating the Netgate 1100 U-boot ==> Reading current settings ==> Updating the U-boot image (this may take a few minutes) 64+0 records in 64+0 records out 4194304 bytes transferred in 55.425440 secs (75675 bytes/sec) ==> Updating settings ==> Restoring settings writing u-boot env(1)... doneAfter which you should see those envs.
You could also manually add them at the uboot prompt or running the uboot-env read/update/write commands if necessary.
Steve
-
@stephenw10 Thanks!
With your rewrite suggestion we were able to modify the uboot.
We are running the latest version and it did happen. It is fixed now.
When we run our test procedure we see a switch disable message. And no pings are passing through the device so it works as it should.Many thanks for the helping hand!
-
Excellent, good to hear!
