WatchGuard Firebox T70

[chard101]

@stephenw10 @networkBob

hi both.

I have a WatchGuard T70 that I'm looking to butcher to install pfsense.

Before I take it apart and start swinging my cleaver is there anything you need from it to help with getting PoE supported?

I'm back at home at the end of the week.

Thanks.

[stephenw10]

The boot log from the Watchguard OS may contain clues about how the switch is configured, so that would be good to see.

If you can avoid overwriting the original OS so we can refer back to it later that would be good. I believe it should boot from any mSATA device. Or even USB if there is no SATA device present.

Steve

[networkBob]

@chard101 Maybe this information would be useful to you: I pulled the board from the T70 and removed the heatsink. The chip underneath the silver heatsink is a Marvell 88E6176-TFJ2, PAXS390, 4JW, 1631 A1P, TW. I do not have the skillset necessary to load a SOC driver and get the last 5 interfaces to work. The LEDs associated with those interfaces do not illuminate, although the PoE voltage is available and functional on the two PoE ports (6,7) albeit with no data. Thank you.

[networkBob]

@networkBob

[networkBob]

@networkBob

[stephenw10]

Ah, some more info there. We can see the headings on the SW1 DIP switch settings. Either I210-88E6176, the default setting and how they are now set, or SoC-88E6176.
So maybe the switch can be configured via one of the igb ports or from a GPIO line on the SoC dircetly. The bootlog from the original OS might provide a clue there.

The 2-wire eeprom is almost certainly what the switch pulls it's default config from. As we discussed before it is probably configured to come up with all ports disabled as that is the best option from a security point of view. Then the OS sets up the ports and VLANs as required. However without the eeprom connected there's a good chance it comes up as a dumb 5 port switch which would be much more useful here if we can't control it.

I don't have one of those but if I did I would move the SW5 DIP switches to the other position. And see if that allows the switch to come up with ports enabled.
If course I'm guessing here so the risk is all yours!

Steve

[chard101]

@stephenw10 @networkBob

Evening chaps.

Bob, thank you very much for posting those additional photos and Stephen, thanks for your suggestions. Much appreciated.

I've finally got a new 240GB msata flash card and the other bits I needs for the job. So, I'm going to shutdown the firebox, take out the original card, put the new one in and install pfsense. I bought a caddy that I can put the original card in and hopefully pull the boot log from it. That's the plan.

I'm going to start this adventure tomorrow anyway, its getting late here. I will try adjusting the SW5 DIP switches after I've completed the install, or would you suggest trying to extract the boot log first and post it?

Fingers crossed we can get this cracked.

[stephenw10]

The first thing I would do is boot the original OS with the console connected and copy/paste the boot messages to a file from there.

Then install pfSense to the new mSATA device in something else and swap it into the T70, make sure that boots. Check you see the same things @networkBob did.

Then try booting with the SW5 DIP switches in the alternate position. I believe that will disconnect the EEPROM from the switch IC so it cannot load a config when it powers up. It should then default to being a unmanaged switch. With any luck all ports enabled and connected in the same untagged vlan. I have no way of testing that though so ymmv!

Steve

[chard101]

Hi Steve,

I got some time to crack on with this and am now running with pfsense on my T70. I've taken a copy of the origional WG OS bootlog and also the bootlog for pfsense too. Hopefully they are attached to this post and prove useful. I still have the origional WG SSD so I can always hopefully mount it and extract files if needed.

As Bob has reported igb0 to 2 work as expected. I noticed that igb3 comes up with an incomplete MAC address, regardless of how SW5 is set.

igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x1000-0x101f mem 0x80000000-0x800fffff,0x80100000-0x80103fff irq 19 a4
igb3: Using MSIX interrupts with 5 vectors
igb3: Ethernet address: 00:a0:c9:00:00:00

If you have any suggestions to try on getting the other ports working that would be apprecaited.

Thanks.

Rich.

Watchguard_T70_boot_log.txt
After_DIPS.txt

[stephenw10]

Ok at least two interesting things there:

[    3.769850] LED/Reset Button Driver for MB-UP2010W...

That's the M440, it uses the same driver for the LEDs/buttons.

[    9.234620] libphy: Marvell 886176: probed
[    9.239259] wg_dsa_init: mdio found 88E6176
[    9.243942] wg_dsa_init: Rename eth3 -> eth10
[    9.267384] Distributed Switch Architecture driver version 0.1
[    9.273984] mv88e6123_61_65_probe: SW16  88E6176

Confirms what the switch is and how it's attached, via the mdio lines on igb3.

The MAC address you see in pfSense is correct, it's not an error reading it. In the original OS each port is addressed via a VLAN and given a separate MAC at that point.

It's using an Insyde BIOS which is more often found in laptops. Unclear if that's good or bad for us, it's different.

Did you actually power cycle the board between moving the DIP switches? That may be required if the switch remains powered in standby.
I assume the ports still did not show link after changing that? And igb3 still shows as down?

Steve

[chard101]

@stephenw10

Hi Steve,

When I flipped the DIP switches I used the power switch at the back to cut the power before switching it back on. I compared it with a bootlog of before I flipped the switches and there was no change, so I set them back again.

Rich.

[stephenw10]

Ah, the only difference in the bootlog might have been something like:
igb3: link state changed to UP

But only then if you had igb3 assigned and enabled.

If you didn't test the external switch ports after doing that then I would test it again. And run ifconfig -vma at the CLI to see if that shows any change on igb3.

Steve

[networkBob]

@chard101
@stephenw10

Hi Rich,

Were you able to get any further with the igb3 ports? :)

I will try Stephen's suggestion regarding ifconfig -vma.

Kind regards,
nBob

[stephenw10]

I acquired one of these for (probably waaay too much!).

Unfortunately the switch remains stubbornly with all it's ports disabled whatever I have done to it.

They do not seem to come up even for a second at reboot (or complete power cycle) or in the BIOS setup. Or even if you short the CMOS so it doesn't boot at all.

It's interesting. The outside looks very Lanner but the PSU (I have) is from Senao who make their access points.

I was able to confirm he other DIP switches, if you change them from MDIO to SoC the WG OS fails to find the switch and other ports etc.

Steve

[stephenw10]

Some success; but horribly hacky!

[2.4.5-RELEASE][root@t70.stevew.lan]/root: ifconfig -vm igb3
igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=753fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,>
        ether 00:a0:c9:00:00:00
        hwaddr 00:a0:c9:00:00:00
        inet6 fe80::2a0:c9ff:fe00:0%igb3 prefixlen 64 scopeid 0x4
        inet 192.168.70.1 netmask 0xffffff80 broadcast 192.168.70.127
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        supported media:
                media autoselect
                media 1000baseSX
                media 1000baseSX mediaopt full-duplex

There are a number of ways we might try to address the switch ports access. The best way would be to enable actual access to switch via the etherswitch framework. We could then actually configure it with VLANs etc to make separate ports. Most of the components to do that exist but unfortunately there are also some pretty big blockers:
You can't compile etherswitch as a module as far as I can tell so you need a new kernel.
Even with that and after importing the mdio module you need a special igb driver to expose the mdio bus so it can be created as a device and allow the switch to be seen.
The work has almost certainly already been done by Netasq/Stormshield as they have devices very similar to this and a FreeBSD base but I'm not sure if that code was ever made public. I could just be missing something!
The T70 also has the intriguing option to create an mdio bus direct from the SoC without going via the NIC. That may be possible but I think would require code. I can see no reference of anyone doing that in FreeBSD though the etherswitch docs, such as they are, imply it could be attached like that.

We could attempt to change the config in the 2-wire eeprom that the switch chip loads to enable the ports. However it looks like that is only accessible via the switch chip itself or via a clip on type programmer maybe. Also I have no idea how that might be formatted etc. Interestingly it looks like the default position for the DIP switches is 'off', the EEPROM is not connected. And connecting seems to make no difference in either OS as far as I can see. So maybe if doesn't have any config in it.

The final nuclear option became apparent to me whilst chasing something else. I couldn't actually find the datasheet for the 88e6176 so I had to guess from other info but most Marvell chips are similar so... The chip can be configured by holding various pins high or low using external components. This way it can be in a cheap 5 port switch with no CPU or even eeprom required. It has a pin 'NO_CPU'; if that is set low implying there is a CPU it automatically disables all the ports when it is reset as it is at power on. The CPU then configured is later. This is a security measure so the ports are not connected together at boot until the OS is ready. That pin (pin 35) is pulled low by a 5K resistor, if that is disconnected it assumes there is no CPU and does not disable the ports. It would be nice if that was one of the DIP switches or a jumper.... nope.

It is R607 as shown below. It is grounded via the adjacent pad on the unpopulated R614. By cutting the track under the blue line it removes the ground and the chip boots as a regular 5 port unmanaged switch.

It should go without saying that this is not without risk. In fact I would say it is high risk! No one should attempt this! In all likelihood it will brick your, still expensive, T70

I may have simply been lucky.

I will say it does not prevent the WG OS configuring the switch if you go back, or if we later found a way to do it from pfSense. It does make it less secure since all the ports are connected by default. PoE still works.

Steve

[networkBob]

Thank you so much @stephenw10 very grateful for your efforts here.

I attempted this approach and it indeed worked perfectly. Had to use a microscope in order to sever that small connection!

In my use case, each of the 5-port switch interfaces would belong to the same flat network segment. So, while the security aspect of this mod is important to consider, for me it makes no difference. In fact, for me it is simpler this way, as I actually wanted these 5 ports to function as an unmanaged switch. Cheers to you @stephenw10 :)

-Bob

[stephenw10]

Nice. Let me know if you see anything unexpected. Those pins are all used for several things but I don't have the specific datasheet for that chip so I'm unsure exactly what. Probably potentially driving an LED somewhere. The NIC LEDs all seem to work as expected here though.

Steve

[networkBob]

@stephenw10 Each of the "1000" interface activity LEDs on my modified WatchGuard T70 operates as expected. As far as I can tell, the "Status", "Attn", and "Mode" LEDs do not illuminate under any circumstances, which for me is not super important. If, one day, the WGXepc package makes it possible to make use of these WatchGuard T70 LEDs from within pfSense, that would be great but I am not expecting this any time soon. Thanks again @stephenw10 :)

Bob

[bruor]

@networkbob how did you get pfsense installed on the msata drive in the T70? Should I install using another system or can I do boot selection using the serial console and install via USB?

[stephenw10]

There's no way to install to it in the T70 directly as the BIOS is locked down, no way to select a boot device other than the mSATA.
So, yes, install in something else and move it across. If that other thing is not a serial console device then be sure to enable the serial console in the webgui before moving it.

Steve