• Hi everyone,

    I installed pfSense on a new WatchGuard T70, which is a tabletop form factor with 8 network interfaces, two of which are PoE.

    The first 3 interfaces are recognized by pfSense as igb0-2 and operate as expected. The last 5 interfaces are recognized by pfSense only as igb3 but none of these 5 interfaces work (the PoE voltage is present).

    Examining the PCB, the first 3 interfaces each have a dedicated Intel chip. The last 5 interfaces appear to be collectively driven by a one large chip with a heatsink. So, I'm thinking maybe a BSD driver is needed in order for the last 5 interfaces to work. At this point, I installed the developer version of pfSense thinking that this might automatically recognize the need for a driver and install the necessary driver, but nothing changed. I have no idea how to force pfSense to install the necessary driver.

    Maybe someone has some insight and can help me?

    Thank you,
    Bob

  • Netgate Administrator

    Ah, I have wanted to get a look inside one of those for a while. Waaaay outside my curiosity price range though. ๐Ÿ˜‰

    You have photos?

    The other 5 ports are almost certainly connected via a switch under the heatsink.

    Do you see link LEDs connecting to those ports?

    The switch is probably not configured by default and so not passing any traffic. It's far more secure to have the switch come up with it's ports disabled so that would be logical.
    However if you do see link it's possible the switch is configured with VLANs by default. Does igb3 show as linked and up?
    If so try running a packet capture on there with something on the external ports generating some traffic. You might see some tagged traffic come in. ๐Ÿคž

    Otherwise you will need to configure the switch and that will depend on what the switch is. If you cannot remove the heatsink you might look at the Watchguard OS boot log for clues.

    Steve


  • Watchguard T70 internal view 1.jpg Watchguard T70 internal view 2.jpg

  • Netgate Administrator

    Ooo, fun!

    Looks like there's decent labelling on things, including those dip switches. Not sure I can make it out but it looks like SW1 and SW5 are set in their 'default' positions? Slightly unclear if that's '2-wire eprom' or select between 2-wire or eprom.

    Were you able to get any sort of boot log?

    Did you install to mSATA in something else and move it across or boot from USB?

    Steve


  • The mSATA is the boot disk (I had removed it temporarily when I took the photos). It boots quite fast from mSATA. Unfortunately, I made the mistake of overwriting the original mSATA that contained the Watchguard OS, so I destroyed the opportunity to observe the boot log.

    As a high-speed 3-interface pfSense box, this thing works very well. But I really would like to get the other ports working. I will do some more testing and report back, along with more photos.

    nBob

  • Netgate Administrator

    Speculation time: I would guess that setting SW5 to it's alternate position allows the switch to pull it's config from the eeprom, rather than be programmed by the OS. There may not be any config in the there to pull of course.... And it might require the other DIP switches to be set also....
    The fact they are labelled MDC/MDIO implies they may allow/disallow programming the switch that way which is commonly how it's done on small switch chips.

    If the switch ports are all down by default, including the internal one, that may well be what's happening. It just has no config so defaults to disabling the ports.

    Steve


  • @stephenw10 @networkBob

    hi both.

    I have a WatchGuard T70 that I'm looking to butcher to install pfsense.

    Before I take it apart and start swinging my cleaver is there anything you need from it to help with getting PoE supported?

    I'm back at home at the end of the week.

    Thanks.

  • Netgate Administrator

    The boot log from the Watchguard OS may contain clues about how the switch is configured, so that would be good to see.

    If you can avoid overwriting the original OS so we can refer back to it later that would be good. I believe it should boot from any mSATA device. Or even USB if there is no SATA device present.

    Steve


  • @chard101 Maybe this information would be useful to you: I pulled the board from the T70 and removed the heatsink. The chip underneath the silver heatsink is a Marvell 88E6176-TFJ2, PAXS390, 4JW, 1631 A1P, TW. I do not have the skillset necessary to load a SOC driver and get the last 5 interfaces to work. The LEDs associated with those interfaces do not illuminate, although the PoE voltage is available and functional on the two PoE ports (6,7) albeit with no data. Thank you.



  • Netgate Administrator

    Ah, some more info there. We can see the headings on the SW1 DIP switch settings. Either I210-88E6176, the default setting and how they are now set, or SoC-88E6176.
    So maybe the switch can be configured via one of the igb ports or from a GPIO line on the SoC dircetly. The bootlog from the original OS might provide a clue there.

    The 2-wire eeprom is almost certainly what the switch pulls it's default config from. As we discussed before it is probably configured to come up with all ports disabled as that is the best option from a security point of view. Then the OS sets up the ports and VLANs as required. However without the eeprom connected there's a good chance it comes up as a dumb 5 port switch which would be much more useful here if we can't control it.

    I don't have one of those but if I did I would move the SW5 DIP switches to the other position. And see if that allows the switch to come up with ports enabled.
    If course I'm guessing here so the risk is all yours! ๐Ÿ˜‰

    Steve


  • @stephenw10 @networkBob

    Evening chaps.

    Bob, thank you very much for posting those additional photos and Stephen, thanks for your suggestions. Much appreciated.

    I've finally got a new 240GB msata flash card and the other bits I needs for the job. So, I'm going to shutdown the firebox, take out the original card, put the new one in and install pfsense. I bought a caddy that I can put the original card in and hopefully pull the boot log from it. That's the plan.

    I'm going to start this adventure tomorrow anyway, its getting late here. I will try adjusting the SW5 DIP switches after I've completed the install, or would you suggest trying to extract the boot log first and post it?

    Fingers crossed we can get this cracked.

  • Netgate Administrator

    The first thing I would do is boot the original OS with the console connected and copy/paste the boot messages to a file from there.

    Then install pfSense to the new mSATA device in something else and swap it into the T70, make sure that boots. Check you see the same things @networkBob did.

    Then try booting with the SW5 DIP switches in the alternate position. I believe that will disconnect the EEPROM from the switch IC so it cannot load a config when it powers up. It should then default to being a unmanaged switch. With any luck all ports enabled and connected in the same untagged vlan. I have no way of testing that though so ymmv!

    Steve


  • Hi Steve,

    I got some time to crack on with this and am now running with pfsense on my T70. I've taken a copy of the origional WG OS bootlog and also the bootlog for pfsense too. Hopefully they are attached to this post and prove useful. I still have the origional WG SSD so I can always hopefully mount it and extract files if needed.

    As Bob has reported igb0 to 2 work as expected. I noticed that igb3 comes up with an incomplete MAC address, regardless of how SW5 is set.

    igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x1000-0x101f mem 0x80000000-0x800fffff,0x80100000-0x80103fff irq 19 a4
    igb3: Using MSIX interrupts with 5 vectors
    igb3: Ethernet address: 00:a0:c9:00:00:00

    If you have any suggestions to try on getting the other ports working that would be apprecaited.

    Thanks.

    Rich.

    Watchguard_T70_boot_log.txt
    After_DIPS.txt

  • Netgate Administrator

    Ok at least two interesting things there:

    [    3.769850] LED/Reset Button Driver for MB-UP2010W...
    

    That's the M440, it uses the same driver for the LEDs/buttons.

    [    9.234620] libphy: Marvell 886176: probed
    [    9.239259] wg_dsa_init: mdio found 88E6176
    [    9.243942] wg_dsa_init: Rename eth3 -> eth10
    [    9.267384] Distributed Switch Architecture driver version 0.1
    [    9.273984] mv88e6123_61_65_probe: SW16  88E6176
    

    Confirms what the switch is and how it's attached, via the mdio lines on igb3.

    The MAC address you see in pfSense is correct, it's not an error reading it. In the original OS each port is addressed via a VLAN and given a separate MAC at that point.

    It's using an Insyde BIOS which is more often found in laptops. Unclear if that's good or bad for us, it's different.

    Did you actually power cycle the board between moving the DIP switches? That may be required if the switch remains powered in standby.
    I assume the ports still did not show link after changing that? And igb3 still shows as down?

    Steve


  • @stephenw10

    Hi Steve,

    When I flipped the DIP switches I used the power switch at the back to cut the power before switching it back on. I compared it with a bootlog of before I flipped the switches and there was no change, so I set them back again.

    Rich.

  • Netgate Administrator

    Ah, the only difference in the bootlog might have been something like:
    igb3: link state changed to UP

    But only then if you had igb3 assigned and enabled.

    If you didn't test the external switch ports after doing that then I would test it again. And run ifconfig -vma at the CLI to see if that shows any change on igb3.

    Steve


  • @chard101
    @stephenw10

    Hi Rich,

    Were you able to get any further with the igb3 ports? :)

    I will try Stephen's suggestion regarding ifconfig -vma.

    Kind regards,
    nBob

  • Netgate Administrator

    I acquired one of these for (probably waaay too much!).

    Unfortunately the switch remains stubbornly with all it's ports disabled whatever I have done to it.

    They do not seem to come up even for a second at reboot (or complete power cycle) or in the BIOS setup. Or even if you short the CMOS so it doesn't boot at all.

    It's interesting. The outside looks very Lanner but the PSU (I have) is from Senao who make their access points.

    I was able to confirm he other DIP switches, if you change them from MDIO to SoC the WG OS fails to find the switch and other ports etc.

    Steve

  • Netgate Administrator

    Some success; but horribly hacky!

    [2.4.5-RELEASE][root@t70.stevew.lan]/root: ifconfig -vm igb3
    igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
            options=6400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
            capabilities=753fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,>
            ether 00:a0:c9:00:00:00
            hwaddr 00:a0:c9:00:00:00
            inet6 fe80::2a0:c9ff:fe00:0%igb3 prefixlen 64 scopeid 0x4
            inet 192.168.70.1 netmask 0xffffff80 broadcast 192.168.70.127
            nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
            media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
            supported media:
                    media autoselect
                    media 1000baseSX
                    media 1000baseSX mediaopt full-duplex
    

    There are a number of ways we might try to address the switch ports access. The best way would be to enable actual access to switch via the etherswitch framework. We could then actually configure it with VLANs etc to make separate ports. Most of the components to do that exist but unfortunately there are also some pretty big blockers:
    You can't compile etherswitch as a module as far as I can tell so you need a new kernel.
    Even with that and after importing the mdio module you need a special igb driver to expose the mdio bus so it can be created as a device and allow the switch to be seen.
    The work has almost certainly already been done by Netasq/Stormshield as they have devices very similar to this and a FreeBSD base but I'm not sure if that code was ever made public. I could just be missing something!
    The T70 also has the intriguing option to create an mdio bus direct from the SoC without going via the NIC. That may be possible but I think would require code. I can see no reference of anyone doing that in FreeBSD though the etherswitch docs, such as they are, imply it could be attached like that.

    We could attempt to change the config in the 2-wire eeprom that the switch chip loads to enable the ports. However it looks like that is only accessible via the switch chip itself or via a clip on type programmer maybe. Also I have no idea how that might be formatted etc. Interestingly it looks like the default position for the DIP switches is 'off', the EEPROM is not connected. And connecting seems to make no difference in either OS as far as I can see. So maybe if doesn't have any config in it.

    The final nuclear option became apparent to me whilst chasing something else. I couldn't actually find the datasheet for the 88e6176 so I had to guess from other info but most Marvell chips are similar so... The chip can be configured by holding various pins high or low using external components. This way it can be in a cheap 5 port switch with no CPU or even eeprom required. It has a pin 'NO_CPU'; if that is set low implying there is a CPU it automatically disables all the ports when it is reset as it is at power on. The CPU then configured is later. This is a security measure so the ports are not connected together at boot until the OS is ready. That pin (pin 35) is pulled low by a 5K resistor, if that is disconnected it assumes there is no CPU and does not disable the ports. It would be nice if that was one of the DIP switches or a jumper.... nope.

    It is R607 as shown below. It is grounded via the adjacent pad on the unpopulated R614. By cutting the track under the blue line it removes the ground and the chip boots as a regular 5 port unmanaged switch.
    switch_chip_mod.jpg

    It should go without saying that this is not without risk. In fact I would say it is high risk! No one should attempt this! In all likelihood it will brick your, still expensive, T70 ๐Ÿ˜‰

    I may have simply been lucky.

    I will say it does not prevent the WG OS configuring the switch if you go back, or if we later found a way to do it from pfSense. It does make it less secure since all the ports are connected by default. PoE still works.

    Steve


  • Thank you so much @stephenw10 very grateful for your efforts here.

    I attempted this approach and it indeed worked perfectly. Had to use a microscope in order to sever that small connection!

    In my use case, each of the 5-port switch interfaces would belong to the same flat network segment. So, while the security aspect of this mod is important to consider, for me it makes no difference. In fact, for me it is simpler this way, as I actually wanted these 5 ports to function as an unmanaged switch. Cheers to you @stephenw10 :)

    -Bob

  • Netgate Administrator

    Nice. Let me know if you see anything unexpected. Those pins are all used for several things but I don't have the specific datasheet for that chip so I'm unsure exactly what. Probably potentially driving an LED somewhere. The NIC LEDs all seem to work as expected here though.

    Steve


  • @stephenw10 Each of the "1000" interface activity LEDs on my modified WatchGuard T70 operates as expected. As far as I can tell, the "Status", "Attn", and "Mode" LEDs do not illuminate under any circumstances, which for me is not super important. If, one day, the WGXepc package makes it possible to make use of these WatchGuard T70 LEDs from within pfSense, that would be great but I am not expecting this any time soon. Thanks again @stephenw10 :)

    Bob


  • @networkbob how did you get pfsense installed on the msata drive in the T70? Should I install using another system or can I do boot selection using the serial console and install via USB?

  • Netgate Administrator

    There's no way to install to it in the T70 directly as the BIOS is locked down, no way to select a boot device other than the mSATA.
    So, yes, install in something else and move it across. If that other thing is not a serial console device then be sure to enable the serial console in the webgui before moving it.

    Steve


  • @stephenw10 awesome thanks, I was able to get it installed, but it doesn't look like I was successful in breaking the trace under the blue line. I'm trying to score the board with a utility knife, how did you sever it?

  • Netgate Administrator

    Yes I used a small craft knife. I think I went over that with the corner of a watchmakers screwdriver. It's a delicate operation!


  • @stephenw10 I might just try to remove that resistor instead


  • @bruor confirming, used the super fine tip on my iron, popped the resistor off, switch is active on igb3!

    Thanks for the help!

  • Netgate Administrator

    Nice!
    I wish there was a better way. Maybe one day...

    Steve