Watchguard Firebox M440

pglover19

@stephenw10 said in Watchguard Firebox M440:

Hmm, OK.

So the BIOS is password protected and there's likely no way to remove that. Though the hardware does look identical to the Lanner default so the standard bios may work.

The expected 3 i354 NICs appear in the pciconf but are not attached to by the driver for some reason. The boot log doesn't show it failing but it also doesn;t show the igb or ix NICs, it appears incomplete. I assume that was copy/pasted from the console rather than the dmesg output? You could also look at the system log since there won't be anything else in it.

Looking at the block in the manual it shows the Marvell switch connected to system via SGMIIx4 and one additional PCIe device. Which is probably this:
none8@pci0:2:0:0:       class=0x020000 card=0x11ab11ab chip=0xe7fe11ab rev=0x03 hdr=0x00
    vendor     = 'Marvell Technology Group Ltd.'
    class      = network
    subclass   = ethernet
If the only way to configure that is via that PCIe device it would require a driver, likely something written from scratch, and that's unlikely to happen.
chip=0x1f418086 is the normal PCI device ID for the C2000 NIC so we need to see if the system log shows some error indicating why it's not attaching. Probably because it doesn't have a PHY in the expected way. Or perhaps it's something completely different.

Steve

So what else would you like me to post. Please be specific in the commands as I am a novice..

stephenw10

The actual file /var/log/dmesg.boot should have some output showing the drivers attaching or failing to attach.

I'm assuming you were able to assign igb0 and access the GUI and SSH?

Steve

pglover19

@stephenw10 said in Watchguard Firebox M440:

The actual file /var/log/dmesg.boot should have some output showing the drivers attaching or failing to attach.

I'm assuming you were able to assign igb0 and access the GUI and SSH?

Steve

I will go ahead and do and upload the dmesg.boot file shortly. First, I need to reinstall pfSense on the SSD. It seems to be corrupt for some reason.

pglover19

Reinstalled pfSense on the SSD and now the M440 is booting pfSense from the SSD. I will post the boot file shortly.

pglover19

I was able to setup the LAN interface and access the GUI using the default 192.168.1.1. I turned on SSH and was able to use WINSCP to retrieve the boot file. Attached is the dmesg.boot file. Let me know what else you need from me. I am available most of the night.

FYI.. I had to rename the file in order to upload it.0_1539392024912_dmesg.boot.txt

stephenw10

Hmm, OK so we have errors but unfortunately that's the same error we see on the SG-2220 for example where it only has two ports. So it could imply that all 4 are simply disabled.
That also matches the block diagram where all the connectivity is via PCIe . The 4 i354 NICs are in the SoC so connectivity with the switch would be via some other bus I would think.

The only connection is that one PCIex1 device. It is labelled Ethernet. But is x1 PCIe enough bandwidth there?

The best case scenario here is that the 4 on-board ethernet ports are connected to the switch chip and we can persuade the driver to attach to them. Then we use the switch in it's default config or find some other way to configure it.

You could test the switch to see what it's default config is. Try to ping between static clients on some of the ports there. It may be configured as a single layer 2 or as 3 groups. Or it may be disabled entirely as that's the most secure thing. That what we do.

Do you have the watchguard OS? The console boot log from that might determine what NICs are required to be attached.

Steve

pglover19

Yes.. I will disconnect the SSD drive and boot from the CF and copy the content on the console to a file. I will upload the file in 10 minutes.

pglover19

Here is the WatchGuard boot content...

0_1539398724589_WatchGuard Boot.txt

pglover19

Anything else you want me to provide you tonight? I'm willing to work with you to get pfSense to recognize the other NIC interfaces.

pglover19

@stephenw10 said in Watchguard Firebox M440:

Hmm, OK so we have errors but unfortunately that's the same error we see on the SG-2220 for example where it only has two ports. So it could imply that all 4 are simply disabled.
That also matches the block diagram where all the connectivity is via PCIe . The 4 i354 NICs are in the SoC so connectivity with the switch would be via some other bus I would think.

The only connection is that one PCIex1 device. It is labelled Ethernet. But is x1 PCIe enough bandwidth there?

The best case scenario here is that the 4 on-board ethernet ports are connected to the switch chip and we can persuade the driver to attach to them. Then we use the switch in it's default config or find some other way to configure it.

You could test the switch to see what it's default config is. Try to ping between static clients on some of the ports there. It may be configured as a single layer 2 or as 3 groups. Or it may be disabled entirely as that's the most secure thing. That what we do.

Do you have the watchguard OS? The console boot log from that might determine what NICs are required to be attached.

Steve

Ok.. I tested the switch while booting up from WatchGuard OS. I assigned 2 computers with different IP addresses (192.168.1.10 and 192.168.1.8) in the same subnet (192.168.1.0/24) and connected them to the M440. I tried ports 1-24 and I cannot ping clients on the subnet. So, it looks like the switch has been disabled entirely.

Let me know what else you need me to test today. Are you optimistic that you can get the M440 working with pfSense?

stephenw10

Yeah sorry it was 3am UK time after my last reply!

Ok, so I wouldn't expect you to be able to talk between the switch ports in WGOS as they are configured as separate interfaces using VLANs. But you might be able to booting in pfSense, or simply booting to no OS. It would be intersting to see how the switch configures itself by default.

So there are a number of interesting things in that boot log.
The 4 onbooard NICs are recognised and brought up as interfaces.
Those are the 4x SGMII connections to the switch. WG renames them sw0-3.

[   17.179312] pss_create:  eth0    ->   sw0
[   17.200593] pss_create:  eth1    ->   sw1
[   17.233937] pss_create:  eth2    ->   sw2
[   17.250595] pss_create:  eth3    ->   sw3
[   17.273944] pss_create:  eth4    ->   eth0
[   17.300633] pss_create:  eth5    ->   eth25
[   17.330657] pss_create:  eth6    ->   eth26

And the other interfaces are assigned as VLANs on those individually:

[   17.364609] pss_create:  eth1   [ 9]  sw0    tag 4064
[   17.370237] pss_create:  eth2   [10]  sw1    tag 4065
[   17.375867] pss_create:  eth3   [11]  sw2    tag 4066
[   17.381490] pss_create:  eth4   [12]  sw3    tag 4067
[   17.387105] pss_create:  eth5   [13]  sw0    tag 4068
[   17.392724] pss_create:  eth6   [14]  sw1    tag 4069
[   17.398347] pss_create:  eth7   [15]  sw2    tag 4070
[   17.403987] pss_create:  eth8   [16]  sw3    tag 4071
[   17.409609] pss_create:  eth9   [17]  sw0    tag 4072
[   17.415210] pss_create:  eth10  [18]  sw1    tag 4073
[   17.420876] pss_create:  eth11  [19]  sw2    tag 4074
[   17.426500] pss_create:  eth12  [20]  sw3    tag 4075
[   17.432144] pss_create:  eth13  [21]  sw0    tag 4076
[   17.437772] pss_create:  eth14  [22]  sw1    tag 4077
[   17.443421] pss_create:  eth15  [23]  sw2    tag 4078
[   17.449035] pss_create:  eth16  [24]  sw3    tag 4079
[   17.454662] pss_create:  eth17  [25]  sw0    tag 4080
[   17.460289] pss_create:  eth18  [26]  sw1    tag 4081
[   17.465910] pss_create:  eth19  [27]  sw2    tag 4082
[   17.471519] pss_create:  eth20  [28]  sw3    tag 4083
[   17.477162] pss_create:  eth21  [29]  sw0    tag 4084
[   17.482815] pss_create:  eth22  [30]  sw1    tag 4085
[   17.488449] pss_create:  eth23  [31]  sw2    tag 4086
[   17.494103] pss_create:  eth24  [32]  sw3    tag 4087

I might have expected to see those on a lagg group to the switch which is what we do. The switch chip might not support it. I can't find any reference to the 98dx3035 to check though. It might have been custom for Lanner maybe.

If the switch comes up that way by default then we only need to get the internal NIC loaded to start talking to the ports. Of course that could be a significant challenge in itself.

What we almost certainly can't replicate is this:

[   17.113076] PEX: Marvell device fixup
[   17.118425] Configuring cross bar for xCat2 device 0xe7fe
[   17.130201]
[   17.134926] stub function bspPciFindDevReset returning MV_OK
[   17.147427]
[   17.147427] presteraSmi_init: Init OK!
[   17.152908] DMA - dma_area: 0xffffc9000d700000(v) ,dma_base: 0x80000000(p), dma_len: 0x200000
[   17.162261] HSU - hsu_area: 0xffffc9000d980000(v) ,hsu_base: 0x88000000(p), hsu_len: 0x800000

I assume that is the switch PCI interface for configuring it.

One thing that may give us more here is to boot pfSense in verbose mode. We might get more detail as to why the drivers are failing to attach in the boot log. To do that edit /boot/loader.conf and add the line boot_verbose="YES"

Steve

pglover19

I booted into pdSense and ports 1-24 is not working. The strange thing is that the lights on the ports shows not activity at all. Completely inactivity when I plug my CAT6 cable in the port. At least when booting from the WatchGuard’s OS the lights on the ports were active.

I will boot with no OS and provide you the detail log file. Give me 4 hours as I need to get some work around the house completed first.

pglover19

Attached is the information request. The switch does not work when booting with no OS. Additionally it does not work when booted with pfSense.

Let me know what additional information you need.

0_1539446726619_BootLog with Verbose.txt

stephenw10

Hmm, well that's disappointing. It implies the switch will require some setting up to use at all.

That log doesn't look any more verbose. Is that line still there in /boot/loader.conf?
Try adding boot_verbose="1" to /boot/loader.conf.local

Looking at the connector pin-out the two "PCIe" connectors that join the mainboard to the switch board in fact carry more than just PCIe. You can see they carry the SGMII signals from igb0-3 and the serial console and USB and a few other things. However it does not carry anything that looks like it might be some other way to connect to the switch like a second com port. It might possibly be usb connected, though I would have expected to see something logged. Try running:
usbconfig dump_device_desc

Another thing we might investigate is the switch debug port, J5. That looks like a serial port from the pinout (it's labelled SP2_X) it may be TTL level though. That might give us access to a switch CLI, if it has one.

Steve

pglover19

@stephenw10 said in Watchguard Firebox M440:

Hmm, well that's disappointing. It implies the switch will require some setting up to use at all.

That log doesn't look any more verbose. Is that line still there in /boot/loader.conf?
Try adding boot_verbose="1" to /boot/loader.conf.local

Looking at the connector pin-out the two "PCIe" connectors that join the mainboard to the switch board in fact carry more than just PCIe. You can see they carry the SGMII signals from igb0-3 and the serial console and USB and a few other things. However it does not carry anything that looks like it might be some other way to connect to the switch like a second com port. It might possibly be usb connected, though I would have expected to see something logged. Try running:
usbconfig dump_device_desc

Another thing we might investigate is the switch debug port, J5. That looks like a serial port from the pinout (it's labelled SP2_X) it may be TTL level though. That might give us access to a switch CLI, if it has one.

Steve

Give me 30 minutes and I will post the information requested..

pglover19

There is no loader.conf.local file in the boot directory.

stephenw10

Create that file if it's not there. You can create the file and put that value in it by running:
echo 'boot_verbose="1"' >> /boot/loader.conf.local

pglover19

@stephenw10 said in Watchguard Firebox M440:

Create that file if it's not there. You can create the file and put that value in it by running:
echo 'boot_verbose="1"' >> /boot/loader.conf.local

Ok.. Here are the results of the usbconfig command...

0_1539473923939_usbconfig results.txt

pglover19

@stephenw10 said in Watchguard Firebox M440:

Create that file if it's not there. You can create the file and put that value in it by running:
echo 'boot_verbose="1"' >> /boot/loader.conf.local

Ok.. Here are the results with verbose turned on I hope... Maybe I'm doing something incorrect.

0_1539474519176_BootLog with Verbose_2.txt

stephenw10

Ok, no USB devices we don't expect, shame.

The verbose console output should contain a lot more information than that.
Do you still see that line in the correct file? Here's a box I just booted verbose to test:

[2.4.4-RELEASE][root@5100.stevew.lan]/root: cat /boot/loader.conf.local
boot_verbose="1"

Steve