Watchguard Firebox M440

stephenw10

Yeah sorry it was 3am UK time after my last reply!

Ok, so I wouldn't expect you to be able to talk between the switch ports in WGOS as they are configured as separate interfaces using VLANs. But you might be able to booting in pfSense, or simply booting to no OS. It would be intersting to see how the switch configures itself by default.

So there are a number of interesting things in that boot log.
The 4 onbooard NICs are recognised and brought up as interfaces.
Those are the 4x SGMII connections to the switch. WG renames them sw0-3.

[   17.179312] pss_create:  eth0    ->   sw0
[   17.200593] pss_create:  eth1    ->   sw1
[   17.233937] pss_create:  eth2    ->   sw2
[   17.250595] pss_create:  eth3    ->   sw3
[   17.273944] pss_create:  eth4    ->   eth0
[   17.300633] pss_create:  eth5    ->   eth25
[   17.330657] pss_create:  eth6    ->   eth26

And the other interfaces are assigned as VLANs on those individually:

[   17.364609] pss_create:  eth1   [ 9]  sw0    tag 4064
[   17.370237] pss_create:  eth2   [10]  sw1    tag 4065
[   17.375867] pss_create:  eth3   [11]  sw2    tag 4066
[   17.381490] pss_create:  eth4   [12]  sw3    tag 4067
[   17.387105] pss_create:  eth5   [13]  sw0    tag 4068
[   17.392724] pss_create:  eth6   [14]  sw1    tag 4069
[   17.398347] pss_create:  eth7   [15]  sw2    tag 4070
[   17.403987] pss_create:  eth8   [16]  sw3    tag 4071
[   17.409609] pss_create:  eth9   [17]  sw0    tag 4072
[   17.415210] pss_create:  eth10  [18]  sw1    tag 4073
[   17.420876] pss_create:  eth11  [19]  sw2    tag 4074
[   17.426500] pss_create:  eth12  [20]  sw3    tag 4075
[   17.432144] pss_create:  eth13  [21]  sw0    tag 4076
[   17.437772] pss_create:  eth14  [22]  sw1    tag 4077
[   17.443421] pss_create:  eth15  [23]  sw2    tag 4078
[   17.449035] pss_create:  eth16  [24]  sw3    tag 4079
[   17.454662] pss_create:  eth17  [25]  sw0    tag 4080
[   17.460289] pss_create:  eth18  [26]  sw1    tag 4081
[   17.465910] pss_create:  eth19  [27]  sw2    tag 4082
[   17.471519] pss_create:  eth20  [28]  sw3    tag 4083
[   17.477162] pss_create:  eth21  [29]  sw0    tag 4084
[   17.482815] pss_create:  eth22  [30]  sw1    tag 4085
[   17.488449] pss_create:  eth23  [31]  sw2    tag 4086
[   17.494103] pss_create:  eth24  [32]  sw3    tag 4087

I might have expected to see those on a lagg group to the switch which is what we do. The switch chip might not support it. I can't find any reference to the 98dx3035 to check though. It might have been custom for Lanner maybe.

If the switch comes up that way by default then we only need to get the internal NIC loaded to start talking to the ports. Of course that could be a significant challenge in itself.

What we almost certainly can't replicate is this:

[   17.113076] PEX: Marvell device fixup
[   17.118425] Configuring cross bar for xCat2 device 0xe7fe
[   17.130201]
[   17.134926] stub function bspPciFindDevReset returning MV_OK
[   17.147427]
[   17.147427] presteraSmi_init: Init OK!
[   17.152908] DMA - dma_area: 0xffffc9000d700000(v) ,dma_base: 0x80000000(p), dma_len: 0x200000
[   17.162261] HSU - hsu_area: 0xffffc9000d980000(v) ,hsu_base: 0x88000000(p), hsu_len: 0x800000

I assume that is the switch PCI interface for configuring it.

One thing that may give us more here is to boot pfSense in verbose mode. We might get more detail as to why the drivers are failing to attach in the boot log. To do that edit /boot/loader.conf and add the line boot_verbose="YES"

Steve

pglover19

I booted into pdSense and ports 1-24 is not working. The strange thing is that the lights on the ports shows not activity at all. Completely inactivity when I plug my CAT6 cable in the port. At least when booting from the WatchGuard’s OS the lights on the ports were active.

I will boot with no OS and provide you the detail log file. Give me 4 hours as I need to get some work around the house completed first.

pglover19

Attached is the information request. The switch does not work when booting with no OS. Additionally it does not work when booted with pfSense.

Let me know what additional information you need.

0_1539446726619_BootLog with Verbose.txt

stephenw10

Hmm, well that's disappointing. It implies the switch will require some setting up to use at all.

That log doesn't look any more verbose. Is that line still there in /boot/loader.conf?
Try adding boot_verbose="1" to /boot/loader.conf.local

Looking at the connector pin-out the two "PCIe" connectors that join the mainboard to the switch board in fact carry more than just PCIe. You can see they carry the SGMII signals from igb0-3 and the serial console and USB and a few other things. However it does not carry anything that looks like it might be some other way to connect to the switch like a second com port. It might possibly be usb connected, though I would have expected to see something logged. Try running:
usbconfig dump_device_desc

Another thing we might investigate is the switch debug port, J5. That looks like a serial port from the pinout (it's labelled SP2_X) it may be TTL level though. That might give us access to a switch CLI, if it has one.

Steve

pglover19

@stephenw10 said in Watchguard Firebox M440:

Hmm, well that's disappointing. It implies the switch will require some setting up to use at all.

That log doesn't look any more verbose. Is that line still there in /boot/loader.conf?
Try adding boot_verbose="1" to /boot/loader.conf.local

Looking at the connector pin-out the two "PCIe" connectors that join the mainboard to the switch board in fact carry more than just PCIe. You can see they carry the SGMII signals from igb0-3 and the serial console and USB and a few other things. However it does not carry anything that looks like it might be some other way to connect to the switch like a second com port. It might possibly be usb connected, though I would have expected to see something logged. Try running:
usbconfig dump_device_desc

Another thing we might investigate is the switch debug port, J5. That looks like a serial port from the pinout (it's labelled SP2_X) it may be TTL level though. That might give us access to a switch CLI, if it has one.

Steve

Give me 30 minutes and I will post the information requested..

pglover19

There is no loader.conf.local file in the boot directory.

stephenw10

Create that file if it's not there. You can create the file and put that value in it by running:
echo 'boot_verbose="1"' >> /boot/loader.conf.local

pglover19

@stephenw10 said in Watchguard Firebox M440:

Create that file if it's not there. You can create the file and put that value in it by running:
echo 'boot_verbose="1"' >> /boot/loader.conf.local

Ok.. Here are the results of the usbconfig command...

0_1539473923939_usbconfig results.txt

pglover19

@stephenw10 said in Watchguard Firebox M440:

Create that file if it's not there. You can create the file and put that value in it by running:
echo 'boot_verbose="1"' >> /boot/loader.conf.local

Ok.. Here are the results with verbose turned on I hope... Maybe I'm doing something incorrect.

0_1539474519176_BootLog with Verbose_2.txt

stephenw10

Ok, no USB devices we don't expect, shame.

The verbose console output should contain a lot more information than that.
Do you still see that line in the correct file? Here's a box I just booted verbose to test:

[2.4.4-RELEASE][root@5100.stevew.lan]/root: cat /boot/loader.conf.local
boot_verbose="1"

Steve

pglover19

@stephenw10 said in Watchguard Firebox M440:

Ok, no USB devices we don't expect, shame.

The verbose console output should contain a lot more information than that.
Do you still see that line in the correct file? Here's a box I just booted verbose to test:
[2.4.4-RELEASE][root@5100.stevew.lan]/root: cat /boot/loader.conf.local
boot_verbose="1"
Steve

I think I got it. Here is another file. Verbose is turned on.. This is the same size as my first verbose file upload.. Let me know...
0_1539476506397_BootLog with Verbose_3.txt

pglover19

@stephenw10 said in Watchguard Firebox M440:

Ok, no USB devices we don't expect, shame.

The verbose console output should contain a lot more information than that.
Do you still see that line in the correct file? Here's a box I just booted verbose to test:
[2.4.4-RELEASE][root@5100.stevew.lan]/root: cat /boot/loader.conf.local
boot_verbose="1"
Steve

I think I got it now... Here is another version of the log file with verbose turned on...

0_1539477665814_BootLog with Verbose_4.txt

stephenw10

Ah there we go! A lot more information there but unfortunately nothing significant regarding the igb NICs.

Noting a couple of relevant links:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228202
Not directly relevant but describes igb debug mode which we could try here to get more.
If it is that same error it's not surprising considering there is no PHY.

https://www.linuxquestions.org/questions/ubuntu-63/ubuntu-18-igb-driver-problem-on-marvell-88e1680-4175633325/
That is concerning as it shows Ubuntu also fails to attach with it's default driver. Potentially there is some issue with a resource conflict or it could be that the switch requires something before it presents to the NICs correctly. Though it doesn't look like that in the WGOS boot log. Nor does it look like they are using a custom driver there.
In that particular case though the OP has failed to understand how the switch is attached. 88e1680 is irrelevant.

Other possible workarounds we cannot try here as the BIOS is locked; disable PXE, disable VT-d.

If that is a regression it would be worth booting 2.3.5 to see if that behaves the same.

Do you have a TTL serial cable of the sort that can be connected to J5?

Steve

pglover19

@stephenw10 said in Watchguard Firebox M440:

Ah there we go! A lot more information there but unfortunately nothing significant regarding the igb NICs.

Noting a couple of relevant links:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228202
Not directly relevant but describes igb debug mode which we could try here to get more.
If it is that same error it's not surprising considering there is no PHY.

https://www.linuxquestions.org/questions/ubuntu-63/ubuntu-18-igb-driver-problem-on-marvell-88e1680-4175633325/
That is concerning as it shows Ubuntu also fails to attach with it's default driver. Potentially there is some issue with a resource conflict or it could be that the switch requires something before it presents to the NICs correctly. Though it doesn't look like that in the WGOS boot log. Nor does it look like they are using a custom driver there.
In that particular case though the OP has failed to understand how the switch is attached. 88e1680 is irrelevant.

Other possible workarounds we cannot try here as the BIOS is locked; disable PXE, disable VT-d.

If that is a regression it would be worth booting 2.3.5 to see if that behaves the same.

Do you have a TTL serial cable of the sort that can be connected to J5?

Steve

No.. I don't have TTL serial cable. I want to be clear on what you want me to do.. Here is what I think.

In the sys/dev/e1000/e1000_osdep.h set DBG to 1.
Boot from version 2.3.5 (where can I get this version).
Connect TTL serial cable to jumper J5.

pglover19

Going back to the BIOS discussion, it is prompting me for a password to enter the setup. Is there a way to clear or reset the password?

stephenw10

The igb driver with dbg set to 1 would require the module recompiling after making that change. Non-trivial.

The 2.3.5 install image can be found here:
https://nyifiles.pfsense.org/mirror/downloads/pfSense-CE-memstick-serial-2.3.5-RELEASE-amd64.img.gz

It would be interesting to see what is on J5. If it was me I would put a scope on it first to check what the voltage level and baud rate is. However most people don't have access to that.
That may be the only way to configure the switch in the absence of a driver for the Marvell PCI interface.

There is probably no way to remove the password from the BIOS. It's set by default so resetting the CMOS for example will not help.

Steve

pglover19

@stephenw10 said in Watchguard Firebox M440:

The igb driver with dbg set to 1 would require the module recompiling after making that change. Non-trivial.

The 2.3.5 install image can be found here:
https://nyifiles.pfsense.org/mirror/downloads/pfSense-CE-memstick-serial-2.3.5-RELEASE-amd64.img.gz

It would be interesting to see what is on J5. If it was me I would put a scope on it first to check what the voltage level and baud rate is. However most people don't have access to that.
That may be the only way to configure the switch in the absence of a driver for the Marvell PCI interface.

There is probably no way to remove the password from the BIOS. It's set by default so resetting the CMOS for example will not help.

Steve

For some reason, the M440 is not booting from the 2.3.5 image on the SSD. I selected the ISO image. Not sure what is going on...

Ok.. Maybe I selected the wrong 2.3.5 image.

pglover19

For some reason, pfSense 2.3.5 will not boot at all on the M440. I tried installing this version on a SSD and Hard Drive and the M440 is not booting from the disk (SSD or HD). Installed v 2.4.4 on the SSD and it is booting fine.. Not sure what is going on with version 2.3.5.

stephenw10

Hmm, it's probably a UEFI boot issue. I could believe it only boots UEFI from SATA. Not sure I ever tested that on the m400.

Do you have a spare CF card you can try the 2.3.5 Nano image on?

Steve

pglover19

I guess we are finding out that pfSense will not recognize the 24 ports that is part of the switch. I assume I made a bad purchase and will need to try to sell these units to get my money back. Really disappointing.