Watchguard Firebox M440

pglover19

Anything else you want me to provide you tonight? I'm willing to work with you to get pfSense to recognize the other NIC interfaces.

pglover19

@stephenw10 said in Watchguard Firebox M440:

Hmm, OK so we have errors but unfortunately that's the same error we see on the SG-2220 for example where it only has two ports. So it could imply that all 4 are simply disabled.
That also matches the block diagram where all the connectivity is via PCIe . The 4 i354 NICs are in the SoC so connectivity with the switch would be via some other bus I would think.

The only connection is that one PCIex1 device. It is labelled Ethernet. But is x1 PCIe enough bandwidth there?

The best case scenario here is that the 4 on-board ethernet ports are connected to the switch chip and we can persuade the driver to attach to them. Then we use the switch in it's default config or find some other way to configure it.

You could test the switch to see what it's default config is. Try to ping between static clients on some of the ports there. It may be configured as a single layer 2 or as 3 groups. Or it may be disabled entirely as that's the most secure thing. That what we do.

Do you have the watchguard OS? The console boot log from that might determine what NICs are required to be attached.

Steve

Ok.. I tested the switch while booting up from WatchGuard OS. I assigned 2 computers with different IP addresses (192.168.1.10 and 192.168.1.8) in the same subnet (192.168.1.0/24) and connected them to the M440. I tried ports 1-24 and I cannot ping clients on the subnet. So, it looks like the switch has been disabled entirely.

Let me know what else you need me to test today. Are you optimistic that you can get the M440 working with pfSense?

stephenw10

Yeah sorry it was 3am UK time after my last reply!

Ok, so I wouldn't expect you to be able to talk between the switch ports in WGOS as they are configured as separate interfaces using VLANs. But you might be able to booting in pfSense, or simply booting to no OS. It would be intersting to see how the switch configures itself by default.

So there are a number of interesting things in that boot log.
The 4 onbooard NICs are recognised and brought up as interfaces.
Those are the 4x SGMII connections to the switch. WG renames them sw0-3.

[   17.179312] pss_create:  eth0    ->   sw0
[   17.200593] pss_create:  eth1    ->   sw1
[   17.233937] pss_create:  eth2    ->   sw2
[   17.250595] pss_create:  eth3    ->   sw3
[   17.273944] pss_create:  eth4    ->   eth0
[   17.300633] pss_create:  eth5    ->   eth25
[   17.330657] pss_create:  eth6    ->   eth26

And the other interfaces are assigned as VLANs on those individually:

[   17.364609] pss_create:  eth1   [ 9]  sw0    tag 4064
[   17.370237] pss_create:  eth2   [10]  sw1    tag 4065
[   17.375867] pss_create:  eth3   [11]  sw2    tag 4066
[   17.381490] pss_create:  eth4   [12]  sw3    tag 4067
[   17.387105] pss_create:  eth5   [13]  sw0    tag 4068
[   17.392724] pss_create:  eth6   [14]  sw1    tag 4069
[   17.398347] pss_create:  eth7   [15]  sw2    tag 4070
[   17.403987] pss_create:  eth8   [16]  sw3    tag 4071
[   17.409609] pss_create:  eth9   [17]  sw0    tag 4072
[   17.415210] pss_create:  eth10  [18]  sw1    tag 4073
[   17.420876] pss_create:  eth11  [19]  sw2    tag 4074
[   17.426500] pss_create:  eth12  [20]  sw3    tag 4075
[   17.432144] pss_create:  eth13  [21]  sw0    tag 4076
[   17.437772] pss_create:  eth14  [22]  sw1    tag 4077
[   17.443421] pss_create:  eth15  [23]  sw2    tag 4078
[   17.449035] pss_create:  eth16  [24]  sw3    tag 4079
[   17.454662] pss_create:  eth17  [25]  sw0    tag 4080
[   17.460289] pss_create:  eth18  [26]  sw1    tag 4081
[   17.465910] pss_create:  eth19  [27]  sw2    tag 4082
[   17.471519] pss_create:  eth20  [28]  sw3    tag 4083
[   17.477162] pss_create:  eth21  [29]  sw0    tag 4084
[   17.482815] pss_create:  eth22  [30]  sw1    tag 4085
[   17.488449] pss_create:  eth23  [31]  sw2    tag 4086
[   17.494103] pss_create:  eth24  [32]  sw3    tag 4087

I might have expected to see those on a lagg group to the switch which is what we do. The switch chip might not support it. I can't find any reference to the 98dx3035 to check though. It might have been custom for Lanner maybe.

If the switch comes up that way by default then we only need to get the internal NIC loaded to start talking to the ports. Of course that could be a significant challenge in itself.

What we almost certainly can't replicate is this:

[   17.113076] PEX: Marvell device fixup
[   17.118425] Configuring cross bar for xCat2 device 0xe7fe
[   17.130201]
[   17.134926] stub function bspPciFindDevReset returning MV_OK
[   17.147427]
[   17.147427] presteraSmi_init: Init OK!
[   17.152908] DMA - dma_area: 0xffffc9000d700000(v) ,dma_base: 0x80000000(p), dma_len: 0x200000
[   17.162261] HSU - hsu_area: 0xffffc9000d980000(v) ,hsu_base: 0x88000000(p), hsu_len: 0x800000

I assume that is the switch PCI interface for configuring it.

One thing that may give us more here is to boot pfSense in verbose mode. We might get more detail as to why the drivers are failing to attach in the boot log. To do that edit /boot/loader.conf and add the line boot_verbose="YES"

Steve

pglover19

I booted into pdSense and ports 1-24 is not working. The strange thing is that the lights on the ports shows not activity at all. Completely inactivity when I plug my CAT6 cable in the port. At least when booting from the WatchGuard’s OS the lights on the ports were active.

I will boot with no OS and provide you the detail log file. Give me 4 hours as I need to get some work around the house completed first.

pglover19

Attached is the information request. The switch does not work when booting with no OS. Additionally it does not work when booted with pfSense.

Let me know what additional information you need.

0_1539446726619_BootLog with Verbose.txt

stephenw10

Hmm, well that's disappointing. It implies the switch will require some setting up to use at all.

That log doesn't look any more verbose. Is that line still there in /boot/loader.conf?
Try adding boot_verbose="1" to /boot/loader.conf.local

Looking at the connector pin-out the two "PCIe" connectors that join the mainboard to the switch board in fact carry more than just PCIe. You can see they carry the SGMII signals from igb0-3 and the serial console and USB and a few other things. However it does not carry anything that looks like it might be some other way to connect to the switch like a second com port. It might possibly be usb connected, though I would have expected to see something logged. Try running:
usbconfig dump_device_desc

Another thing we might investigate is the switch debug port, J5. That looks like a serial port from the pinout (it's labelled SP2_X) it may be TTL level though. That might give us access to a switch CLI, if it has one.

Steve

pglover19

@stephenw10 said in Watchguard Firebox M440:

Hmm, well that's disappointing. It implies the switch will require some setting up to use at all.

That log doesn't look any more verbose. Is that line still there in /boot/loader.conf?
Try adding boot_verbose="1" to /boot/loader.conf.local

Looking at the connector pin-out the two "PCIe" connectors that join the mainboard to the switch board in fact carry more than just PCIe. You can see they carry the SGMII signals from igb0-3 and the serial console and USB and a few other things. However it does not carry anything that looks like it might be some other way to connect to the switch like a second com port. It might possibly be usb connected, though I would have expected to see something logged. Try running:
usbconfig dump_device_desc

Another thing we might investigate is the switch debug port, J5. That looks like a serial port from the pinout (it's labelled SP2_X) it may be TTL level though. That might give us access to a switch CLI, if it has one.

Steve

Give me 30 minutes and I will post the information requested..

pglover19

There is no loader.conf.local file in the boot directory.

stephenw10

Create that file if it's not there. You can create the file and put that value in it by running:
echo 'boot_verbose="1"' >> /boot/loader.conf.local

pglover19

@stephenw10 said in Watchguard Firebox M440:

Create that file if it's not there. You can create the file and put that value in it by running:
echo 'boot_verbose="1"' >> /boot/loader.conf.local

Ok.. Here are the results of the usbconfig command...

0_1539473923939_usbconfig results.txt

pglover19

@stephenw10 said in Watchguard Firebox M440:

Create that file if it's not there. You can create the file and put that value in it by running:
echo 'boot_verbose="1"' >> /boot/loader.conf.local

Ok.. Here are the results with verbose turned on I hope... Maybe I'm doing something incorrect.

0_1539474519176_BootLog with Verbose_2.txt

stephenw10

Ok, no USB devices we don't expect, shame.

The verbose console output should contain a lot more information than that.
Do you still see that line in the correct file? Here's a box I just booted verbose to test:

[2.4.4-RELEASE][root@5100.stevew.lan]/root: cat /boot/loader.conf.local
boot_verbose="1"

Steve

pglover19

@stephenw10 said in Watchguard Firebox M440:

Ok, no USB devices we don't expect, shame.

The verbose console output should contain a lot more information than that.
Do you still see that line in the correct file? Here's a box I just booted verbose to test:
[2.4.4-RELEASE][root@5100.stevew.lan]/root: cat /boot/loader.conf.local
boot_verbose="1"
Steve

I think I got it. Here is another file. Verbose is turned on.. This is the same size as my first verbose file upload.. Let me know...
0_1539476506397_BootLog with Verbose_3.txt

pglover19

@stephenw10 said in Watchguard Firebox M440:

Ok, no USB devices we don't expect, shame.

The verbose console output should contain a lot more information than that.
Do you still see that line in the correct file? Here's a box I just booted verbose to test:
[2.4.4-RELEASE][root@5100.stevew.lan]/root: cat /boot/loader.conf.local
boot_verbose="1"
Steve

I think I got it now... Here is another version of the log file with verbose turned on...

0_1539477665814_BootLog with Verbose_4.txt

stephenw10

Ah there we go! A lot more information there but unfortunately nothing significant regarding the igb NICs.

Noting a couple of relevant links:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228202
Not directly relevant but describes igb debug mode which we could try here to get more.
If it is that same error it's not surprising considering there is no PHY.

https://www.linuxquestions.org/questions/ubuntu-63/ubuntu-18-igb-driver-problem-on-marvell-88e1680-4175633325/
That is concerning as it shows Ubuntu also fails to attach with it's default driver. Potentially there is some issue with a resource conflict or it could be that the switch requires something before it presents to the NICs correctly. Though it doesn't look like that in the WGOS boot log. Nor does it look like they are using a custom driver there.
In that particular case though the OP has failed to understand how the switch is attached. 88e1680 is irrelevant.

Other possible workarounds we cannot try here as the BIOS is locked; disable PXE, disable VT-d.

If that is a regression it would be worth booting 2.3.5 to see if that behaves the same.

Do you have a TTL serial cable of the sort that can be connected to J5?

Steve

pglover19

@stephenw10 said in Watchguard Firebox M440:

Ah there we go! A lot more information there but unfortunately nothing significant regarding the igb NICs.

Noting a couple of relevant links:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228202
Not directly relevant but describes igb debug mode which we could try here to get more.
If it is that same error it's not surprising considering there is no PHY.

https://www.linuxquestions.org/questions/ubuntu-63/ubuntu-18-igb-driver-problem-on-marvell-88e1680-4175633325/
That is concerning as it shows Ubuntu also fails to attach with it's default driver. Potentially there is some issue with a resource conflict or it could be that the switch requires something before it presents to the NICs correctly. Though it doesn't look like that in the WGOS boot log. Nor does it look like they are using a custom driver there.
In that particular case though the OP has failed to understand how the switch is attached. 88e1680 is irrelevant.

Other possible workarounds we cannot try here as the BIOS is locked; disable PXE, disable VT-d.

If that is a regression it would be worth booting 2.3.5 to see if that behaves the same.

Do you have a TTL serial cable of the sort that can be connected to J5?

Steve

No.. I don't have TTL serial cable. I want to be clear on what you want me to do.. Here is what I think.

In the sys/dev/e1000/e1000_osdep.h set DBG to 1.
Boot from version 2.3.5 (where can I get this version).
Connect TTL serial cable to jumper J5.

pglover19

Going back to the BIOS discussion, it is prompting me for a password to enter the setup. Is there a way to clear or reset the password?

stephenw10

The igb driver with dbg set to 1 would require the module recompiling after making that change. Non-trivial.

The 2.3.5 install image can be found here:
https://nyifiles.pfsense.org/mirror/downloads/pfSense-CE-memstick-serial-2.3.5-RELEASE-amd64.img.gz

It would be interesting to see what is on J5. If it was me I would put a scope on it first to check what the voltage level and baud rate is. However most people don't have access to that.
That may be the only way to configure the switch in the absence of a driver for the Marvell PCI interface.

There is probably no way to remove the password from the BIOS. It's set by default so resetting the CMOS for example will not help.

Steve

pglover19

@stephenw10 said in Watchguard Firebox M440:

The igb driver with dbg set to 1 would require the module recompiling after making that change. Non-trivial.

The 2.3.5 install image can be found here:
https://nyifiles.pfsense.org/mirror/downloads/pfSense-CE-memstick-serial-2.3.5-RELEASE-amd64.img.gz

It would be interesting to see what is on J5. If it was me I would put a scope on it first to check what the voltage level and baud rate is. However most people don't have access to that.
That may be the only way to configure the switch in the absence of a driver for the Marvell PCI interface.

There is probably no way to remove the password from the BIOS. It's set by default so resetting the CMOS for example will not help.

Steve

For some reason, the M440 is not booting from the 2.3.5 image on the SSD. I selected the ISO image. Not sure what is going on...

Ok.. Maybe I selected the wrong 2.3.5 image.

pglover19

For some reason, pfSense 2.3.5 will not boot at all on the M440. I tried installing this version on a SSD and Hard Drive and the M440 is not booting from the disk (SSD or HD). Installed v 2.4.4 on the SSD and it is booting fine.. Not sure what is going on with version 2.3.5.