Watchguard XTM 5 Series

Billyboy

Hi all,
I have two XTM5 (505 and 515) with the BIOS:

Vendor: American Megatrends Inc.
Version: 080015
Release Date: 02/03/2010

and upgraded hardware on both:

CPU: Intel E5800 @ 3.2 Ghz
RAM: 2 GB
SSD: 60 GB

running on the latest pfsense 2.3.4-p1.

…

I recognized problem:

When I pull a cable out of any of the em ports, Pfsense needs more than 5 Minutes to change the interface to down (both in the GUI DASHBOARD as well as on the Interfaces status page). However the port LEDs are switched off immediately.
...

To resolve this problem, if have done some more investigations:

I have done a fresh install 2.3.4-P1 on SSD (through PC, choosing embedded Kernel). I resetted to factory defaults and configured the two default interfaces only: em0 (WAN, DHCP) and em1 (LAN, static IP).

Still same issue, it takes minutes till PFSense recognize the disconnected cable.

Are there tuning parameters for the EM / Intel NICs?

I checked "Disable hardware checksum offload" already, no change.

Any Idea?

stephenw10

Still no idea how that could happen I'm afraid.

To recap you see the link as up reported by ifconfig during that time?

You see that same behaviour on both your boxes?

Steve

DeLorean

@stephenw10:

Still no idea how that could happen I'm afraid.

To recap you see the link as up reported by ifconfig during that time?

You see that same behaviour on both your boxes?

Steve

I know Billyboy from outside the forum,
and i done some testing to reconstruct the problem.
I have tested with
pfSense 2.2.6
pfSense 2.3.2
pfSense 2.3.3
pfSense 2.3.4

The problem is the following :
When you disconnect the WAN cable, or the LAN cable from Opt1,Opt2,Opt3…etc
that after the cable is disconnected, the Web UI stills show the interfaces as online.
Normally when you then refresh the Web UI or press F5, the interface must show offline, but it doesn't.
All the interfaces em0,em1,em2,em3,em4,em5 keeps showing that they are online,
even after multiple times pressing F5.
The only interface that response the right way after disconnecting the cable, is the fx0 (100Mbps) interface.
This behaviour occurs in 2.3.4 , 2.3.3, 2.3.2 , only in 2.2.6 the interfaces shows the correct status (online or offline).
If a LAN cable is disconnected in 2.2.6 , and press F5, the interface is immediately showing offline.

This behaviour occurs on the 2 boxes that i have here, so its definitely not a hardware issue, but a software issue.
Same result with the embedded version and full version.
So it cannot be, that with behaviour CARP of WAN Failover can work properly by other users that use CARP or Failover.

Grtz
DeLorean

Billyboy

@DeLorean:

@stephenw10:

Still no idea how that could happen I'm afraid.

To recap you see the link as up reported by ifconfig during that time?

You see that same behaviour on both your boxes?

Steve

…

I have tested with
pfSense 2.2.6
pfSense 2.3.2
pfSense 2.3.3
pfSense 2.3.4

The problem is the following :
When you disconnect the WAN cable, or the LAN cable from Opt1,Opt2,Opt3...etc
that after the cable is disconnected, the Web UI stills show the interfaces as online.
......

After a maximum of 10 minutes (differs from try to try) the system recognizes the disconnected cable. But reconnection is recognized immediately.

Tried the latest beta/nightly as well, no success.

I have already tried the following without success:

Enabled all TCP offloads
Disabled MSI/MSIx and flow-control
hw.pci.enable_msix=0
hw.pci.enable_msi=0
hw.em.fc_setting=0

As we have seen this now on 4 boxes, this is probably a general problem/bug!!!

Who can test this on his existing box with 2.3.4?

Billyboy

@slaven:

I just checked and the fan in my PSU is connected via a 2-pin connector. My box uses the same PSU as described in https://www.watchguard.com/docs/corporate/wg_xtm5De-MFR_instructions.pdf on page 8. Doing some more digging on the PSU (ST-220FUB-05E made by Seventeam) it seems as the PSU fan is temperature controlled as well. I will have to torture my PSU a little bit to find out, if the fan really is controlled by a temp probe. The PSU fan is a different Sunon fan than the three CPU / system fans - only 20 vs 28 mm in depth.

I have made some good experience with Noctua NF-A4x20 fans lately (http://noctua.at/en/products/fan/nf-a4x20-flx/specification). They run at 5000 rpm @ 12 V and are really silent. Airflow sure is less than on the original Sunon - ~ 10 vs 28 m³/h at max speed. But the Noctual fan has almost identical static pressure (both at max rpm). As the Sunon fans do not need to run at maximum RPM to cool the system accordingly, static pressure on the Noctua fan is higher relative to RPM. Especially in a CPU cooling configuration as used in the XTM5 the Noctua should work well in theory, as we will need high static pressure first, airflow comes second.

I am about to upgrade my box with four if theses fans, but I am still trying to figure out if I should get the PWM or the standard version of the Noctua fan for CPU and system fans. The price is identical.

Did you succeed?
Was it worth the money and effort?

dlucas46

Hi all,

For those of you with Xeons that would like coretemp to report the correct temp, you can try this recompiled coretemp module.

I have set the TJMax value to 70c

Remove the png extension and upload to /boot/coretemp2.ko

Chmod 755 coretemp2.ko

In your /boot/loader.conf.local add the following:

coretemp2_load="YES"

Reboot.

You should now have a correct temperature reading. I did this several months ago and its been working fine.

If your CPU is in the same family as L5420 this should also work for you.

coretemp2.ko.png

mredding

Not sure anyone found this yet, but I was convinced there must be a PCIe "female-to-female" adapter that would be usable and I came across this:

https://www.aliexpress.com/store/product/PCI-Express-x1-x4-x8-x16-Male-to-Male-PCI-E-3-0-Male-to-Female/113308_32830684089.html

The "R33FF" model appears to be a x16 female to female adapter. It's a little pricey at $51 plus shipping. Anyone try anything like this to use the PCIe slot? I'm not sure I want to use the PCIe slot for anything frankly, but hopefully this helps someone (or someone can help me come up with an excuse to try it.)

A Former User

This post is deleted!

mredding

I thought I would share how I got pfSense 2.4 (mostly) running on an SSD on my XTM 5 version 2 box (initially a XTM 515) since I encountered a few snags along the way that I hadn’t seen brought up in this thread. In the previous forty some pages, there were a lot of questions about what the version 2 boxes had under the hood: It’s a Celeron E3400 processor with 2x1GB RAM. As far as I can tell, everything else is similar. My BIOS firmware declared it was “WG BIOS 1.3” on the LCD, which is newer than the 1.2 BIOS that is (modified or otherwise) floating around in this thread. See later on for more on the BIOS.

The only thing I haven’t resolved is that the WAN interface fails to get an IP address on boot. More details at the end; any help would be appreciated.

Anyway, here goes:

Remove the unsupported Cavium card & the 1GB CF card.
Take a Dremel tool to a 2.5” to 3.5” bay adapter to make it fit. Mine had holes that lined up relatively closely with the power supply screws, so after cutting the adapter to size I just drilled those holes out a bit larger.
I hooked up the SSD to my laptop via USB to SATA adapter similar to this one: http://www.newertech.com/products/usb3_universaldriveadap.php
I used VirtualBox on Ubuntu to install pfSense to the hard drive. This presented a few hiccups:

I allowed access to a raw hard disk (/dev/sdc in my case) using this procedure: https://www.serverwatch.com/server-tutorials/using-a-physical-hard-drive-with-a-virtualbox-vm.html However, I needed root permissions to both create the VirtualBox hard drive that pointed to the real drive (sudo VboxManage…) and I also needed to run VirtualBox as root as well for it to work. I’m sure there is a better way to manage permissions and not run as root, but I really wasn’t concerned enough to investigate.

I set up the VM with 2GB of RAM, the same amount I had on the XTM 5.

I enabled the serial console in the VM using a “host pipe” as explained here so I could use it in VirtualBox if necessary: https://www.gonwan.com/2014/04/07/setting-up-serial-console-on-virtualbox/

I also included two network adapters so I could set up WAN and LAN in VirtualBox if necessary. The first, for WAN, I left as NAT, and the second for LAN I created a host-only network on vmnet0 with DHCP disabled. I changed the Host IP to 192.168.56.10 so I could give pfSense 192.168.56.1. See https://www.virtualbox.org/manual/ch06.html for more on VirtualBox networking. I theoretically wouldn’t need these network adapters or the serial console, since others installing previous versions of pfSense to a hard disk simply did so, dropped it in to the Firebox before rebooting, and configured it from there, but…

Setup from the ISO does not enable the serial console by default, which I realized after I had already installed pfSense to the hard drive and tried to boot it on the Firebox. Perhaps there is a way to do this from setup itself or the console after installation, but I couldn’t find it readily. So, I fired up pfSense in VirtualBox, configured the network adapters, and connected to the web interface at https://192.168.56.1/. From there, the serial console can be enabled in System > Advanced. I connected to the host pipe with minicom to test the serial console and reboot. Voila!

After installing the SSD in the Firebox, I grabbed an old Windows XP laptop out of storage that actually had a serial port on it to connect to the serial console. After putting the appropriate settings in PuTTY, I pushed the button, and just got an error “ding.” No error message; just “ding.” So, I dug out a USB to Serial adapter and used my laptop with Ubuntu. Minicom and gtkterm worked generally okay, so out of curiosity I installed PuTTY in Ubuntu. I put in the settings, pushed the button… error “ding.” I’m probably missing something obvious. Anyway, I found that pfSense kept em0 as WAN and em1 as LAN from when I set that up in VirtualBox, so the networks didn’t have to be reconfigured. Everything worked seemingly well.
Install flashrom and LCDproc. I only had to change the driver to the Firebox one and the port to parallel; I left all other LCDproc options on the Web Configurator alone.
I know it’s not completely necessary, but I wanted to unlock the BIOS. However, I didn’t want to flash someone else’s random BIOS I found on a forum! I wanted to modify my BIOS with some random tool I found on the internet instead! ;D So, after finding the now ancient AMIBCP 3.51 (The links in this thread are dead; the link I used was this: https://ulozto.net/!PfXQpYPhn/amibcp-3-51-zip ) all I did was change the access level to 3 and enabled the “Always CF Card Boot” menu item in Advanced. I left everything else alone. I couldn’t find where to mess with the Arm/Disarm LED, I wasn’t sure how to enable speedstep (and later posts make it sound like it doesn’t work anyway,) and I thought decompressing modules was a little complicated and didn’t care what it said on the LCD at boot. My Arm/Disarm LED never lit up either before or after BIOS modification. If there’s other things that could/should be enabled, let me know (Steve?). I have attached both the original BIOS and my modified one to this post in a zip file; as usual, use with extreme caution. I flashed my modified BIOS, pulled the battery for a while, and when I put everything back together and booted it up I had full access to the BIOS menus.

MD5SUMs for the very brave:
8eaeb054452c9b8f6ba98d8a5c99ca9f XTM5v2_BIOS.rom
5599976bee52736c37806fbd8a4af9b7 MJR-BIOS.rom

8] The final hiccup, and why I said it almost works: I connected the XTM 5 to my present router for testing. On boot, it will not get an IP address on the WAN interface. I always have to make it try again somehow (via the web configurator refresh button, for example.) Any thoughts? As a stopgap, I was thinking of writing a script and that pings Google DNS, if it fails, make dhclient get a new DHCP lease on WAN, and have the script run as a CRON job every hour or so. Any help would be appreciated.

Thanks,
Matt

EDIT: I can't speel gud

XTM5v2_BIOS_Files.zip

DeLorean

@mredding:

I know it’s not completely necessary, but I wanted to unlock the BIOS. However, I didn’t want to flash someone else’s random BIOS I found on a forum! I wanted to modify my BIOS with some random tool I found on the internet instead! ;D So, after finding the now ancient AMIBCP 3.51 (The links in this thread are dead; the link I used was this: https://ulozto.net/!PfXQpYPhn/amibcp-3-51-zip ) all I did was change the access level to 3….

I'm also searching my way in Bios modding, but where did you see access level 3 in Amibcp 3.51 ?
I only see : Supervisor, User, Extended user and Reserved
Which Windows version did you use for the Amibcp tool?
I have tested it with Windows 7 and don't know if that make a difference in the working of Amibcp.

Thanks in advance.

Grtz
DeLorean

mredding

@DeLorean:

@mredding:

I know it’s not completely necessary, but I wanted to unlock the BIOS. However, I didn’t want to flash someone else’s random BIOS I found on a forum! I wanted to modify my BIOS with some random tool I found on the internet instead! ;D So, after finding the now ancient AMIBCP 3.51 (The links in this thread are dead; the link I used was this: https://ulozto.net/!PfXQpYPhn/amibcp-3-51-zip ) all I did was change the access level to 3….

I'm also searching my way in Bios modding, but where did you see access level 3 in Amibcp 3.51 ?
I only see : Supervisor, User, Extended user and Reserved
Which Windows version did you use for the Amibcp tool?
I have tested it with Windows 7 and don't know if that make a difference in the working of Amibcp.

Thanks in advance.

Grtz
DeLorean

In the Setup Configuration tab, Under Security, there should be a User Access Level option. Under both Failsafe and Optimal, I changed it from 02 to 03. See also https://forum.pfsense.org/index.php?topic=43574.msg262490.html#msg262490

Good Luck,
Matt

EDIT: I think I found where Steve changed the Arm/Disarm LED settings: Under the BootBlock SIO Table, the 27th, 28th, and 29th SIO Registers listed are 30, F0, and F1, and are changed to 01, CF, and 20 respectively. See the new attachment. I think I might try it later today. Not sure how these values correspond with https://forum.pfsense.org/index.php?topic=43574.msg261279.html#msg261279 though.

Assuming this is right, the only thing I haven't figured out is speedstep, which based on this post it sounds like that's a pretty futile endeavor: https://forum.pfsense.org/index.php?topic=43574.msg740652.html#msg740652. Well, that and changing what the BIOS says it is.

EDIT x2: That worked. The Arm/Disarm LED turns red on boot. Now I just have to figure out how to make pfSense turn it green, and get my WAN working on boot without user intervention.

BIOS-Security.png_thumb

BIOS-LED.png_thumb

DeLorean

Thx mredding

Grtz
DeLorean

mredding

I copied https://sites.google.com/site/pfsensefirebox/home/WGXepc64 to /conf/WGXepc and I wrote a little shell script to check if the network is up and change the Arm/Disarm LED accordingly:


/conf/WGXepc -l off > /dev/null 2>&1
sleep 1
until ping -c 1 8.8.8.8 > /dev/null 2>&1; do dhclient em0 && sleep 9; done
/conf/WGXepc -l green > /dev/null 2>&1

My modified BIOS turns the LED red on boot. If the network is up, the script turns the LED off so you know it's doing something, pings Google DNS, and turns it green. If the ping fails, it asks dhclient to do it's job, waits 9 seconds, and then tries to ping Google again. I have this run as a shellcmd script, and you could theoretically run it every hour or something as a cronjob too, and you'd get a little flash of the LED to let you know it checked and its WAN connection is still up. Even though I wrote this for my specific problem, I think it's a pretty good use of the LED.

I still have no idea why I do not get a WAN IP address on boot automatically. The DHCP system logs shown in the Web Configurator show only DHCPd as doing anything on boot; there are no dhclient log messages unless I invoke it manually via my script or by other means like the refresh button. When I watch the boot process via serial console, it hangs on WAN for about 10 seconds before declaring "…done." I'm really not familiar enough with FreeBSD to dig deeper in the logs to see what's going on without some guidance. Any thoughts would be appreciated.

Thanks,
Matt

Billyboy

@mredding:

I thought I would share how I got pfSense 2.4 (mostly) running on an SSD on my XTM 5 version 2 box (initially a XTM 515) since I encountered a few snags along the way that I hadn’t seen brought up in this thread. hour or so. Any help would be appreciated.
….
On boot, it will not get an IP address on the WAN interface.
...

Thanks,
Matt

EDIT: I can't speel gud

Hi Matt, did you try to pull a cable on the 1Gbit ports to see if the disconnection is recognized in the GUI/Dashboard/Interface. We had problems with this on at least 4 boxes (see posts on page 43) in the 2.3.x release. It took up to 10 minutes before PFSense recognized it. This leads to a nearly not working CARP. Maybe the DHCP problem is caused by this as well…

Kind regards

Billyboy

mredding

@Billyboy:

Hi Matt, did you try to pull a cable on the 1Gbit ports to see if the disconnection is recognized in the GUI/Dashboard/Interface. We had problems with this on at least 4 boxes (see posts on page 43) in the 2.3.x release. It took up to 10 minutes before PFSense recognized it. This leads to a nearly not working CARP. Maybe the DHCP problem is caused by this as well…

Kind regards

Billyboy

Thank you, I hadn't put together that those problems may be related. Alas, the Web Configurator shows pfSense detects the cable disconnect almost immediately, and regains a connection after inserting the cable almost just as quickly, so I guess that's not it.

EDIT: Okay, this is weird, but it's working now. I inserted a blank 4GB CF card into the slot and set the SSD as the primary boot device in the BIOS. I formatted the CF card as UFS and set it to mount read only in /etc/fstab. The idea was I thought it was kind of a waste to have an empty card slot, so I would remount it as read-write when I wanted to copy backups to it. This was the only thing I changed. I have no idea why a CF card in the slot would change how the network behaves; it very well could be a coincidence. I'm going to investigate further, but figured I'd share.

By chance, did you experience your problem with XTM 5s that had empty CF card slots?

EDIT x2: I pulled the CF card, and it is still getting an IP address automatically now. No idea how I fixed it.

dlucas46

@747Builder:

@dlucas46:

Hi all,

For those of you with Xeons that would like coretemp to report the correct temp, you can try this recompiled coretemp module.

I have set the TJMax value to 70c

Remove the png extension and upload to /boot/coretemp2.ko

Chmod 755 coretemp2.ko

In your /boot/loader.conf.local add the following:

coretemp2_load="YES"

Reboot.

You should now have a correct temperature reading. I did this several months ago and its been working fine.

If your CPU is in the same family as L5420 this should also work for you.

dlucas46,

Thank you for providing this. could you also provide the source code patch?

I have recompiled for 2.4 against freebsd 11.1.

The code is as follows (lines 213 - 220) coretemp.c :


 else if (cpu_model == 0x17) {
		switch (cpu_stepping) {
		case 0x6:	/* Mobile Core 2 Duo */
			sc->sc_tjmax = 105;
			break;
		default:	/* Unknown stepping */
			break;
		}

You need to change the tjmax value to 70

The reason this error occurs is because the core2duo and the L series Xeons have the same family id (0x6) and the same model (0x17).

The coretemp module really needs to do some more checking and try and identify the cpu by another value that is unique.

coretemp2.ko.png

Billyboy

@mredding:

@Billyboy:

Hi Matt, did you try to pull a cable on the 1Gbit ports to see if the disconnection is recognized in the GUI/Dashboard/Interface. We had problems with this on at least 4 boxes (see posts on page 43) in the 2.3.x release. It took up to 10 minutes before PFSense recognized it. This leads to a nearly not working CARP. Maybe the DHCP problem is caused by this as well…

Kind regards

Billyboy

By chance, did you experience your problem with XTM 5s that had empty CF card slots?

Yes, I experinced the problem with empty CF card slot. Meanwhile I upgrade to 2.40, the cable pull problem disappeared. DHCP on WAN IF (em5 in my case) works for me.

–- But ---

Since the upgrade (from 2.2.6=>2.3.4=>2.40) I have a very high CPU load while there is no traffic on PFSense. Load average always min. 0,50 and more (0.69, 0.49, 1.19) and a CPU load traveling between 20% and 50%, never below 20%.

For heavens sake, I just updated my test system...

DeLorean

@Billyboy:

Since the upgrade (from 2.2.6=>2.3.4=>2.40) I have a very high CPU load while there is no traffic on PFSense. Load average always min. 0,50 and more (0.69, 0.49, 1.19) and a CPU load traveling between 20% and 50%, never below 20%.

For heavens sake, I just updated my test system…

Do a clean install of pfSense 2.4.0 , updating from a older versions gives more chance for problems then a clean install,
sometimes with a upgrade you take the errors from a previous version over.
Version 2.4.0 runs fine on a XTM 5 box, and the previous problem with not detecting correctly the LAN cable status (connected or not) is fixed in 2.4.0

Grtz
DeLorean

vizi0n

Hi!

I have installed an E8400 in my XTM 510 but I do not see the speedstep option in the bios. Should it be supported? I don't see the CPU temperature either on the LCD

A Former User

This post is deleted!