Firebox X550 - Odd network drop outs.


  • I have been running my pfSense install on a Watchguard X550 for a while now.  It was a bit fun to get it configured initially, but its been running ok for a fair while with only a few minor niggles.

    Originally i was running on the production release, however at some point the support for the LCDProc daemon was dropped by default so i had to move to one of the development releases.  I have been running dev releases for about a year now and there has been a recurring issue that occasionally for no apparent reason the WAN interface would drop its connection.  Sometimes just for a short while, and sometimes completely.

    I bastardised a script i found elsewhere that would ping hosts, disable/enable the interface when all pings stopped, and then reboot if this didnt fix it.  This was how i was running for about 6 months.  After a while the internal interface started doing it too for no real reason.  I wrote an alternate version that would do the same for the internal interface, but by the time i figured out how to make this work effectively, the problem seemed to stop.

    Now, the problem seems to have come back with a vengeance.  Both wan and lan interfaces (sk0 and sk3 in my case)  are dropping at random, constantly.  Sometimes it will stay up for a few hours, other times (like right now) its up and down every 30 seconds or less.  (see attached image)

    Note: the script i wrote is currently disabled.

    Any time it goes off permanently i can solve it briefly by just pulling the cables our and re-plugging them back in, however this is, obviously, not a long term solution.

    I figured this issue might be because i was running a dev release, so i upgraded to the latest stable release, but the issue is still there.

    Does anyone have any idea what i need to do to make this work reliably.  I am wondering if interface polling will solve it?

  • Netgate Administrator

    As you found lcdproc is in the stable release now.

    Device Polling is almost universally a terrible idea and has been for a long while now. Though it costs nothing to try it.  ;)

    The most likely cause is that you have components on the board letting go. Most of those boxes will have seen many hundreds of thousands of hours runtime. And even the one that haven't are still built from old components.

    Are you seeing that on all 4 NICs? What if you re-assign them to a different order?

    Since it won't run 2.4 anyway it may be time to upgrade.

    Steve


  • I haven't tried different NICs yet, but that's an interesting shout.  Will give it a go.  As noted i am using 0 and 3 atm, so i can try 1 and 2.

    The box itself seems in very good condition and I know its full heritage since new so its not been mis-treated.  Nothing internally looks amiss so i don't think there's anything bad like faulty caps or anything but i guess that could be a cause.

    Whats the deal with 2.4 tho? Why wont it run on the X550?


  • @whiteknight:

    Whats the deal with 2.4 tho? Why wont it run on the X550?

    In pfSense 2.4, i386 (32-bit x86) is no longer supported, 64-bit hardware is required.

  • Netgate Administrator

    That^

    The X-Core-e boxes are 32bit only. It will be supported for a year after 2.4 is released so there's no real urgency but since you have (potentially) failing hardware and 2.4 is imminent it could be upgrade time.

    Steve


  • Oh ok, that sucks.  The fireboxes are really nice units (assuming they work).

    Are there any similar after-market mods that give a similar "professional" looking result that do have x64 hardware ?  Newer fireboxes etc…

  • Netgate Administrator

    I personally had years of fun with those boxes but everything must end and those I have left are becoming less reliable.

    Hard for me to really recommend anything besides our own hardware  ;) (https://store.pfsense.org/)

    There are newer fireboxes that are 64bit. See: https://doc.pfsense.org/index.php/PfSense_on_Watchguard_Firebox

    Also other old hardware.

    If you are buying new gear though try to get something that supports AES-NI:
    https://www.netgate.com/blog/pfsense-2-5-and-aes-ni.html

    Steve


  • Thanks for that.

    And yes, i appreciate that you cant really offer "off-brand" advice but this is only for my home network so i cant really justify $1800-$3600 on a firewall.  But i do need something with more grunt (and functionality) than the little 2 port unit would offer.  Also something that doesn't sound like a 747 taking off would be great :)

    The XTM 5 looks like it might be a workable solution as the CPU supports 64bit, and they can be had relatively cheaply on ebay.  Obviously this still has the potential for age related issues, but worth a try at least.

  • Netgate Administrator

    I will say that while I have an XTM5 I use for testing and have also had hours of fun with that it cannot run any CPU that supports AES-NI.

    Steve


  • Interesting development.  The Cisco/Linksys router that I decided to fall-back onto as a stop gap solution decided to start doing exactly the same thing.  When i picked it up it was red-hot.  After taking it out of the rack its cooled down and started behaving again.

    Looks like it could simply be a cooling issue.

    We have had particularly warm weather recently, and the rack the kit is stored in can get warm but the Watchguard didn't seem that hot from the temp readouts…. i might try it again out of the rack and see, and try tweaking the fan speed too.

    Also what benefit does AES-NI give me for normal firewall/routing/filtering duties?  I thought that was just for encryption and vpn?  Is the lack of that a show-stopper?  Or will VPN still be available, just slower or less secure?

  • Netgate Administrator

    It has been warm this week in the UK (relatively  ;)). If you have the fan speed turned down that could be it. Watchguard had the fans at max all the time. The CPU is directly cooled but the average airflow through the box is what keeps everything else cool, there may well be some hot spots.

    Lack of AES-NI will likely be a show stopper. You should assume 2.5 will not run on anything (x86) that doesn't support it. Again we will be supporting 2.4 for sometime after that though.
    I won't go any deeper than that here, there are a number of other threads discussing it.

    Steve