PfSense on a Riverbed Steelhead



  • @Okijames I've recently acquired a CX-770 too. Because you actually own a working one, I need to ask you some things. I suspect that mine might have a hardware issue. I'm trying to alleviate things that I might be doing wrong. When your is first powered on, what color are the System, HDD0, and HDD1 LED's? Does yours contain a LOM module that's mounted under the 320GB drive?
    Final question, are you able to read your BIOS into an image that I might be able to flash? Thanks in advance...



  • @cjohnson Happy to help...

    HDD LED colors: Orange at power-on, then blue before POST completes

    DOM module: No module on my 570 or 770, RIOS boots from the HDD

    Read the BIOS: Sure thing, what tool should I use? FWIW, my BIOS is version 2.15.1236, copyright 2013. Does your BIOS not offer nic bypass control?



  • @Okijames
    I'm trying to diagnose what I believe to be a partially functionally borked BIOS in it.

    LED COLORS: I just want to make sure that I understand fully, all three are orange at power on?
    What about if you enter BIOS and just let it sit there? Do they remain orange?

    ***L***OM not DOM. Lights Out Management. It's a separate modular board that plugs into the mainboard, it's mounted underneath the removable tray that holds the SSD and HDD. You can just see the edge of it, (if it's there) just under the side of the tray where the SSD is mounted. It has a webserver that should be available via the "PRI" port. I tcpdumped the port and looked at the source IP attached to the GARP's it was sending since I didn't know how the port was setup (DHCP vs Static IP etc.).

    If you reset the BIOS by pulling the CMOS battery (without the main PSU connected) for 30-ish seconds, then reinstall it; does the serial console (assuming 115200,8n1 config) immediately start showing you stuff on a terminal shortly after you power it on?

    As far as capturing the BIOS, I believe that's probably something "flashrom" could do, but I'm not certain exactly if; and/or how you'd do that. Maybe @stephenw10 knows since he's done stuff like this before with Watchdog hardware I think? link text



  • @cjohnson

    System LED: Red at power-on, Orange at the end of POST just prior to boot (~60sec from power-on). IIRC Blue very late in the boot process, after RIOS is fully up and running. Under pfSense it stays Orange.

    HDD LEDs: Orange at power-on, Blue during POST (~40sec from power-on)

    LOM: Oops, thought you mistyped. Yes the LOM is there. AFAIK it only offers cli/text via a client using ipmitool, not a web interface. RIOS cli "remote" commands are used to set the IP. See the Riverbed CLI user guide for details.

    Behavior after BIOS reset: Defaults to 9600,8,n,1 BTW which aligns with RIOS. ~60sec to show first text via serial console, coincides with an audible beep, offering Delete or F2 to enter BIOS setup. Note if you enter BIOS setup, the System LED remains Red rather than turning Orange.



  • Ahh well crap. It looks like a I've got a hosed up BIOS then. I get nothing via the serial port, nor do I get any POST beeps. I don't even get any beeps/tones if i power it up with no RAM plugged in. On the upside, the LOM card seems to be working though.

    I'm certain that the LOM card does offer a webUI. I was hopeful that it had remote KVM capabilities, but it doesn't. You can control power functions as well as look at fan speeds and stuff within it. Assuming that it has a static IP asigned, you can find out what IP it has by tcpdumping the traffic from the "PRI" port after power on. About 20 seconds or so after power on, you'll see GARP's coming from that port. If you point a web browser at the source IP contained within those GARP's, you'll get a login prompt. The OEM (advantech) default credentials for the webUI are admin/admin. On the other hand if it's setup for DHCP, you can check your DHCP server to see what IP it handed out. Obviously, this is a security risk depending on if you're using this interface or not, what network segment it's attached to, if the default creds are still set.....etc.... etc...

    Anyhow, I'll keep an eye on this thread to see if someone comes up with an idea on how to dump the stock BIOS. That's only part of the problem though, even if I get a BIOS dump, I have to come up with a way to program it. Maybe I'll have to get on of those USB attached SOIC-8 socketed flash tools?
    I'm not certain just yet, but it looks like the BIOS ROM is socketed on these things. So, that'll make surgery SLIGHTLY easier down the line.
    If flashrom turns out to be a viable option, maybe I'll get my hand on another (fully working) one of these boxes, boot it up off of a liveUSB, hot-swap my ROM into the socket while it's running, then flash it with an image from you. I found this link Fingers crossed......


  • Netgate Administrator

    @Okijames said in PfSense on a Riverbed Steelhead:

    Under pfSense it stays Orange.

    Game on! 😉

    Flashrom can probably detect and read that ROM file from an older device like that. Running it is always some risk though.

    Steve



  • @cjohnson Why do you think it's a BIOS problem vs any number of other reasons it could be dead?

    PS, I see a pair 770 on ebay for $99ea. Might be more time/cost effective to buy one or both.



  • Warning: Whiskey and tinkering don't always mix. I have now overwritten the RIOS boot HDD with FreeBSD. Yippee Ki Yay!



  • Why do you think it's a BIOS problem vs any number of other reasons it could be dead?
    At the time mostly, because of the LED/fan behavior; and a gut feeling based upon experience.

    PS, I see a pair 770 on ebay for $99ea. Might be more time/cost effective to buy one or both.
    This is exactly what I had done already. The second one wasn't here yet and I was working with what I had on hand at the time. The second box has since arrived. I swapped the BIOS ROM over from the working one and it booted right up to RiOS. After that, I shut it down and booted it back up on an ubuntu usb. Flashrom dumped the good ROM to a file; hot swapped in the bad one and dumped that to a file as well.

    I'm gonna' open 'em both up in a hex editor later and see what the difference(s) between the two is/are before I boot it up again and hot swap/re-flash the bad one with the image from the good one.

    I might do some digging around in the RiOS drive while I'm in there flashin' ROMs and see what sorta' goodies I can find related to LED color/controls and bypass relay controls too.



  • @cjohnson Good to hear there's hope for both units. FWIW I overwrote my RIOS drive in pursuit of a BIOS dump. Success on the BIOS dump (adios RIOS) and happy to send your way if you want another working example.



  • for anyone who's wondering....just another fine point of detail about the bypass functionality of the four network interfaces (LAN 0_0, WAN 0_0, LAN 0_1, WAN 0_1) on the right hand side (while viewed from the front) of the box.

    1. set either/both pairs of those NICs to "no bypass" in BIOS (they're labeled as NIC 3&4, NIC 5&6 in BIOS)
    2. save the changes and exit BIOS
    3. put the box in standby state (hold the front power switch down for a few seconds, before POST completes)
    4. the link status LED (not the activity LED) of the NIC pair you've set to "no bypass" will now be lit up amber

    BIOS label "NIC 3&4"
    pins 4&5 of LAN 0_0 will be connected to pins 4&5 of WAN 0_0
    pins 7&8 of LAN 0_0 will be connected to pins 7&8 of WAN 0_0

    BIOS label "NIC 5&6"
    pins 4&5 of LAN 0_1 will be connected to pins 4&5 of WAN 0_1
    pins 7&8 of LAN 0_1 will be connected to pins 7&8 of WAN 0_1

    this essentially means that with the box in this "standby" state, you'll be passing only PoE (not ethernet data) to/from LAN 0_0 to/from WAN 0_0 and, to/from LAN 0_1 to/from WAN 0_1. as soon as you power it up, the PoE shunt is opened.


  • Netgate Administrator

    Hmm, that seems odd. I might have expected the other two pairs to be linked for 100M pass-through. PoE pass-through that gets interrupted seems pretty much pointless!

    Steve



  • Thanks for all your guidance. I thought I'd followed it , but my CX-755 is giving an error:

    smbmsg: Error performing SMBus IO: Device not configured
    

    Any ideas?
    Although I'm learning as I go, apparently not quickly enough. Thx.



  • @sambiggs Try loading ichsmb and then smb manually. Post the output of these commands...

    kldload ichsmb
    kldload smb



  • I'd put those into the config.xml, right before the smbmsg commands. This is copied from the PuTTY output:

    ichsmb0: <Intel 631xESB/6321ESB (ESB2) SMBus controller> port 0x540-0x55f irq 19 at device 31.3 on pci0
    smbus0: <System Management Bus> on ichsmb0
    smb0: <SMBus generic I/O> on smbus0
    smbmsg: Error performing SMBus IO: Device not configured
    

  • Netgate Administrator

    Had you tried scanning the smbus at all? That can easy lock up requiring a reboot.

    Did you try entering all the commands manually first without putting anything in the config file?

    Steve



  • Sequence was:

    1st:

    • Drop to shell and add the following to /boot/loader.conf.local to load the smbus drivers
    • ichsmb_load=“YES”
    • smb_load=“YES”
    • add "smbmsg -s 0x48 -c 0x55 -o 6 0x03 0xfc 0x01 0xfe 0x66 0x99" to config.xml

    Result

    • Perhaps there was a "satisfying click" and I was then able to get a link light on em0, but it is not usable in pfSense.
    • em1-3 show no link lights

    2nd:

    • remove the added lines from /boot/loader.conf.local
    • add "kldload ichsmb" and "kldload smb" to config.xml

    Result

    • No click, and no noticeable difference. The error shown above appeared in the console output

    I haven't knowingly tried scanning the smbus, as I am still dumbly following your previous posts, and either that wasn't there, or it was buried in the stuff about your 1050 model, which I avoided in the cause of simplicity.

    I haven't properly understood the sideline about LED colour, but fwiw I get a single steady orange.



  • I still have the original RiOS drives with all the shell scripts and pythons scripts and all that stuff in it. I've been trying to reverse engineer it to figure out how it operates the status LED. If someone wants a look at it, I could probably send a few files your way....



  • @sambiggs Probably best to ignore the messages related to the 1050, it's a very different box. Refer instead to my "soup-to-nuts" post from Nov 29th, with a few modifications because your CX-755 has 2 pair of bypass NICs vs the single pair on my CX-550...

    The NIC numbering will be different, as a guess yours might look like this.

    em0 = LAN0_0
    em1 = WAN0_0
    em2 = LAN0_1
    em3 = WAN0_1
    em4 = Primary
    em5 = Aux

    You will need an additional smbmsg line in config.xml to enable the second pair of bypass ports...

    <shellcmd>smbmsg -s 0x48 -c 0x55 -o 6 0x03 0xfc 0x00 0xff 0x66 0x99</shellcmd>
    

    As Stephen said, the smbus is touchy. I'd recommend removing the loader.conf and config.xml edits. Power cycle the box. Then see what happens when you manually enter the commands...

    kldload ichsmb
    kldload smb

    smbmsg -s 0x48 -c 0x55 -o 6 0x03 0xfc 0x01 0xfe 0x66 0x99
    smbmsg -s 0x48 -c 0x55 -o 6 0x03 0xfc 0x00 0xff 0x66 0x99



  • OK, cleared those and ran commands manually.
    No apparent error messages

    Enter an option: 8
    
    [2.3.5-RELEASE][root@pfSense.localdomain]/root: kldload ichsmb
    ichsmb0: <Intel 631xESB/6321ESB (ESB2) SMBus controller> port 0x540-0x55f irq 19 at device 31.3 on pci0
    smbus0: <System Management Bus> on ichsmb0
    [2.3.5-RELEASE][root@pfSense.localdomain]/root: kldload smb
    smb0: <SMBus generic I/O> on smbus0
    [2.3.5-RELEASE][root@pfSense.localdomain]/root: smbmsg -s 0x48 -c 0x55 -o 6 0x03 0xfc 0x01 0xfe 0x66 0x99
    [2.3.5-RELEASE][root@pfSense.localdomain]/root: smbmsg -s 0x48 -c 0x55 -o 6 0x03 0xfc 0x00 0xff 0x66 0x99
    [2.3.5-RELEASE][root@pfSense.localdomain]/root:
    

    HOWEVER
    After testing a little more thoroughly, the ports are not arranged as I had assumed (Yes, I know, "ass out of U & ME") but are em2 em3 em0 em1 em4 em5. When I tested previously, I may have been looking for connection on a port that was disabled

    They are all working
    ...but it may be that they were working before. I cannot be sure because of my poor testing.

    Thanks for assistance, and for taking the trouble to share your efforts


Log in to reply