PfSense on a Riverbed Steelhead



  • Thanks @johnpoz!

    Yeah, 3-4x on the power consumption. Fans, though pretty quiet, are 100% louder than the fanless APU2.

    Thinking I'll bump up the RAM and disk capacity and use it to experiment with various Container platforms.


  • LAYER 8 Global Moderator

    Is it a SD770? From quick look those are about 50W idle - so more like 5-6x your apu2, and noise looks like about 45dba.. While not all that bad - sure isn't quiet..



  • CX-770 currently idling at ~27W, not bad. And honestly the fans are pretty quiet, faint background noise sitting right next to me on a desk. Might be near silent if I replaced them with some nice Noctuas.

    Under load though, I'm sure it wouldn't be quite so pleasant.



  • 2 Noctua NF-A4x20 PWM ordered. :)


  • Netgate Administrator

    @stephenw10 said in PfSense on a Riverbed Steelhead:

    I would guess ~30W

    @Okijames said in PfSense on a Riverbed Steelhead:

    currently idling at ~27W

    Ha. 😁



  • @pauloalb

    I have a Riverbed 1050L with 2 InPath or 4Gbe NICs and am trying to enable them for PfSense, except the I2C Layout looks a bit different. I have been probing with i2ctools.

    [root@localhost~]# i2cdetect -l
    i2c-3 i2c nvkm-0000:08:00.0-bus-0005 I2C adapter
    i2c-10 i2c nvkm-0000:08:00.0-aux-000c I2C adapter
    i2c-1 i2c nvkm-0000:08:00.0-bus-0001 I2C adapter
    i2c-8 i2c nvkm-0000:08:00.0-aux-000a I2C adapter
    i2c-6 i2c nvkm-0000:08:00.0-bus-0008 I2C adapter
    i2c-4 i2c nvkm-0000:08:00.0-bus-0006 I2C adapter
    i2c-11 i2c nvkm-0000:08:00.0-aux-000d I2C adapter
    i2c-2 i2c nvkm-0000:08:00.0-bus-0002 I2C adapter
    i2c-0 i2c nvkm-0000:08:00.0-bus-0000 I2C adapter
    i2c-9 i2c nvkm-0000:08:00.0-aux-000b I2C adapter
    i2c-7 i2c nvkm-0000:08:00.0-bus-0009 I2C adapter
    i2c-5 i2c nvkm-0000:08:00.0-bus-0007 I2C adapter
    i2c-12 smbus SMBus I801 adapter at 0400 SMBus adapter <<---- I am assuming it is i2c-12 or the SMBUS

    However I found several slave addresses:

    [root@localhost ~]# i2cdetect -y 12
    0 1 2 3 4 5 6 7 8 9 a b c d e f
    00: -- -- -- -- -- 08 -- -- -- -- -- -- --
    10: -- -- -- -- -- -- -- -- 18 19 1a 1b 1c 1d -- 1f
    20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 2f
    30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
    40: -- -- -- -- 44 -- -- -- 48 49 4a 4b 4c 4d 4e 4f
    50: 50 -- 52 -- 54 55 56 57 58 -- -- -- -- -- -- --
    60: 60 61 -- -- -- -- -- -- -- 69 -- -- -- -- -- --
    70: -- -- -- -- -- -- -- --
    [root@localhost ~]#

    No 0x24, but 0x48 appears with the following:

    [root@localhost ~]# i2cdump -y 12 0x48
    No size specified (using byte-data access)
    0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
    00: 27 00 4b 50 XX XX XX XX XX XX XX XX XX XX XX XX '.KPXXXXXXXXXXXX
    10: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    20: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    30: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    40: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    50: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    60: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    70: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    80: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    90: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    a0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    b0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    c0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    d0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    e0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX
    f0: XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XX XXXXXXXXXXXXXXXX

    From the command:
    smbmsg -s 0x48 -c 0x55 -06 0x03 0xfc 0x01 0xfe 0x66 0x99
    I gather it means write 6 Bytes to SMBUS address 0x48 with initial command byte of 0x55 the data "0x03 0xfc 0x01 0xfe 0x66 0x99"

    But in my case or for the RB1050L, it looks like 0x48 is only 4Bytes wide, and the command sends anywhere from 6Bytes for "Universal Mode" and 8Bytes for "Bypass/NoLine"

    Can anyone that has this working please provide a dump of their 0x48 of their SMBUS so we can see what data should be there? Will the data we write show up at that address location? I don't feel comfortable writing things to random addresses that can potentially damage something.



  • For the add-on cards you're probably best off using native drivers from the manufacturer. Silicom was a popular supplier, and they offer FreeBSD drivers, see if you can find your card here...

    https://www.silicom-usa.com/cats/server-adapters/networking-bypass-adapters/gigabit-ethernet-bypass-networking-server-adapters/



  • @Okijames The InPath NICs in the Riverbed 1050 which I have, are actually embedded into the motherboard.



  • @freska99 Oops, sorry memory isn't what it used to be. :)

    So the problem from here is multifold. We would need to know several things, some easier than others...

    -What CPU chipset / smbus controller is in use? Clues in dmesg and/or physically looking at the chips

    -What's the command structure for said chipset? Find Intel docs

    -Find the correct smbus address for bypass control. As you've done above, it is a moderate amount of poking/guessing.

    -Guess the correct byte sequence to flip the relays. This seems pretty difficult! The numbers for the 250/550 were taken from code in another project. I have no idea how the original author figured out the correct byte sequence. Guessing he had inside knowledge.



  • FWIW I'm assuming you tried looking for settings in the BIOS. Wishful thinking, but the 570/770 do have such BIOS settings so maybe...

    Or setting the nics to stop failing to bypass in RIOS?

    no interface <interface-name> fail-to-bypass enable



  • Whatever you do, please find and read the chipset docs before poking around with the smb read or write commands. I somehow permanently set my 550's 1.66GHz CPU to 1.33GHz. No telling what else I could have screwed up if, for example, I randomly guessed byte sequences :)



  • @Okijames Thanks for the quick reply:

    The board looks like a modified Tyan server board running with Intel ICHx SMBUS controller with Intel Xeon 2Core 1.88GHz, I was able to read from the SMBUS using ICHx drivers in both Linux and under RWEverything in Windows (although in Windows the address is 2x or double the HEX value given to the i2cdump command for some odd reason)

    Yes I already tried looking in the BIOs there are only options to enable / disable the PRI/AUX Interfaces. The command 'no interface <interface> fail-to-bypass enable' has also been tried. All it does is causes the ports to go into "BLOCK MODE" so it will neither bypass nor work (under the OS it shows as Ethernet Disconnected). The BYP/BLK LED is turned on. Bypass causes the same issue with Ethernet Cable Disconnected except the relays form a physical path from one physical port to the other.

    The real question I am trying to answer is what is the address of the microcontroller driving the relays. I am thinking that there might be a "signature" or "fingerprint" if someone dumped the contents of address 0x48 from the 250/550, that the contents might be similar to what I have on the 1050 so I can co-relate the correct address.

    I will try taking a look inside for what specific chips are being used tomorrow. I am quite paranoid about writing anything to SMBUS especially when hardware models/revisions do not match the python script or what others are discussing, so more research is needed.


  • Netgate Administrator

    One of the problem you have here is that a lot of smbus devices are auto-addressing. The controller we were looking at appears at a different address in FreeBSD than it does in Linux. Your may well also.

    Steve



  • @freska99 The only process I can think if is laborious but that only makes your eventual success that much sweeter right?

    -1 RTFM: Get the chipset docs and determine the proper commands for READING the smbus settings. Note that first and subsequent read operations may return different results. The smbus is touchy! The read operations may actually cause changes, though I think they are temporary.
    I hope not, but you might literally have to reboot after reading each address.

    -2 Get a baseline: Scan and READ your smbus infra under Linux and FreeBSD. Document and compare the (probably different) addresses, current settings etc. Do your best to "align" the addresses, accounting for OS and driver differences. Linux seems to have better tools, but you'll eventually have to translate to BSD for pfSense. Note: I am assuming that both of the inpathx_y are setup identically with regard to their "fail-to-bypass" setting. If not, set them identically and redo this step.

    -3 First test: Boot back into RIOS and disable link state propagation with "no in-path lsp enable". Why? First you don't want LSP anyway, read the support docs if curious. Second, it might make 2 or 4 of your smbus addresses disappear. On my 550 it made 0x64 and 0x66 disappear. Fewer devices to scan and probe. Unfortunately it also caused me to guess that 0x60 (the next closest address seems logical right?) was the relay control address, but NOOO, it was ) 0X48. I issued the smb write command to 0x60, and 0x5c, resulting in a seemingly permanent slower clock rate for my CPU. :(
    Anyway... Boot back to Linux/BSD, take note of the address disappearance. This should help with the whole "aligning" of addresses under the different OSs.

    -4 Setup a diff: Luckily you have a 4port on-board bypass nic, so 2 sets of" inpathx_y" port pairs under RIOS. Boot back to RIOS and setup inpaths with opposite configs, for example...

    interface inpath0_0 fail-to-bypass enable
    no interface inpath0_1 fail-to-bypass enable
    write mem

    -5 Scan for diffs: Boot back to Linux/BSD. Hopefully you will find that register settings under one of the smbus addresses has changed. Make note of the register settings that appear to equate to the different bypass states.
    You may want to try one more diff with inpath interfaces enabled and disabled with...

    in-path interface inpathx_y enable
    no in-path interface inpathx_y enable
    write mem

    -6 Apply your discovered register settings under BSD. Success?

    Not 100% sure but I think the "normal" nic behavior you want under pfSense equates to the following RIOS settings...

    in-path interface inpathx_y enable (enables the nics)
    no interface inpathx_y fail-to-bypass enable (disables fail-to-wire between ports)
    no in-path lsp enable (does not propagate link down state to between ports)

    The nics might then need to be configured and "up"ed under pfSense. You may need to drop to shell to do this.



  • @Okijames I have started looking into the manuals regarding the chips I found inside the box and doing some of your suggestions. Of interest are the following:

    • The InPath NICs have 2 Intel controllers under heatsink, each controllers handles two ports

    • Next to each of these controllers are ATMEL 932 25160AN EEPROMs using being addressed thru SPI Serial Interface

    • (The Primary and AUX NICs also have one controller each but they use ATMEL 952 25640AN EEPROMs instead)

    • Found an I2C / SMBUS TMP75 Temperature Sensor, looking into this to narrow down addresses based in its output

    • Found Three of these Chips: PCA9556 Octal SMBUS & i2C Registered Interface

    • Found PA9516A NXP 5-Channel I2C Bus Hub

    • Found two unknown chips on either sides of and right next to the relays labelled "4965 AD .W03B"

    • Didn't find anything that is suggestive of a Microcontroller like the nxp lpc932a1fa located in other models

    Also was looking into the code under RIOs and it looks like a lot of it is written in Python. Came across one interesting file called "hw_ctl.py". It states sequence for bypass as:

    elif val == 'bypass':
        set_val(turn_on(curr, range(7)), 0x32)
        set_val(turn_off(curr, 6), 0x32)
      set_val(turn_on(curr, 6), 0x32)
        curr = get_curr(0x32)
        set_val(0xff, 0x30)
        set_val(rest, 0x30)
    

    Function of: set_val(new_value, register_to_read_or_write_to)

    To Read the registers:

    • curr = popen("%s %s 0x1 0x0 2> /dev/null" % (ipmi_cmd, reg)).readline()[:-1]
    • There there is a comment that if "curr" is empty the message "could not talk to i2c device" is displayed

    To set the registers with a sequence for bypass, disconnect, normal etc..

    • popen("%s %s 0x0 0x1 %s 2> /dev/null" % (ipmi_cmd, reg, hex(new)))

    The reg variable is named "rest = 0x30" and "curr=0x32" not sure how they differ

    The popen function just opens a tool, in this case "impitool", the ipmi_cmd variable is:
    /sbin/ipmitool raw 0x6 0x52 0x7

    So to read the value of the NICs i think it would be
    ipmitool raw 0x6 0x52 0x7 0x32 0x1 0x0 (curr register)
    ipmitool raw 0x6 0x52 0x7 0x30 0x1 0x0 (rest register)

    The syntax would be:
    ipmitool raw 0x6 0x52 0x7 <register> <read=0x1 0x0 | write=0x0 0x1> [new value if writing]

    I will next try to break into the shell of RIOS and test these commands and scripts. The device does have an IPMI IP Addresses so I might even able to set the states thru the network. We will see.


  • Netgate Administrator

    Looks promising.



  • Definitely promising!

    Terrible I didn't think to ask earlier but... Have you tried the rbmode script from github?

    Modify it to print out the calculated values as stephenw10 did several posts back.

    Also regarding the chipset... Under BSD the output of "kldload ichsmb" is helpful. On my 770 this is what I get...

    kldload ichsmb
    ichsmb0: <Intel DH89xxCC SMBus controller> port 0xf000-0xf01f mem 0xfe704000-0xfe7040ff irq 18 at device 31.3 on pci0

    I found the docs for this chipset here...
    https://www.mouser.com/datasheet/2/612/intel-communications-chipset-89xx-series-datasheet-257583.pdf

    Page 176 shows the command encoding syntax. Thankfully the syntax is the same at the 631xesb chipset used in the 550s, so it is likely the 1050 chipset does the same.



  • @Okijames @stephenw10

    Good news: I figured it out! Seems like there are tons of helpful information stored within the python scripts on the RiOS OS drive!

    Note: The following is only valid for the Riverbed 1050X Series or Motherboard P/N 400-00100-01 / Tyan S6631

    Here is what I can confirm:

    • The commands to enable the NICs are sent thru IPMI using the "raw" data mode i.e. ipmitool raw
    • The register via ipmi to trigger relays is 0x30
    • The registers accessed via IPMI corelates with SMBUS Addresses

    According to information gleaned from hw_ctl.py and hw_ctl.xml, the different bits of Register 0x30 controls different settings for each set of inpath NICs. hw_ctl.xml has all the definitions for the different models of SteelHeads and what commands needs to be sent to the relays to enable the NIC, among other things like which SMBUS address/register and what data to send to enable LEDs. Seems like the bigger 1U+ Tyan Based Mobos use IPMI to communicate with SMBUS to drive the relays instead of doing it directly. hw_ctl.py contains classes and methods with the actual commands; It looks up values from hw_ctl.xml in order to determine the correct sequences according to model retrieved from SMBIOS.

    Bits Syntax: *[rr aaa bbb]; where rr=reserved bits; aaa=controls LAN1 & LAN2; bbb=controls LAN3 & LAN4
    For rr - Set it to 00 when issuing commands, rr will automatically reset to 01 when system is restarted
    InPath0:

    • Normal Mode L1 & L2: rraaa000
    • Disconnect L1: rraaa001
    • Disconnect L2: rraaa010
    • Disconnect L1+L2: rraaa011
    • Bypass L1 to L2: rraaa111

    InPath1:

    • Normal Mode L3 & L4: rr000bbb
    • Disconnect L3: rr001bbb
    • Disconnect L4: rr010bbb
    • Disconnect L3+L4: rr011bbb
    • Bypass L2 to L3: rr111bbb

    IPMI Syntax:
    IPMI Can be issued either locally or thru the network, I have tested both and it works

    • To Read: ipmitool [-H <ip address> -U Admin -P <pw>] <raw 0x6 0x52 0x7><register><0x1 0x0>
    • To Write: ipmitool [-H <ip address> -U Admin -P <pw>] <raw 0x6 0x52 0x7> <register> <0x0 0x1><val. to write>

    Thus to enable all the NICs:

    • First read and take note of register 0x30 for safety: ipmitool raw 0x6 0x52 0x7 0x30 0x1 0x0
    • (This should returns 0x7F or r=01 aaa=111 bbb=111 after a reboot)
    • Write 0x00 to Register 0x30: ipmitool raw 0x6 0x52 0x7 0x30 0x0 0x1 0x00
    • After hitting enter, you should hear a click and see the NICs come online!

    Under RedHat Linux It seems Register 0x30 is reflected under smbus or i2c-12 device 0x18

    Just After A Reboot: 0x30 = 7F, we see the repeating sequence under SMBUS 12 0x18
    [root@localhost bin]# i2cdump -y 12 0x18
    No size specified (using byte-data access)
    0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
    00: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    10: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    20: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    30: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    40: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    50: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    60: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    70: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    80: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    90: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    a0: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    b0: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    c0: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    d0: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    e0: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..
    f0: 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 7f 7f 00 00 ??..??..??..??..

    After enabling the NICs by setting IPMI Register 0x30 to 0x00, SMBUS12 0x18 becomes:

    [root@localhost bin]# i2cdump -y 12 0x18
    No size specified (using byte-data access)
    0 1 2 3 4 5 6 7 8 9 a b c d e f 0123456789abcdef
    00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
    f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

    Some Other Registers that looks related, but I cannot confirm since I have not probed them deeply as I already accomplished my goal. The values taken from SMBUS from these other registers also exhibit the repeating pattern as seen between 0x30 and 0x18:

    IPMI Register <> SMBUS Linux Register
    0x30 <> 0x18 - Relay Control
    0x32 <> 0x19* - Possibly watchdog control, bits stores value whether to "bypass" or "disconnect"
    0x34 <> 0x1A - Becomes 0xFF when BIOs control is handed to OS, otherwise 0F
    0x3A <> 0x1D - Becomes 0xFF after NICs enters NORMAL, otherwise 0xED

    *0x32<>0x19 - inpath0 bypass inpath1 disconnect: 0xE6
    *0x32<>0x19 - inpath0 bypass inpath1 bypass: 0xFE
    *0x32<>0x19 - When RiOS Optimisation Service is Running: 0xFD

    *Edit - minor correction to normal mode l3&l4 *



  • @Okijames

    I have looked at the RbMode script but I didn't dare try it as the registers it is writing to doesn't match up. Now I can confirm that the RbMode will not work with the 1050 as Riverbed's own code suggest this certain model uses ipmi to communicate with the SMBUS to trip the relays as opposed to doing it directly. The corresponding register looks like 0x18 as opposed to 0x24 in the script and dump 0x18 has this weird repeating pattern of the value you set many many times as you see in my last post, so if you were to write to smbus directly, are you supposed to set all 255 registers or something else? Maybe that is what IPMI is doing in the background but I am not ready to risk it to find out. Also the script suggest in the model it was written for the relays are controlled via several bytes, but here a single byte sent to ipmi address 0x30 controls all 4 Ports depending on which bits are set or not.

    The original RIOS also suggest there is a SYSFS implementation too, that you can simply echo bypass > /sys/<path to nic> to set the states, so I am guessing the sysfs kernel module is communicates with ipmi which communicates with smbus.

    Original Implementation is something along the lines of:
    <sysfs -> ipmi -> smbus -> relay>
    <python script -> ipmi -> smbus -> relay>

    Very satisfied I figured it out XD



  • For those using ipmiutil (Linux/Windows) the commands are:

    -Read: ipmiutil cmd [-H <ip> -U <username> -P <pw>] 00 20 18 52 07 30 01 00
    -Write: ipmiutil cmd [-H <ip> -U <username> -P <pw>] 00 20 18 52 07 30 00 01 00

    Bus: 00
    RSSN: 20
    *NetFN&Lun: 18
    Command: 52
    Data1: 30 01 00 (Read from Register 0x30)
    Data2: 30 00 01 00 (Write 0x00 to Register 0x30)

    *Just thought I mentioned it because ipmitool's syntax is ipmitool raw <netfn> <command> <data>, while ipmiutil's is not just netfn but netfn AND LUN combined: nnnnnnll where n's are the 6bits from the netfn and ll represents the LUN which is 0. Rearranging and adding the LUN to netfn 0x6 we get 0x18. I first didn't realise and entered 06 and it didn't work.

    Enjoy your now fully functional repurposed Riverbed!



  • @freska99 Excellent, congrats! Mind posting the contents of the two hw_ctl files so future folks with other models have a roadmap?



  • @Okijames hw_ctl.zip

    Thanks =) its nice hearing those relays click in! Ive zipped up a few files that others might find useful hw_ctl.py, hw_ctl.xml, hal.py, hal_common.sh, hal is the hardware abstraction layer.



  • @Okijames I've recently acquired a CX-770 too. Because you actually own a working one, I need to ask you some things. I suspect that mine might have a hardware issue. I'm trying to alleviate things that I might be doing wrong. When your is first powered on, what color are the System, HDD0, and HDD1 LED's? Does yours contain a LOM module that's mounted under the 320GB drive?
    Final question, are you able to read your BIOS into an image that I might be able to flash? Thanks in advance...



  • @cjohnson Happy to help...

    HDD LED colors: Orange at power-on, then blue before POST completes

    DOM module: No module on my 570 or 770, RIOS boots from the HDD

    Read the BIOS: Sure thing, what tool should I use? FWIW, my BIOS is version 2.15.1236, copyright 2013. Does your BIOS not offer nic bypass control?



  • @Okijames
    I'm trying to diagnose what I believe to be a partially functionally borked BIOS in it.

    LED COLORS: I just want to make sure that I understand fully, all three are orange at power on?
    What about if you enter BIOS and just let it sit there? Do they remain orange?

    ***L***OM not DOM. Lights Out Management. It's a separate modular board that plugs into the mainboard, it's mounted underneath the removable tray that holds the SSD and HDD. You can just see the edge of it, (if it's there) just under the side of the tray where the SSD is mounted. It has a webserver that should be available via the "PRI" port. I tcpdumped the port and looked at the source IP attached to the GARP's it was sending since I didn't know how the port was setup (DHCP vs Static IP etc.).

    If you reset the BIOS by pulling the CMOS battery (without the main PSU connected) for 30-ish seconds, then reinstall it; does the serial console (assuming 115200,8n1 config) immediately start showing you stuff on a terminal shortly after you power it on?

    As far as capturing the BIOS, I believe that's probably something "flashrom" could do, but I'm not certain exactly if; and/or how you'd do that. Maybe @stephenw10 knows since he's done stuff like this before with Watchdog hardware I think? link text



  • @cjohnson

    System LED: Red at power-on, Orange at the end of POST just prior to boot (~60sec from power-on). IIRC Blue very late in the boot process, after RIOS is fully up and running. Under pfSense it stays Orange.

    HDD LEDs: Orange at power-on, Blue during POST (~40sec from power-on)

    LOM: Oops, thought you mistyped. Yes the LOM is there. AFAIK it only offers cli/text via a client using ipmitool, not a web interface. RIOS cli "remote" commands are used to set the IP. See the Riverbed CLI user guide for details.

    Behavior after BIOS reset: Defaults to 9600,8,n,1 BTW which aligns with RIOS. ~60sec to show first text via serial console, coincides with an audible beep, offering Delete or F2 to enter BIOS setup. Note if you enter BIOS setup, the System LED remains Red rather than turning Orange.



  • Ahh well crap. It looks like a I've got a hosed up BIOS then. I get nothing via the serial port, nor do I get any POST beeps. I don't even get any beeps/tones if i power it up with no RAM plugged in. On the upside, the LOM card seems to be working though.

    I'm certain that the LOM card does offer a webUI. I was hopeful that it had remote KVM capabilities, but it doesn't. You can control power functions as well as look at fan speeds and stuff within it. Assuming that it has a static IP asigned, you can find out what IP it has by tcpdumping the traffic from the "PRI" port after power on. About 20 seconds or so after power on, you'll see GARP's coming from that port. If you point a web browser at the source IP contained within those GARP's, you'll get a login prompt. The OEM (advantech) default credentials for the webUI are admin/admin. On the other hand if it's setup for DHCP, you can check your DHCP server to see what IP it handed out. Obviously, this is a security risk depending on if you're using this interface or not, what network segment it's attached to, if the default creds are still set.....etc.... etc...

    Anyhow, I'll keep an eye on this thread to see if someone comes up with an idea on how to dump the stock BIOS. That's only part of the problem though, even if I get a BIOS dump, I have to come up with a way to program it. Maybe I'll have to get on of those USB attached SOIC-8 socketed flash tools?
    I'm not certain just yet, but it looks like the BIOS ROM is socketed on these things. So, that'll make surgery SLIGHTLY easier down the line.
    If flashrom turns out to be a viable option, maybe I'll get my hand on another (fully working) one of these boxes, boot it up off of a liveUSB, hot-swap my ROM into the socket while it's running, then flash it with an image from you. I found this link Fingers crossed......


  • Netgate Administrator

    @Okijames said in PfSense on a Riverbed Steelhead:

    Under pfSense it stays Orange.

    Game on! 😉

    Flashrom can probably detect and read that ROM file from an older device like that. Running it is always some risk though.

    Steve



  • @cjohnson Why do you think it's a BIOS problem vs any number of other reasons it could be dead?

    PS, I see a pair 770 on ebay for $99ea. Might be more time/cost effective to buy one or both.



  • Warning: Whiskey and tinkering don't always mix. I have now overwritten the RIOS boot HDD with FreeBSD. Yippee Ki Yay!



  • Why do you think it's a BIOS problem vs any number of other reasons it could be dead?
    At the time mostly, because of the LED/fan behavior; and a gut feeling based upon experience.

    PS, I see a pair 770 on ebay for $99ea. Might be more time/cost effective to buy one or both.
    This is exactly what I had done already. The second one wasn't here yet and I was working with what I had on hand at the time. The second box has since arrived. I swapped the BIOS ROM over from the working one and it booted right up to RiOS. After that, I shut it down and booted it back up on an ubuntu usb. Flashrom dumped the good ROM to a file; hot swapped in the bad one and dumped that to a file as well.

    I'm gonna' open 'em both up in a hex editor later and see what the difference(s) between the two is/are before I boot it up again and hot swap/re-flash the bad one with the image from the good one.

    I might do some digging around in the RiOS drive while I'm in there flashin' ROMs and see what sorta' goodies I can find related to LED color/controls and bypass relay controls too.



  • @cjohnson Good to hear there's hope for both units. FWIW I overwrote my RIOS drive in pursuit of a BIOS dump. Success on the BIOS dump (adios RIOS) and happy to send your way if you want another working example.



  • for anyone who's wondering....just another fine point of detail about the bypass functionality of the four network interfaces (LAN 0_0, WAN 0_0, LAN 0_1, WAN 0_1) on the right hand side (while viewed from the front) of the box.

    1. set either/both pairs of those NICs to "no bypass" in BIOS (they're labeled as NIC 3&4, NIC 5&6 in BIOS)
    2. save the changes and exit BIOS
    3. put the box in standby state (hold the front power switch down for a few seconds, before POST completes)
    4. the link status LED (not the activity LED) of the NIC pair you've set to "no bypass" will now be lit up amber

    BIOS label "NIC 3&4"
    pins 4&5 of LAN 0_0 will be connected to pins 4&5 of WAN 0_0
    pins 7&8 of LAN 0_0 will be connected to pins 7&8 of WAN 0_0

    BIOS label "NIC 5&6"
    pins 4&5 of LAN 0_1 will be connected to pins 4&5 of WAN 0_1
    pins 7&8 of LAN 0_1 will be connected to pins 7&8 of WAN 0_1

    this essentially means that with the box in this "standby" state, you'll be passing only PoE (not ethernet data) to/from LAN 0_0 to/from WAN 0_0 and, to/from LAN 0_1 to/from WAN 0_1. as soon as you power it up, the PoE shunt is opened.


  • Netgate Administrator

    Hmm, that seems odd. I might have expected the other two pairs to be linked for 100M pass-through. PoE pass-through that gets interrupted seems pretty much pointless!

    Steve



  • Thanks for all your guidance. I thought I'd followed it , but my CX-755 is giving an error:

    smbmsg: Error performing SMBus IO: Device not configured
    

    Any ideas?
    Although I'm learning as I go, apparently not quickly enough. Thx.



  • @sambiggs Try loading ichsmb and then smb manually. Post the output of these commands...

    kldload ichsmb
    kldload smb



  • I'd put those into the config.xml, right before the smbmsg commands. This is copied from the PuTTY output:

    ichsmb0: <Intel 631xESB/6321ESB (ESB2) SMBus controller> port 0x540-0x55f irq 19 at device 31.3 on pci0
    smbus0: <System Management Bus> on ichsmb0
    smb0: <SMBus generic I/O> on smbus0
    smbmsg: Error performing SMBus IO: Device not configured
    

  • Netgate Administrator

    Had you tried scanning the smbus at all? That can easy lock up requiring a reboot.

    Did you try entering all the commands manually first without putting anything in the config file?

    Steve



  • Sequence was:

    1st:

    • Drop to shell and add the following to /boot/loader.conf.local to load the smbus drivers
    • ichsmb_load=“YES”
    • smb_load=“YES”
    • add "smbmsg -s 0x48 -c 0x55 -o 6 0x03 0xfc 0x01 0xfe 0x66 0x99" to config.xml

    Result

    • Perhaps there was a "satisfying click" and I was then able to get a link light on em0, but it is not usable in pfSense.
    • em1-3 show no link lights

    2nd:

    • remove the added lines from /boot/loader.conf.local
    • add "kldload ichsmb" and "kldload smb" to config.xml

    Result

    • No click, and no noticeable difference. The error shown above appeared in the console output

    I haven't knowingly tried scanning the smbus, as I am still dumbly following your previous posts, and either that wasn't there, or it was buried in the stuff about your 1050 model, which I avoided in the cause of simplicity.

    I haven't properly understood the sideline about LED colour, but fwiw I get a single steady orange.



  • I still have the original RiOS drives with all the shell scripts and pythons scripts and all that stuff in it. I've been trying to reverse engineer it to figure out how it operates the status LED. If someone wants a look at it, I could probably send a few files your way....


Log in to reply