Pfsense Install on Nokia IP390



  • Hi there.

    I recently just bought 2 Nokia IP390 units with a faulty IPSO installation. The units hardware is in perfect working order, but, does not function properly with the incorrect installations.

    I would like to install pfsense to these boxes, but, I don't know how. All tutorials I've came across so far assume that you're installing pfsense onto an old computer, not a dedicated firewall :L

    I'll give you some info:
    The IP390 is a flash-based hardware firewall, and uses a CF card to store the main operating system, and can also support the use of a 2.5" SATA hdd for use in logging, caching etc.
    It's factory firmware, IPSO, is based upon FreeBSD (Which I believe pfsense also uses?)
    There is no direct way to input to the devices. All interaction must be done through the Console port (An RJ-45 Rollover port, identical to those used on most Cisco units.)

    Any help would be appreciated!


  • Netgate Administrator

    Hmm, details seem a little sparse on the forum. However it is possible because this:
    https://www.youtube.com/watch?v=7AZZGem_CgA  :)

    Basically write the NanoBSD image of pfSense onto a suitable CF card, stick it in the box and boot it. Watch the console output to complete the install. See also:
    https://doc.pfsense.org/index.php/InstallationGuide#Embedded

    Steve



  • @stephenw10:

    Hmm, details seem a little sparse on the forum. However it is possible because this:
    https://www.youtube.com/watch?v=7AZZGem_CgA  :)

    Basically write the NanoBSD image of pfSense onto a suitable CF card, stick it in the box and boot it. Watch the console output to complete the install. See also:
    https://doc.pfsense.org/index.php/InstallationGuide#Embedded

    Steve

    Thanks!
    I'll give that a try when I can get my hands on another CF card, but, one question: Does this overwrite the current BIOS / Bootmanager on the device already?


  • Netgate Administrator

    It will overwrite everything on the CF card which would include the boot manager if you choose to use the card in the box already (if it's 1GB or bigger). It does nothing to the BIOS which is stored on the motherboard. If in the future you get hold of a working IPSO CF card you can just swap back.

    Steve



  • @stephenw10:

    It will overwrite everything on the CF card which would include the boot manager if you choose to use the card in the box already (if it's 1GB or bigger). It does nothing to the BIOS which is stored on the motherboard. If in the future you get hold of a working IPSO CF card you can just swap back.

    Steve

    Alright, thank you very much. My current plan is to leave the current IPSO cards as they are, and just use a new CF card for pfsense. I'll do some work on the IPSO cards when I have pfsense running, it does seem like an easy fix, but first I would like to get the hardware running!



  • So I've installed pfSense onto a CF card, and got my IP390 to boot to the pfSense config…

    That's as far as it got. I skip setting up VLAN interfaces (I currently don't need them to my knowledge),
    and it goes on to configure the WAN interface.

    I press "a" for autodetect, and then it just sends me in a loop of "No link-up detected" and goes back to asking about the WAN interface.

    I've tried every port on the damn thing, first the AUX port (The one that is designated for use connecting to the WAN), and ETH-1 through 4 all spew the same message.

    pfSense recognises ETH-1 through 4, and presumably the WAN interface although there is no activity lights on the interface to tell. All the NICs on-board are Intel PRO 1000 chipsets.

    Can I have some advice? I'm currently connecting the WAN port to my Modem, then through to the internet.



  • According to the manual (http://www.manualowl.com/p/Nokia/IP390/Manual/3822), the AUX port is a secondary serial port, not the WAN connection.

    The youtube video link posted by Steve shows the PMC slots populated with the optional 4-port lan card, and later in the video, it shows 4 intel ports in use (em0, em1, em4, and em5).  Does pfSense show em0-em3 on your unit?

    Can you capture and read through the boot messages from your console?  Or post the boot log here.  That would tell what nics are detected.

    Also, maybe the units were not working due to a hardware issue rather than a faulty IPSO flash?


  • Netgate Administrator

    Exactly. The auto detect function doesn't seem to work with all NICs unfortunately. At the config screen it lists all the available interfaces above the first question. I would expect it to list em interfaces something like:

    Valid interfaces are:
    
    em0   00:00:24:ce:45:74   (up) Intel(R) PRO/1000 Network Connection 7.2.3
    em1   00:00:24:ce:45:75   (up) Intel(R) PRO/1000 Network Connection 7.2.3
    em2   00:00:24:ce:45:76   (up) Intel(R) PRO/1000 Network Connection 7.2.3
    em3   00:00:24:ce:45:77   (up) Intel(R) PRO/1000 Network Connection 7.2.3
    

    Just enter the WAN and LAN interfaces manually. You may have a little fun and games finding out which port on the box is which interface number. The ports are usually detected in some logical order (0-4 left to fight for example) but not always!  ;)

    Steve



  • @charliem:

    According to the manual (http://www.manualowl.com/p/Nokia/IP390/Manual/3822), the AUX port is a secondary serial port, not the WAN connection.

    The youtube video link posted by Steve shows the PMC slots populated with the optional 4-port lan card, and later in the video, it shows 4 intel ports in use (em0, em1, em4, and em5).  Does pfSense show em0-em3 on your unit?

    Can you capture and read through the boot messages from your console?  Or post the boot log here.  That would tell what nics are detected.

    Also, maybe the units were not working due to a hardware issue rather than a faulty IPSO flash?

    Hi there – Yeah, thanks, I didn't know that because the manual I have printed incited that the AUX port is the WAN interface -- Silly me!
    Yeah, I've ordered 2x Optional 2-port Gigabit PMCs, due to arrive tomorrow -- I probably won't resume work on this until they arrive, simply in the event pfSense doesn't like having new ICs installed after pfSense has configured itself.
    I'll do that tomorrow, but the auto detection of NICs is eth0 - 3 of INTEL PRO 1000 Gigabit NIC
    Yeah, I did look into that issue, but I found out that the IPSO cards both have faulty file permissions which I might fix at a later date, but the hardware is all in working order.


  • Netgate Administrator

    Adding NICs after initial config is not normally a problem. One issue than can happen is if you add more em NICs it might offset the existing em NICs. Even so you would still just re-assign the interfaces.
    It would be useful to complete the install to test the connectivity of the existing NICs. Embedded boxes like that sometimes have custom options waiting to trip you up.  ;) You can always re-image the CF card easily enough.

    Steve



  • @RBT-RS:

    I'll do that tomorrow, but the auto detection of NICs is eth0 - 3 of INTEL PRO 1000 Gigabit NIC

    Sure it's not em0 to em3?  Coming from linux, it took me a while to get used to NICs being named according to the underlying hardware, like em0 or igb0, rather than being presented as eth0 for any hardware type.



  • Alright, pfSense is now running brilliantly on the IP390!

    One thing to add is that under the default interfaces, ETH-1 is actually em0 in pfSense, and ETH-4 is em3.

    Also, would it be possible to add another hard drive for use caching and stuff?


  • Netgate Administrator

    Nice!  :) So relatively logical interface detection then. Just watch out for what I said above if you add more.
    You can add a harddrive and use it for caching but there is no system for doing so built into pfSense. Others have done it using some custom scripts etc but it's almost certainly easier to just use the harddrive as the boot device (full install) and forget about the CF card.
    See: https://forum.pfsense.org/index.php?topic=67823.0

    Steve



  • Thanks Steve.

    Another quick thing, I just managed to do some data recovery on the IPSO CF card, and have managed to extract the backup IPSO image (Used in case of a critical system failure) and original kernel.

    Would it be possible for me to use the original kernel with a pfSense install? If so, how would I go about doing this? (Note that IPSO also runs off freeBSD, so there shouldn't be any compatibility problems)



  • @RBT-RS:

    Would it be possible for me to use the original kernel with a pfSense install? If so, how would I go about doing this? (Note that IPSO also runs off freeBSD, so there shouldn't be any compatibility problems)

    Almost certainly no, not possible.  Why would you want to?  If you want to experiment, you could try booting IPSO for a comparison to pfSense.


  • Netgate Administrator

    Indeed, pfSense uses a custom kernel and I would expect IPSO does also. The base FreeBSD versions are probably different. I'd be amazed if it was compatible.

    Steve



  • Hmm.. Alright. The only reason why I wanted to use the IPSO kernel was because the warning light on the unit remains on which indicates an internal voltage error. The unit runs fine, so I can assume the warning is false, but it would be nice to get that working properly



  • Try booting IPSO: if the LED goes off you're OK, and it's likely a IPSO userland utility that controls it.  If it stays on, you have some more investigation to do.  Of course, wire cutters or black tape could fix the problem too …


  • Netgate Administrator

    Since that same indicator can show over temperature it's probably driven from the board rather than the psu which is good. It's probably driven from the SuperIO chip where the voltage and temperature sensors are connected.
    There maybe some options in the BIOS to change the indicator behaviour otherwise you could probably tweak it manually with a utility and a script.

    Steve



  • Hmm, alright.

    Yeah, booting into IPSO turns the warning indicators off, so the unit is fine.

    Alright, thank you. I don't think I can actually change the bios settings, or at least not from the console. How would I go about getting a utility / script working?


  • Netgate Administrator

    Can you access the bios at all? It may be at 115,200 on the serial console.
    It's not straight forward doing it from software. It will probably require a lot of trial and error and educated guesses. See my own effort here: https://forum.pfsense.org/index.php?topic=32013.0
    The first thing would be to try to determine how the led is connected to the board. Then gather as much info about the board as you can. In particular the superio chip and the north/south bridge chipset. If it's not made by nokia who made it etc.

    Steve



  • Hi Steve,

    Uh, I can't say I know what you're talking about, sorry. The serial console connection is currently set to 9600 bps, 8 data bits, No parity and 1 stop bit.

    The following image shows the assembly behind the console ports and status LEDs:

    The large chip in the foreground is an Intel nhe3600esb I/O controller. the status LEDs connect through a sister board, which connects to the motherboard by a ribbon cable just behind the I/O controller.

    Seems like in your attempts with a similar circumstance required digging through the kernel. I'll take a look at that in a minute or so, once I can find a program which can read it :/


  • Netgate Administrator

    The proximity to the ICH chip (6300esb) would have me guessing it's controlled by one of the GPIO pins on that. That was where I found the control on the earlier firebox models. The only issue is that I don't think the 6300 had a flashing mode.  :-\  There are lot of components on that daughter board considering it's only got 3 leds on it. Flashing could be done there.

    Steve



  • Hmm. The daughter board actually governs 4 LEDs, as, from left to right, you have: System activity indicator (green), System warning (Yellow !), System On/Off (Nokia logo, illumunated blue) and the system fault indicator (Red X ). I've worked out that 5 of the LEDs on the daughter board govern the system On/Off light alone.

    Here's a picture of the daughter board taken off:

    There's a few chips on here, any ideas? My initial thoughts were that they were voltage regulators, but I'm pretty sure the System fault indicator is ran from the BIOS, which might explain the chip on the top right of the board. That would also explain why the fault indicator is working properly.


  • Netgate Administrator

    I agree there's probably some voltage or current regulation. The outputs from the ICH are probably not capable of sinking enough current to drive the LEDs directly so instead there are transistors for each LED (Q1, Q2 etc) and a line driver/buffer (the 7407 IC). The other large device, labelled 10-16L, is a capacitor perhaps to filter switching spikes. I like that there's a mystery space for an LED, CR9, that Nokia decided not to use for whatever reason.  :)

    Do the 4 LEDs come on as soon as you power on  the box? Do they vary at all during boot?

    The BIOS setup in embedded hardware like these boxes is often available on the serial console via 'console redirection'. It's usually at a baud rate of 115200bps. To access it you often have to press TAB because you can't send DEL over serial. Interestingly it looks like that box has two BIOS ROM chips.

    Edit: Maybe on the other serial port, AUX, perhaps.

    Steve



  • You're correct. The board has support for 2 BIOS chips, but only the right one is populated with a BIOS chip. According to the stickers on the MB, the BIOS is an American Megatrends AMI BIOS 786Q.

    As for accessing the BIOS, I'll give that baud rate a try tomorrow. How will I know when I'm in the BIOS? Will it print text like the normal 9600 baud rate does?

    I believe that the LED for CR9 Might be for and LED for the IP560. I don't exactly know what that LED would be, but I would imagine that Nokia use the same status LED boards across most of their IP series products to save money.


  • Netgate Administrator

    Searching for info on the IP390 there are a number of people who have posted IPSO boot logs and it includes the memory count. That is before the bootloader so it's coming from the BIOS. It's in the same log at the same baud rate so I would suggest whatever baud rate IPSO uses should be good for accessing the BIOS setup. If you're seeing the memory count you're probably good, try pressing TAB. Of course Nokia could have it all shut down deliberately to stop access but they haven't forced only signed boot images for instance.

    Steve


  • Netgate Administrator

    Just for reference the IP530 uses the SuperIO chip to control those LEDs:
    http://www.coreboot.org/Board:nokia/ip530#Front_LEDs
    Does not mean the IP390 does though.  ;)

    Steve



  • Alright, I just managed to get into the BIOS by pressing tab after the memory test had completed.
    It came up with a message:

    American Megatrends AMBIOS (version #)
    Press ~~ to enter setup.~~
    

    It wouldn't allow me to enter text on that screen.
    Here's pictures of the BIOS:
    http://s26.postimg.org/j5bb13bi1/bios1.png
    http://s26.postimg.org/mq76kbg1l/bios2.png
    Here's inside the SuperIO menu:
    http://s26.postimg.org/j7v6nxf5l/bios2_5.png
    http://s26.postimg.org/fpj6rje9l/bios3.png
    http://s26.postimg.org/41p4wzp4p/bios4.png
    http://s26.postimg.org/cl8iuqxh5/bios5.png
    http://s26.postimg.org/f3u7vfj7d/bios6.png

    That's all of it. There doesn't appear to be any options for the status LEDs on the front panel.. Any ideas?

    Sidenote: I should have said the IP690, not IP530. The IP690 is the model up from the 390.


  • Netgate Administrator

    It was worth looking, only one of the Watchguard boxes had a custom bios menu for the led. If it is connected to the ICH it's worth looking in the southbridge setup menu.
    I assume that the yellow warning led comes on as soon as you power up the box and then just stays on?

    Steve



  • Here's a picture of the southbridge menu:

    Yeah. On boot the only LED which does not switch on is the System fault. All the other LEDs, System activity (Is meant to be on), System warning (Should not be on) And System Power (Should be on) Are all on.



  • Any chance something shows up as /dev/led* or /dev/gpio*?  If one or both show up under IPSO, but not pfSense, that could be a big clue.



  • Hmm. I see what you're getting at, but how can I check if /dev/led* or /dev/gpio* show up under IPSO? What command should I enter?



  • @RBT-RS:

    Hmm. I see what you're getting at, but how can I check if /dev/led* or /dev/gpio* show up under IPSO? What command should I enter?

    Do you get a shell prompt on the serial console with IPSO?  If not, try the AUX port, which is set up for remote management via modem.  Or try to ssh into the unit.

    For pfSense, you can use the shell on the serial console, or ssh in to the unit.  Choose '8' from the menu, then use 'ls' command:

    *** Welcome to pfSense 2.2-ALPHA-pfSense (amd64) on pfsense ***
    
     WAN (wan)       -> bge1       -> v4/DHCP4: xx.yyy.zzz.12/20
     LAN (lan)       -> bge0       -> v4: 192.168.2.128/24
     OPT1 (opt1)     -> bge1_vlan2 -> v4: 192.168.100.5/32
    
     0) Logout (SSH only)                  8) Shell
     1) Assign Interfaces                  9) pfTop
     2) Set interface(s) IP address       10) Filter Logs
     3) Reset webConfigurator password    11) Restart webConfigurator
     4) Reset to factory defaults         12) pfSense Developer Shell
     5) Reboot system                     13) Upgrade from console
     6) Halt system                       14) Disable Secure Shell (sshd)
     7) Ping host                         15) Restore recent configuration
    
    Enter an option:
    
    [2.2-ALPHA][root@pfsense.localdomain]/var/log(1): ls -l /dev/gpio* /dev/led*
    ls: No match.
    
    

    The ls command would be the same in either IPSO or pf Sense.



  • I'm getting no output on the AUX port under pfSense, and the console port gets to "bootup complete" and won't accept further input. I'll try to ssh to pfSense in a minute.



  • Alright, I managed to get the pfSense shell working through the serial port - There was an option under System > Advanced to enable the serial console.

    So I booted into the shell, and ran that ls command:

    And the same command under IPSO:

    As you can see, IPSO didn't find anything under /dev/led* or /dev/gpio* … Any more ideas? Thanks for helping by the way.


  • Netgate Administrator

    That would have been nice but was a pretty long shot, FreeBSD has very few specialist drivers like that.


  • Netgate Administrator

    You have two options here (three if you decide it's just not worth the trouble!):
    Modify the BIOS to turn off the LED at boot. It's probably one of the settings in the BIOS tables that are fairly easy to spot.
    Switch it off after boot using a custom utility.

    Both of those require knowing where the LED is driven from. Modifying and then flashing the BIOS is inherently risky.
    I would certainly be willing to work with you to find the LED if you wish. You can read the posts in the arm/disarm led thread where ifloris and I found the location on the X-core box without me having access to it: https://forum.pfsense.org/index.php?topic=32013.msg171469#msg171469

    Steve



  • Yeah, I think option 2 would be the best option, given that if I was to flash the BIOS i'd have to go buy a few bits of hardware to get the new bios on the chip, and a couple of BIOS chips so I don't need to overwrite the original.

    Should I upload the original IPSO kernel so you can take a look through it?
    Also, would the readio and writeio programs work on the IP390 do you think?


  • Netgate Administrator

    You wouldn't need any additional hardware to flash the bios. You can do it from within pfSense the flashrom utility (available as a FreeBSD package) or boot a DOS image somehow and use AMI's flash utility. Though if you want to keep the original chip you would. The two bios sockets might introduce to extra unknowns. You can extract the existing BIOS to see if it gives us any clues using flashrom relatively risk free.

    I'm not sure you've understood quite what the kernel is (I apologise if I'm wrong!). Looking through the IPSO file system for clues would help though. There is probably some script that calls the custom driver to change the LED for example.

    The readio and writeio programs will certainly work. They have almost no saftey features that might stop them working!

    Steve