RCC-VE installation

csutcliff

Hi, had a bit of a rough time installing pfsense on my new RCC-VE 2440's so since I figured it out I thought I would share in case there is anyone else in the same boat. Apologies if any of this is wrong, inaccurate or unnecessary!

Basically there are two internal serial ports and the first (default) one is not the one we need to use for the console, this means if you try and boot a standard pfsense-memstick-serial image you won't get very far and the last output on the console is "booting from hard disk" or something similar.

Firstly you need a system that can read/write the memory stick (or the image before flashing) I used a pfsense livecd in a VM but you could use anything that works (actual pfsense box, freebsd etc)

mount the memory stick/image ~~and edit the [/mountlocation]/boot.config to read "-S115200 -Dh" (adding the extra "h"). I'm not sure this is necessary but I did it anyway!~~ edit: not necessary

Then edit the /[mountlocation]/boot/loader.conf file to have an extra line underneath 'comconsole_speed="115200"' that reads 'comconsole_port="0x2f8"'

unmount the memory stick and fire up the console of the rcc-ve. boot with the memory stick in and you should see the familiar pfsense boot sequence etc.

I went straight for an install to msata (embedded kernel) and I haven't tried installing to the internal emmc as I didn't want to lose the built in centos for various reasons.

The console changes on the memory stick don't copy over to the install so don't unplug the memory stick after the install finished and you should boot back to the live cd (usb seems to be preferred over anything else).

this time use the live cd instead of the installer and mount the partition you just installed pfsense to (mine was ada0s1a). ~~Make the same changes to the files like you did on the memory stick~~ this time add the comconsole_port line to [pfsensemountpoint]/boot/loader.conf.local so that it should survive upgrades etc, reboot and unplug the memory stick before it resets and you should be booting up into an installed pfsense with working serial console, set your interfaces like normal and enjoy!

I only worked this out around 40 mins ago so I have no idea if there are any other issues but if I find any I'll post them here.

Thanks for listening!

mwp821

I've been dorking around with this for an hour. Thank you for the quick fix! I'm so excited about this little box.

I'm also worried about losing the built-in CentOS install. Does Netgate ever make stock images/installers available? I might try to boot off another medium and `dd' an image of the original OS as a backup.

sscardefield

I've got a dumb question, how do you connect to these USB console ports? Just a regular USB to mini-USB cable? And then I'd guess I have to install some sort of virtual COM port or COM emulator on my machine?

mwp821

Yes, that's right, and you'll need this driver: http://www.silabs.com/products/mcu/Pages/USBtoUARTBridgeVCPDrivers.aspx

sscardefield

Thanks, that worked perfectly. I too would rather not wipe out the CentOS installation, but I do not have an mSATA drive and would rather not have to buy one if possible. Would it be possible to put Nano on a USB pendrive and somehow tell the 2440 to boot from that?

@mwp821:

I'm also worried about losing the built-in CentOS install. Does Netgate ever make stock images/installers available? I might try to boot off another medium and `dd' an image of the original OS as a backup.

My guess is the images will eventually be here:

https://firmware.netgate.com/

mwp821

@sscardefield:

Would it be possible to put Nano on a USB pendrive and somehow tell the 2440 to boot from that?

I was able to boot the pfSense Live CD off a USB memstick, so I don't see why not! The boot priority appears to be set to prefer the two rear USB ports over the eMMC. You just have to fix the console serial port per the above.

almabes

Thanks…You save me quite a lot of time messing around with this.

I had to figure out the mount command, but after that, I was golden.

mwp821

Just a quick heads up that if you're installing from a USB memstick to the internal flash (eMMC), the "Quick/Easy Install" may clobber the memstick. I had to redo mine.

If you want to install to and boot from the eMMC, there are some more steps you have to follow. The eMMC can normally be found at /dev/da0, but when you're booted from the USB memstick, it's /dev/da1. Unfortunately, the pfSense installer writes out a "bad" fstab. Here's how I fixed it:

Prepare the USB memstick with the serial console port fix as described above.
Boot into the installer like normal.
Select "Custom Install".
Select "da1" (or whatever device corresponds to "Generic Ultra HS-COMBO").
You can take the defaults for everything else (or customize as you see fit).
Select the embedded kernel (no VGA console).
Return to Select Task.
Exit.
Enter Ctrl-C to abort the reboot.
Press return and let it fire up a /bin/sh for you.
Execute the following commands (depending on your terminal emulator, you may have to paste these commands in one at a time):


mkdir /tmp/emmc
/sbin/mount /dev/da1s1a /tmp/emmc
echo 'comconsole_port="0x2f8"' >>/tmp/emmc/boot/loader.conf.local
sed 's/^\/dev\/da1s/\/dev\/da0s/' /tmp/emmc/etc/fstab >/tmp/emmc/etc/fstab.new
mv /tmp/emmc/etc/fstab /tmp/emmc/etc/fstab.old && \
  mv /tmp/emmc/etc/fstab.new /tmp/emmc/etc/fstab
/sbin/umount /tmp/emmc
/sbin/halt

If you prefer to edit loader.conf.local and fstab in-place instead of running the above commands, you need to run this before `vi' is usable:

export TERM=vt100

And finally:

Remove the USB memstick.
Press any key to reboot!

That should do the trick.

Disclaimer: Run these commands at your own risk!

UPDATE: Moved the comconsole_port setting to loader.conf.local, otherwise it gets overwritten when you click Save on the "System: Advanced: Admin Access" page in the webConfigurator.

sscardefield

I am having an issue trying to get the Nano running. Here is what I have done.

1. Created a Nano USB drive using pfSense-2.2.1-RELEASE-4g-amd64-nanobsd-20150313-0816
2. Mounted the USB drive on an existing Alix pfSense box
3. Modified /boot/loader.conf on the USB drive with the following:


hw.ata.atapi_dma="0"
hw.ata.ata_dma="0"
loader_color="NO"
autoboot_delay="5"
hw.ata.wc="0"
kern.ipc.nmbclusters="0"
beastie_disable="YES"
vm.kmem_size="435544320"
vm.kmem_size_max="535544320"
comconsole_speed="115200"
comconsole_port="0x2f8"
hw.usb.no_pf="1"

4. Unmounted the USB drive.
5. Put the USB drive in the 2440 and power it on.

At that point I can see the device booting up on the console, but it gets to the following point and freezes (or I just no longer get any console output):


Booting from Hard Disk...
Booting from 0000:7c00

At this point I can't tell if it is just losing the console or if it isn't booting. I am unable to access 192.168.1.1 on any of the eth ports like you would expect on a normal Nano install. Any ideas?

sscardefield

@mwp821:

Enter Ctrl-C to abort the reboot.
Press return and let it fire up a /bin/sh for you.
Execute the following commands (depending on your terminal emulator, you may have to paste these commands in one at a time):


mkdir /tmp/emmc
/sbin/mount /dev/da1s1a /tmp/emmc
sed '/^comconsole_speed/i\
comconsole_port="0x2f8"
' /tmp/emmc/boot/loader.conf >/tmp/emmc/boot/loader.conf.new
mv /tmp/emmc/boot/loader.conf /tmp/emmc/boot/loader.conf.old && \
  mv /tmp/emmc/boot/loader.conf.new /tmp/emmc/boot/loader.conf
sed 's/^\/dev\/da1s/\/dev\/da0s/' /tmp/emmc/etc/fstab >/tmp/emmc/etc/fstab.new
mv /tmp/emmc/etc/fstab /tmp/emmc/etc/fstab.old && \
  mv /tmp/emmc/etc/fstab.new /tmp/emmc/etc/fstab
/sbin/umount /tmp/emmc
/sbin/halt

mwp821

@sscardefield:

Sorry for all the questions, I am just trying to wrap my head around this. So these last couple steps, modifying the console settings in the fresh install on the eMMC… why is this necessary? Once you have a fresh install on the eMMC, can't you just boot from that, let pfSense load, enable SSH, and change the console settings in loader.conf that way? Or will the box not boot at all without the correct console settings?

Don't be sorry for asking questions! That's what the forum's for. I'm just a newbie trying to wrap my head around this myself.

A fresh install won't boot, period, unless you fix the device names in /etc/fstab on the target device (i.e. the partition of the eMMC where /etc is). And you won't be able to see anything unless you fix the console settings in /boot/loader.conf.

@mwp821:

I've been dorking around with this for an hour. Thank you for the quick fix! I'm so excited about this little box.

I'm also worried about losing the built-in CentOS install. Does Netgate ever make stock images/installers available? I might try to boot off another medium and `dd' an image of the original OS as a backup.

I'm pretty sure the QSG tells you how to PXE boot the CentOS install remotely.

mwp821

@gonzopancho:

I'm pretty sure the QSG tells you how to PXE boot the CentOS install remotely.

Sweet. The URL they gave me for the QSG (https://firmware.netgate.com/qsg/) comes up blank, so I've been flying a bit blind here. I did find a user manual for the RCC-VE platform but it didn't have anything in there about where I could find firmware images and how I could reinstall the factory operating system.

sscardefield

@mwp821:

Don't be sorry for asking questions! That's what the forum's for. I'm just a newbie trying to wrap my head around this myself.

A fresh install won't boot, period, unless you fix the device names in /etc/fstab on the target device (i.e. the partition of the eMMC where /etc is). And you won't be able to see anything unless you fix the console settings in /boot/loader.conf.

Oh, I didn't even notice the fstab modifications, makes sense. Thanks for all the info/help.

almabes

Another tweak…

Certain intel igb cards, especially multi-port cards, can very easily exhaust mbufs and cause kernel panics, especially on amd64. The following tweak will prevent this from being an issue:
In /boot/loader.conf.local - Add the following (or create the file if it does not exist):
kern.ipc.nmbclusters="1000000"
That will increase the amount of network memory buffers, allowing the driver enough headroom for its optimal operation.

see: https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#Intel_igb.284.29_and_em.284.29_Cards

EDIT: gonzopancho says don't do this, it's idiotic. See his post explaining why later in the thread…

frater

I successfully patched a USB-stick and was able to completely boot a 4G embedded image.
Because I had a mSATA disk I now wanted to try it with the installer.

I made a new USB and patched the 0x2f8 on that install stick.
I could completely run the installer but then after a reboot it did the same as before and I lost terminal access.

I think the installer made a new loader.conf for the SATA-disk and didn't add that "0x2f8" so all output is going to COM1…

Where can I find the source file that the installer is using to make a new loader.conf????

Because I'm not that "BSD" I have difficulty mounting the SATA-disk and changing loader.conf on the installed partition.

Edit: I had difficulty mounting the Sata-partition, because it was unclean. Did a fsck and now it's running.... I will have some other problems soon, I guess...

almabes

Also…
For about $60 I installed a 64GB Plextor mSATA SSD. That allowed me to keep the CentOS installation on the 4GB of eMMC. According to ADI's documentation, there are some utilities installed there you may want to keep. Also, it could be possibly a good debugging OS in case something in pfSense goes really sideways.

The boot loader defaults to booting from the mSATA instead of the eMMC if it's installed, which worked out great in my case. I didn't see anything in the documentation about changing the boot order.

sscardefield

@almabes:

Also…
For about $60 I installed a 64GB Plextor mSATA SSD. That allowed me to keep the CentOS installation on the 4GB of eMMC. According to ADI's documentation, there are some utilities installed there you may want to keep. Also, it could be possibly a good debugging OS in case something in pfSense goes really sideways.

I agree. I am currently running pfSense on the eMMC, but would like to have the CentOS as well. As soon as Netgate posts the QSG (https://firmware.netgate.com/qsg) or gives us instructions and how to restore the factory image then I will probably buy an mSATA drive and throw pfSense on there.

mwp821

@almabes:

According to ADI's documentation, there are some utilities installed there you may want to keep.

Yeah, I saw those in the RCC-VE platform documentation. Hopefully the team has made or will make those utilities available from within FreeBSD/pfSense as well. Maybe I should have waited to buy a SG-2440…

@almabes:

Another tweak…

Certain intel igb cards, especially multi-port cards, can very easily exhaust mbufs and cause kernel panics, especially on amd64. The following tweak will prevent this from being an issue:
In /boot/loader.conf.local - Add the following (or create the file if it does not exist):
kern.ipc.nmbclusters="1000000"
That will increase the amount of network memory buffers, allowing the driver enough headroom for its optimal operation.

see: https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#Intel_igb.284.29_and_em.284.29_Cards

the kernel doesn't panic when you exhaust mbufs, it panics when you set this limit too high (and your number is too high), because
the system runs out of memory.

For each mbuf cluster there is “mbuf” structure needed. These each consume 256 bytes, and are used to organize mbuf clusters in chains. An mbuf cluster takes another 2048 bytes (or more, for jumbo frames). There’s possibility to store some additional useful 100B data into the mbuf, but it is not always used.

When there are no free mbuf clusters available, FreeBSD enters the zonelimit state and stops answering network requests. You can see it as the zoneli state in the output of the top command. It doesn't panic, it appears to 'freeze' for network activity.

If your box has 1GB of RAM or more, 25K mbuf clusters will be created by default. Occasionally this is not enough. If it is, then perhaps doubling that value, and maybe doubling again, are in order. But 1M mbuf clusters? Are you serious?

You just advised people to consume 1,000,000 mbuf clusters (at 2K each). Let me know if I need to explain how much RAM you needlessly advised people to allocate for no good purpose.

I am well-aware that someone wrote something completely uninformed here: https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#mbuf_.2F_nmbclusters
so please don't quote it back to me.