Revised New Build



  • O.K., so I have finally decided to build a PfSense router/firewall using these parts:

    SUPERMICRO CSE-502L-200B Server Chassis

    SUPERMICRO MBD-X7SPE-HF-D525-O Flex MB

    CRUCIAL 8GB 204-Pin CT2KIT51264BC1067 SO-DIMM

    Note that the motherboard and chassis are the same as is included with the SUPERMICRO SYS-5015A-EHF-D525 server, but buying them as separate items yields about a $70.00 savings.  I like the features of this chassis.

    I still have not decided on a hard drive or SATA DOM.  In considering the latter, all the vertical SATA DOM modules I have looked at measure between 1.3 and 1.5 inches high.  Since the chassis is only 1.7 inches high, I don't think it has the physical space to accommodate such a module.  I suppose I could use a (male to female) SATA extension cable to accommodate the SATA DOM, but that seems a little Rube Goldberg to me as it defeats the intent of the on-board module.  Any thoughts on this matter?  Would I do better to just boot from a USB drive–and would there be any problems doing so (over an SATA DOM)?

    Thanks. ???



  • Perhaps I should clarify my intent.  I would like to run PfSense directly off of the SATA DOM or USB Flash Memory for the time being, at least until the price of hard drives comes down again.  Per my previous post, I am running off of a 15/1 Mbit DSL lines, so the Atom processor is more than adequate for my needs.  The Supermicro Chassis is the most elegant framework for the Supermicro motherboard–although I seriously considered and rejected the Mini-Box platform.  It appears that the Supermicro motherboard does support boot from USB, but I would like to know the limitations of running PfSense from USB before finalizing my purchase. ::)


  • Netgate Administrator

    If you booting from flash memory of any sort you will probably be running an embedded install. In that situation the speed of the flash will only effect the boot time slightly, once booted in runs almost entirely from RAM anyway. The only advantage of using a DOM is that it will make for a slightly easier install as it's connected in place of a HD so there is no need to edit the fstab to tell it where to boot from.

    It looks like that board has an internal USB socket that you could use with a short USB stick.

    The Atom will be easily capable of saturating your 15/1 line, even with encrypted traffic if you need that.

    Steve



  • I've ran both thumbdrives and CF-SATA adapters in pfSense for sometime now.
    They generally work great with embedded (NanoBSD) variants of pfSense but loading times from a clean boot can be slower than a HDD (depends on the thumbdrive/ CF card used).

    For the thumbdrive setup previously, I unscrewed a USB extension bracket and used the plug internally.  It hid away nicely in the HDD bay.  Much better than sticking it in the back of the setup where there's the risk of the thumbdrive getting knocked off by accident.

    For Embedded (NanoBSD), the system runs off RAM but there is periodic saving of config and logs (and RRD graphing data) to the embedded device.  This can be configured (interval between saving) or disabled.

    I'm currently using a CF-SATA adapter with a Transcend 133X CF card.  I haven't enabled DMA mode (disabled by default on NanoBSD) but it loads plenty fast.

    The main advantage of an using a thumbdrive/ CF card lies in the fact that it is solid state and more resilient to physical shock.  Furthermore, it is less prone to corruption in the event of a power failure (as long as the power outage doesn't happen while the logs/ RRD data is being written to the drive).

    Power consumption is also significantly lower (a thumbdrive uses less than 1W even after power conversion losses in PSU) and is generally less sensitive to and produces less heat.
    The last part is especially important for systems with limited cooling.  I used to run a WD Raptor 10K RPM drive in my old pfSense rig and the drive was toasting the rest of the setup.

    The main disadvantage is that you have limited space (maximum size for Nano is 8GB or 4GB per slice) for disk I/O heavy packages like Squid.  You can still run Squid with only RAM caching but that may not be enough for your needs.

    Also, depending on your setup, the USB booting may need some changes to the stock config.  The default timeout of 10 seconds may not be sufficient for the USB drivers to load nicely on the setup.  In that case, some intervention is needed for the first boot until the changed timing is commited to the bootloader file.



  • O.K. great, thanks to the two of you for posting informative responses to my questions.  I found a SATA DOM that will fit in the server chassis, but it costs in the neighborhood of $140-$230!  I think I will go with a USB thumb drive: a top-rated 4 GB unit at New Egg runs about $11–if it ever fails, replacing it would be cheap.  Regarding the latter, do you think that 4 GB is sufficient or should I go with 8 GB?  Also, which version of the embedded firmware should I use?  Please keep in mind I would like to use the IPMI (ver. 2) feature of the motherboard, so I assume I should use one of the VGA versions?  And which size?  If you can elaborate on how to change the default boot timeout, I would additionally appreciate that information.  I will shut down the log reporting features.


  • Netgate Administrator

    4GB will be fine.
    There are some interesting notes on running pfSense on that board here:
    http://www.servethehome.com/supermicro-x7spehfd525-8gb-ddr3-ipmi-pfsense-freenas-unraid-linux-power-consumption/

    Here is info on the boot delay:
    http://doc.pfsense.org/index.php/Boot_Troubleshooting#Booting_from_USB

    I've never used IPMI so can't help you there.  ;)

    Steve



  • AFAIK the IPMI is living its on life and not OS dependent. I use it often on my Dell SC1435.



  • @stephenw10:

    There are some interesting notes on running pfSense on that board here:
    http://www.servethehome.com/supermicro-x7spehfd525-8gb-ddr3-ipmi-pfsense-freenas-unraid-linux-power-consumption/

    Thanks for link. :)



  • Thanks again, guys, the information you have supplied is exactly what I needed and the article on the PfSense application of the motherboard I was considering was interesting and informative.  I went ahead and ordered all my components from New Egg yesterday.  Hopefully, they will arrive in time that I may assemble them this weekend.  I ordered a 4 GB USB thumb drive along with an internal 6-inch header to USB-A cable.  I should be able to strap the thumb drive somwhere inside the chassis with a plastic wire tie.  I can always replace it with something more elegant at a later date.  Once I get the rig up and running, I will consider purchasing a riser card and a Viking PCI ADSL-2 modem card (if I can find a vendor that sells them in the USA).  I will assume that the correct version of the PfSense embedded firmware that I will need to download (I'd like to run the 64-bit version) is "pfSense-2.0.1-RELEASE-4g-amd64-nanobsd_vga.img" ???


  • Netgate Administrator

    Unless you have bought a particularly long USB drive you should just be able to plug it into the internal USB socket on the board.

    That is the correct image if you need VGA. That board has a serial port though so you could use the standard Nano image with a null modem cable. Either way you will probably be faced with a mount root error the first time you boot. Don't panic! That's normal, you simply have to tell pfSense where your USB drive is connected and then edit the fstab so it knows next time.

    Steve



  • Thanks stephenw10, I goofed in assuming the on-board headers did not include a USB-A port.  I just purchased a mini-USB device at Staples and loaded the firmware on it as the USB drive I purchased through New Egg is indeed too tall to plug into the onboard USB-A port.  I received all my other hardware in the mail today and will assemble it tomorrow.  Wish me well.



  • Up-n-Running!  And it's already a heck of a lot faster than my old Trendnet router!  Now I have to make sure all logging is turned off.

    :D


  • Netgate Administrator

    Logging is all to ram only so no worries there. If you used the NanoBSD image it's all taken care of anyway. No need to worry about flash memory life.

    Steve



  • O.K., thanks Steve.  I may buy a SATA DOM eventually–USB works but it takes a long time to boot!


  • Netgate Administrator

    How long? How often do you plan on re-booting!?
    My machine is usually up until there's either a new release or I do something to kill it.  ::)

    Steve



  • Once it is fully configured I plan to leave it on continuously without rebooting.  It takes ca. two minutes for PfSense to boot up after the motherboard posts.


  • Netgate Administrator

    That's similar to my Firebox booting from CF. Not unusually slow.

    Steve



  • By the way, it turned out I did not have to do any compensation for the boot to USB option.  I have rebooted my build several times and have not run into such a problem.  I just purchased an ADSL2+ PCI card (it is made in Australia and I had to buy it from a vendor in Europe).  It will take a couple of weeks to arrive, but I plan to use it to bypass my external DSL modem.


  • Netgate Administrator

    The Vikiking card from Traverse? (They also seem to have re-branded as rocksolid electronics)
    I ran their earlier Pulsar ADSL card under IPCop for years, totally reliable.

    Steve



  • "The Vikiking card from Traverse?"

    Yes, I guess I'll have to adjust the settings to get it to work with my Verizon DSL service.



  • When using Snort in the embedded version of PfSense that I am using, will the updates download to RAM or will they write on my solid state memory?  I don't want to install anything that will do periodic memory writes. ???


  • Netgate Administrator

    Hmm, I'm not too sure about this but if it's available for embedded installs then someone else has probably already thought of it.
    How often does snort update? Even the most basic flash memory still has a large number of writes in it's lifespan.

    Steve



  • @stephenw10:

    How often does snort update?

    It is user selectable. Tho every 12 hours is recommended.



  • stephenw10:

    You mentioned you had experience with something similar to the Viking card.  I was able to install the card in my PfSense router and place it in bridge mode.  I adjusted VPI/VCI to 0 and 35 per the Westel modem I use with my Verizon DSL account.  All seemed to go well until I plugged my DSL telephone line into the card.  The power and LAN lights are on but the DSL light just flashes slowly–the card does not appear to be trying to negotiate a sync with the DSLAM in my local central office.  I will check to see if it a simple problem with my phone cord, but it seems odd that everything else has gone o.k. with the setup and then the card will not sync to my DSL service.  I already tried changing the settings between ADSL2, ADSL2Plus, etc.  Any ideas? ???


  • Netgate Administrator

    The Pulsar adsl card was a true modem rather than a router on a PCI card like the Viking is.
    There will almost certainly be some logging available on the Viking card. If not on the web interface maybe on a telnet interface?

    Steve



  • I can't seem to get the web interface to work.  I am using IPMI with the console.  I guess I have to download the command index file and do some more research.  I may not have hit on the right ADSL settings as yet.


  • Netgate Administrator

    How do you have it setup?
    This should still apply to your situation:
    http://doc.pfsense.org/index.php/How_can_i_access_my_PPPoE_Modem_on_WAN#For_2.0

    Steve



  • Steve:

    It turns out that my Viking card was not getting 12VDC (to power its DSL front end) through my PCIe slot.  My riser card adapter, however, has a 12VDC power connector; once I found the right adapter cable I was able to install it and supply 12VDC to the adapter and Viking cards (in addition to the 5VDC already present).  The Viking card now works, yippie! :D



  • Steve:

    The ability to access my modem's web interface is something in which I still might be interested.  The instructions in the link you supplied, however, are a little confusing.  My modem has a default address of 192.168.1.1 (I don't know yet if it can be changed) but my LAN network is on 192.168.0 etc.  Is it possible for me to access the modem's web interface and how EXACTLY (please) should I set up PfSense to do so?

    Thanks.


  • Netgate Administrator

    Most modem/routers (but not all!) have a web interface that is still accessible even when it's in 'pass through' or 'bridge' mode in order to see the line stats etc. This is useful!
    You can almost certainly change the default subnet of the router and I would because you may end up using 192.168.1.1 later on (indeed it's the default address of the pfSense LAN). Set it to something less common, say, 192.168.200.1.
    You may have to play about with the modem settings to do this. You might end up having to reset it, I did!

    Which part if the instructions are you unclear on?

    Steve



  • Steve:

    I did change the router subnet: I set my LAN to 192.168.0.1-254
    My modem card default address is 192.168.1.1

    I tried following the instructions, added the "opt" interface o.k. but I am completely bewildered as to configuring NAT in the firewall.

    Do I make up an address for the modem in my LAN subnet, e.g. 192.168.0.30, and use it to access my 192.168.1.1 modem?  How do I configure NAT under firewall to do so.  I see six rules when I save under "Manuel."  Which one do I use?  The instructions in the link you provided lack, as we use to say, "human engineering."



  • @Nonsense:

    I tried following the instructions, added the "opt" interface o.k. but I am completely bewildered as to configuring NAT in the firewall.

    Take a deep breath and proceed slowly.

    http://doc.pfsense.org/index.php/How_can_i_access_my_PPPoE_Modem_on_WAN#For_2.0 says

    Add an Outbound NAT rule as described above but do NOT choose the WAN interface, choose your new OPT interface.

    Scroll upwards through the document from the "For 2.0" section and you will come to the "Configure NAT" section which explains why NAT (Network Address Translation) is needed/recommended. Where this section says WAN you will need to give the OPTx name of the interface you have just added.

    Now, where exactly in that section do you get stuck?

    @Nonsense:

    Do I make up an address for the modem in my LAN subnet, e.g. 192.168.0.30, and use it to access my 192.168.1.1 modem?

    No.

    @Nonsense:

    How do I configure NAT under firewall to do so.  I see six rules when I save under "Manuel."  Which one do I use?

    None, you add a new rule as described in the "Configure NAT" section of the document.

    @Nonsense:

    The instructions in the link you provided lack, as we use to say, "human engineering."

    You are probably correct. I sympathise with technical writers. Many readers are used to scanning (rather than reading) technical documentation and seem to get flustered if they come across "too many" unfamiliar terms "too quickly". And it can be quite difficult for a full-time developer/part-time technical writer who is very familiar with a field to enter into the mind set of a technically competent person who is unfamiliar with the details of a field and so doesn't have a lot of the context that people working in the field naturally assume is shared.



  • They say, "A picture is worth a thousand words."  The instructions need to work from the general to the specific and then give a specific example while defining terms and conditions ahead of time; e.g., they might say, "if you have set your router to an address of 192.168.0.200 and your modem has a default address of 192.168.1.1, here is the information you should specify when you edit the outbound firewall rule . . ."  It would be easier to reverse-engineer the process that way–I am still bewildered by what I should specify in the outbound firewall rule and what IP address I would type into my browser to access my modem when all is said and done. ???


  • Netgate Administrator

    Ok, once you have added the extra interface and it's in the same subnet as the modem here is the problem:
    Your modem does not have a route to your client box.
    This is because the modem is set up to expect to have gateway on it's WAN but in bridge mode it never connects. The only thing it can see is the new interface on your pfSense box but it doesn't know to use that as a gateway.
    I can think of at least 4 ways of solving this problem.
    1. Make your pfSense box NAT the connection to the new interface by adding a manual rule. That way traffic arriving at the modem appears to have come from the new interface and it can reply there. This is what you are trying to do.
    2. Make your pfSense box NAT the connection to the new interface by adding an extra gateway on the new interface.
    3. Add a gateway to modems LAN connection. This isn't always possible via its GUI, my modem couldn't do it.
    4. Expand the modems LAN subnet to include addresses in your pfSense LAN. This way it knows which interface to send replies out of. e.g. 192.168.0.1/16. This is what I have done.

    The NAT rule for option 1 should be:
    Interface: your new interface
    Source: network 192.168.1.1/24
    Destination: network 192.168.0.1/24
    Translation: Interface address

    Steve



  • :D

    O.K, let's try this again, if you have the patience for me.  Please title your next post, "Network Translation for Dummies" and assume NOTHING.  Steve, I tried your suggestions 1 and 4, but everytime I try to get into my modem card's web interface I get directed to the pfsense dashboard instead.
    :(


  • Netgate Administrator

    Ah, sorry about that.  ;)

    Ok, that's weird.
    If you have the NAT rule in place you should now be able to access the modem on 192.168.0.1.

    Do you have any thing left over from previous attempts?

    Can you post a screen shot opf your NAT rule?

    Can you ping the modem?

    Steve



  • Steve:

    Every time I try to add the NAT rule (per your option 1) using as you suggested,

    The NAT rule for option 1 should be:
    Interface: your new interface
    Source: network 192.168.1.1/24
    Destination: network 192.168.0.1/24
    Translation: Interface address

    what happens is the rule automatically defaults to:

    Source: network 192.168.1.0/24
    Destination: network 192.168.0.0/24

    Also, "interface" does not give an option to enter the IP address I created when I generated my (modem card) interface.

    I am using the latest version of the PfSense embedded software.

    Could you elaborate on how you did your option 4 please?


  • Netgate Administrator

    @Nonsense:

    Also, "interface" does not give an option to enter the IP address I created when I generated my (modem card) interface.

    Hmm, that would be a problem. Is the new interface enabled and 'up'?
    The networks defaulting to .0 instead of .1 is not a problem.

    On my modem, a Draytek V120, I have changed the LAN IP to 192.168.0.1/16. It has the option of entering the LAN subnet via it's webgui so I set it to 255.255.0.0
    Now it has a route to other IPs within that /16 so it can send return packets.

    This trick is a bit nasty and I know it doesn't work with all routers/modems. I have a router here I use as a wifi AP and that still can't return packets.

    Steve




  • I have to get to my modem first before I can change its settings.

    :D

    Perhaps you can elaborate upon option 2?

    I read in another forum that one has to disable PPPoE in order to talk to the modem card.


  • Netgate Administrator

    Ok, option 2.
    pfSense automatically NATs the connection between WAN and LAN in it's default configuration. In fact it will automatically NAT between any internal interface and any interface that has a gateway, which it then treats as a WAN type.
    So you can get pfSense to NAT between your LAN and the new interface you created by simply adding a gateway to it.

    1. Make sure you have NAT set to automatic in Firewall: NAT: Outbound:
    2. Add a gateway to your new interface. Goto Interfaces: Yournewinterface: Gateway: 'add a new one'. Set the gateway to the IP address of your modem.

    In order to setup my modem as it is shown I had to unplug it and connect to it directly with a laptop manually configured to be in the same subnet. I believe you can do something similar with the Viking by using the extra port on the back and moving some jumpers?
    It may be that it disabled the web GUI when set to PPPoE bridge mode, but it seems unlikely as you'd then have no access to it. Do you have a link to that post?

    You can test to see if the modem is responding to anything by pinging it from the pfSense box directly. This will also check that your new interface is configured correctly. You can do this without any other trickery because the two devices are already in the same subnet. Taking this a step further you may be able to telnet to your modem from pfsense and reconfigure it that way. Here's me doing that:

    
    [2.0.1-RELEASE][root@pfsense.fire.box]/root(1): ping 192.168.0.1
    PING 192.168.0.1 (192.168.0.1): 56 data bytes
    64 bytes from 192.168.0.1: icmp_seq=0 ttl=255 time=0.536 ms
    64 bytes from 192.168.0.1: icmp_seq=1 ttl=255 time=0.290 ms
    64 bytes from 192.168.0.1: icmp_seq=2 ttl=255 time=0.286 ms
    64 bytes from 192.168.0.1: icmp_seq=3 ttl=255 time=0.285 ms
    ^C
    --- 192.168.0.1 ping statistics ---
    4 packets transmitted, 4 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 0.285/0.349/0.536/0.108 ms
    [2.0.1-RELEASE][root@pfsense.fire.box]/root(2): telnet 192.168.0.1
    Trying 192.168.0.1...
    Connected to 192.168.0.1.
    Escape character is '^]'.
    
    Password: *****
    
    Type ? for command help
    
    > ?
    % Valid commands are:
    upnp         ddns         exit         internet     ip           ipf
    csm          ddos         urlf         log          portmaptime  quit
    srv          sys          show         mngt         wan          adsl
    wol          vigbrg       tsmail
    
    > show?
    % Command missing, Valid commands are:
    upnp         ddns         exit         internet     ip           ipf
    csm          ddos         urlf         log          portmaptime  quit
    srv          sys          show         mngt         wan          adsl
    wol          vigbrg       tsmail
    
    > show
    % Valid subcommands are:
    lan1         lan2         dhcp         dmz          dns          openport
    nat          session      status       adsl
    
    > show lan1
    %% 1st subnet settings:
    %%      IP address: 192.168.0.1
    %%      Subnet mask: 255.255.0.0
    %%      RIP : [Disable]
    
    

    Your modem telnet interface will be different (if it exists!).

    Steve


Locked