Any gotchas I should be aware of running pfSense on an alix 2D13?



  • posted this earlier today. I thought I had posted it under Hardware, but just noticed it's under general questions. Moving to Hardware

    I've successfully written the 2.2.4 embedded image to a 4gb CF microdrive, and it booted ok. The web interface is slow (which I expected), but nat'd traffic also seems slow (which I didn't expect. FreeBSD is supposed to be the best OS for routers, right? I did a short test or two with monowall a month or two ago before deciding to try pfSense, but I don't recall the network feeling a bit laggy as it does now).

    Anyway, the big thing bothering me right now is that the device is randomly rebooting or freezing within about 90min of booting. I don't think there is anything special about that time frame though.

    Initially, I thought adding the traffic graph widget on the landing page was killing the cpu, causing the reboots (I'm only aware of the reboots because I'm forced to log back into the web interface, the uptime and network counters have reset and the dhcp leases are empty).

    I took the traffic graph off the landing page, but I'm still getting random freezes and reboots.

    I managed to capture the serial console output this last time it froze (couldn't access the web interface, no response to dns requests and no natting) and there were roughly 75 of these:

    
    pfr_unroute_kentry: delete failed.
    
    

    I googled that, and everything seemed to point to CARP or a HA configuration, of which I'm using neither. My config is pretty stock right now with the only changes being I moved the LAN subnet to 192.168.6.0/24 and I turned on the "dns resolver returns dhcp hosts".

    I have a 3rd network port set to Opt1, but it's unconfigured and not plugged in at this point.

    So, are these kind of problems normal with an alix box?
    Should I throw my dreams of a low power firewall out the window and go back to having violently overpowered pc's sitting at my network border?


  • Banned

    @Alixy:

    So, are these kind of problems normal with an alix box?

    No. Random freezes and reboots and whacky logs -> faulty HW very likely.



  • Hello,

    pfSense is a FreeBSD based OS, that would surely also likes many other OS´s renewed code, added code,
    changing something and so on, and so I am really sure that with this hardware you could be really happy
    if you go with pfsense 2.1.5 and not higher. This box is delivering you ~80 MBit/s WAN throughput and
    if the WebGui and the widgets are not really important for you, it will do the job as a home firewall for sure.

    But as I told earlier some lines above, do you or would you really imagine or expect that Windows Server
    2012R2 is running on hardware, that was made for MS Windows Server 2003 also smooth and liquid as
    the MS Server 2003 version was doing? Or do you thing that nothing under the WebGui will ever change
    and the oldest hardware on the market will even let the newest pfSense system running liquid and/or likes
    running on soap? All three years the most of us are changing the ordanary PC hardware against newer ones.
    Likes Boards, RAM , CPUs and SSDs/HHDs related to new MS Windows 7,8,10 and so on versions.

    But owed to our Firewall hardware we expect the oldest hardware should do the job for us and this also
    likes the former 10 till 15 years for the next 20 years because we don´t want to pay only one more for
    newer hardware? Hmmm, if I am want the lucky one I spend money also for my firewall or router hardware!

    You want to set up a HA cluster and then all is doubling the price for sure, but what you are thinking
    or better why pfSense was changing from FreeBSD 8.3 to FreeBSD 10.1 and exactly to this moment the
    brand new PC Engines APU boards with 2 GB and 4 GB RAM comes to the market? And the Supermicro
    boards with Intel Atom C2x58 SoCs are also out to the market, and the pfSense store is now on top selling
    boards that are combining the best of two worlds!!!! The three miniPCIe + SIM slots and more RAM capacity
    from the PC Engines APU boards and the Power that came with the Intel Atom SoCs + AES-NI & QuickAssist
    but al peoples want to grab cheap hardware. Why?

    Did you enable the DMA mode for your Microdrive? Or is it only read able, then you would be more happy
    perhaps with a "cheaper" industrial CFCard.

    Is this Alix board sorted with 256 MB or 512 MB of RAM?
    Were you inserting mini PCI cards?

    The most peoples will not be willing to spend much money for a smooth and liquid running system
    but the hardware is there able to get, I really don´t understand this, the mostly users are saying first
    that they not really needing this or this but after buying then something the WebGui is slow, the widgets
    are slow, the throughput is slow and so on.

    Please don´t forget the Alix boards a pretty old! Something around 10 - 15 years now on the market without
    changes and upgrades, even now the APU boards are faster and more reliable to run the newest pfSense
    on it without a pain in the ass on each feature or option, given by pfSense.

    On this older boards perhaps a smaller Linux distro would even running nearly smooth or liquid.
    Perhaps you give RouterOS, IPFire, DD-WRT or OpenWRT a try or you will be able to set up a small
    snort sensor with this board. mOnOwall is not more under maintain or will be getting an update
    so it is standing still and it also would be running perhaps a little bit faster then the new pfSnese
    version over the 2.1.5 but not more.

    If you need something with an external PSU and easy to handle but also not to high pricing
    you should have a look to this one here it is able to get as a board only option to fiddle something
    together by your own or as a so called ready to go option ready assembled.
    Jetway NF9HG-2930 Thin mini-ITX Network Motherboard
    2 x miniPCIe & SIM slot & 4 x Intel GB LAN Ports & max. 8 GB RAM & 4 Core CPU
    Jetway NF9HG-2930 Intel Celeron Quad Core Fanless PC w/ 4X Intel LAN, 2GB, M350
    2 x miniPCIe & SIM slot & 4 x Intel GB LAN Ports & max. 8 GB RAM & 4 Core CPU
    Jetway Intel N2930 Network PC w/ 5X Intel LAN, 2GB, JBC200F9N-E4IN-B, ADE4INLANG
    2 x miniPCIe slots & 5 x Intel GB LAN Ports & max. 8 GB RAM & 4 Core CPU

    All without AES-NI & Intel QuickAssist, if this would be a urgent need you could also go with the
    SG-2220 unit from the pfSense store, if this is only for a standard set up and for you at home
    it should running for years for you.



  • I'm using 2D13 for about 5 years already, since 1.2.3 release. No hardware issues so far :)
    Considering APU for a new build.


  • Banned

    Yes, these things are really reliable if you use them with their (RAM/CPU) limitations in mind. Even then, out of RAM does not result in a reboot on sane HW (likely webgui or some big service will get killed); likewise, taxing the cpu in insane ways (generating DH 4096 is a good test here; will take an hour or so on Alix) does not cause any freezes/reboots if the HW is sane.



  • @AndrewZ:

    I'm using 2D13 for about 5 years already, since 1.2.3 release. No hardware issues so far :)

    Are you still running 1.2.3 on your 2d13 or have you upgraded? If so, what version are you running now?

    @AndrewZ:

    Considering APU for a new build.

    Holy crap! I bought my alix board about 16 months ago and just saw the stats on their apu boards. 3xGige, 64bit and 2gb for the same price I paid for my 2d13. If I was looking at these today, I'd probably jump on this, but until I figure out why my 2d13 is having these issues, I'm a little wary of pc engines hardware.

    The other weird thing: Today is the first I've heard of amd's apu, and looking at pc engines pic for their apu1d board, I see three long chips in the middle, but they're not that large and nothing I'd guess as the cpu. Is it on the other side of the board?

    To be fair, I'm a software guy not a hardware guy so I was pretty shocked looking at the tiny motherboard for apple's latest insanely thin macbook.



  • @Alixy:

    Are you still running 1.2.3 on your 2d13 or have you upgraded? If so, what version are you running now?

    I'm on 2.2.4 now and used all the intermediate versions including some betas.



  • @doktornotor:

    Yes, these things are really reliable if you use them with their (RAM/CPU) limitations in mind.

    If it is a hardware problem, I really hope it's the power supply or the microdrive. The only variables I have right now are

    • power supply

    • alix 2d13

    • cf microdrive

    • pfsense

    I bought the alix and power supply (and aluminum case for the alix) from an alix reseller in the US in the beginning of 2014, but haven't had time to mess with it until now (haven't used it at all, but I'm a bit far out of any possible warranty to send it back as a DOA board either).

    The two things making me a little nervous are

    • the only other OS I tried on the alix seemed to run ok, or at least longer than an hour.

    • Now that I determined that the usb bus on my old laptop was causing broken images to be written to the microdrive, the microdrive has been pretty solid

    Still, I have to admit that these microdrives are 4gb hitachi's and I bought then in a 4-pack from a website that sells old/discontinued hardware and that was in 2008. On one hand, I only ever used one of them, so the one I'm using now is "pristine" (or at least, unused sitting on a shelf since I got it) but ya, maybe it's the card.

    To control for that variable and since CF cards are stupid cheap now, I'm getting a $15 8gb sandisk cf card today. When that arrives, I'll write the embedded image, do a read test to verify the image and see if I still get random alix crashes.



  • The other weird thing: Today is the first I've heard of amd's apu, and looking at pc engines pic for their apu1d board,

    Don´t be running in the next trap please, have a near look to that APU board, one is coming woth 2 GB RAM
    and the other comes with 4 GB RAM for only some $ more, buy the one with 4 GB and enjoy it!  ;)

    I see three long chips in the middle, but they're not that large and nothing I'd guess as the cpu. Is it on the other side of the board?

    The CPU is upside down related to the new thermal design and colling option that the case is taking much
    of the thermal heat away.  :-*

    To control for that variable and since CF cards are stupid cheap now, I'm getting a $15 8gb sandisk cf card today. When that arrives, I'll write the embedded image, do a read test to verify the image and see if I still get random alix crashes.

    Would be perhaps much better to set up there something like a Linux OS like CentOS together with
    CACTI and MRTG or plain a Asterisk PBX VOIP server for the DMZ of your pfSense!

    the only other OS I tried on the alix seemed to run ok, or at least longer than an hour.



  • @BlueKobold:

    The other weird thing: Today is the first I've heard of amd's apu, and looking at pc engines pic for their apu1d board,

    Don´t be running in the next trap please, have a near look to that APU board, one is coming woth 2 GB RAM
    and the other comes with 4 GB RAM for only some $ more, buy the one with 4 GB and enjoy it!  ;)

    I'd rather not buy anything. The 2d13 should be fine for what I'm trying to do with it.

    @BlueKobold:

    But as I told earlier some lines above, do you or would you really imagine or expect that Windows Server
    2012R2 is running on hardware, that was made for MS Windows Server 2003 also smooth and liquid as
    the MS Server 2003 version was doing?

    I'll skip the obvious joke about windows performance, but there is a key difference between the situation you describe and network routing: 100mb/s (which is all the alix2d13 has) is also 20 year old technology. It was top of the line for ethernet back then, but home broadband is only now catching up to that speed (at least in the US. Three years ago, I was getting 15mb/s on cable at my home and a friend lived in paris was getting 115mb/s which made me super jealous).

    Also the underpowered 2d13 is faster than the hardware routers and firewalls used 20 years ago. So yes, for the simple task of nat/firewall, it is absolutely reasonable to expect this to be enough (though I would like more system memory) and it does seem to be enough given the number of people using this configuration (and yes, if I wanted all the bells and whistles and to support 200 people on pc's, I might need a bit more hardware, but this is just nat/firewall/dhcp/dns for a single user and, currently, just one computer).

    The real problem I'm having is my inexperience with *bsd is making diagnosing the issue difficult.

    For example, just trying to track how often it reboots: I was logging the serial console output, and trying to find a line with a timestamp that occurs every reboot. I started grep'ing with "starting on vr0" searching for lines like this:

    
    Aug  8 22:51:00 php-fpm[73373]: /rc.newwanipv6: rc.newwanipv6: Info: starting on vr0
    
    

    but it turns out that occasionally doesn't pop up on the serial console after a reboot. 
    Then I tried "Starting all packages", but that also doesn't occur every reboot.
    Now I'm doing this:

    
    egrep '^PC Eng|^Aug'
    
    

    which grabs the first line after a reboot and line line starting with august, but that requires manually tweaking the output as well.

    Using that, (and grabbing the first Aug line after each PC Engines line) gives the following reboots for the last 12 hours:

    
    Aug  8 22:51:00 php-fpm[73373]: /rc.newwanipv6: rc.newwanipv6: Info: starting on vr0.
    Aug  8 23:07:25 php-fpm[74168]: /rc.newwanipv6: rc.newwanipv6: Info: starting on vr0.
    Aug  8 23:53:19 php-fpm[6141]: /rc.newwanipv6: ROUTING: setting default route to 10.0.0.1
    Aug  9 00:28:42 php-fpm[77977]: /rc.newwanipv6: rc.newwanipv6: Info: starting on vr0.
    Aug  9 01:33:37 php-fpm[6001]: /rc.start_packages: Restarting/Starting all packages.
    Aug  9 01:49:05 php-fpm[75297]: /rc.start_packages: Restarting/Starting all packages.
    Aug  9 02:56:32 php-fpm[76896]: /rc.start_packages: Restarting/Starting all packages.
    Aug  9 04:50:36 php-fpm[83484]: /rc.start_packages: Restarting/Starting all packages.
    Aug  9 05:22:09 php-fpm[68175]: /rc.newwanipv6: rc.newwanipv6: Info: starting on vr0.
    Aug  9 07:28:57 php-fpm[6575]: /rc.newwanipv6: ROUTING: setting default route to 10.0.0.1
    Aug  9 08:05:39 php-fpm[5761]: /rc.newwanipv6: ROUTING: setting default route to 10.0.0.1
    Aug  9 10:09:39 php-fpm[82244]: /rc.newwanipv6: ROUTING: setting IPv6 default route to [an actual ipv6 address]
    
    

    12 reboots in 12 hours. The last only made it 2 hours because the box locked up solid (with no errors on the serial console… that was disconcerting) and I had to power cycle it once I noticed it was frozen.

    @BlueKobold:

    Or do you thing that nothing under the WebGui will ever change
    and the oldest hardware on the market will even let the newest pfSense system running liquid and/or likes
    running on soap? All three years the most of us are changing the ordanary PC hardware against newer ones.
    Likes Boards, RAM , CPUs and SSDs/HHDs related to new MS Windows 7,8,10 and so on versions.

    I don't expect the webgui to be fast because I'm the only user and I don't use it often (or at all, once I've assimilated the xml dtd for the config).

    But again, I think you are confusing two different sets of requirements. We change pc's because the underlying task we're asking the pc to perform has changed radically.

    100mb/s routing/nat'ing/firewalling hasn't changed much in the last 20 years, which is why (the equivalent of) 20 year old hardware should be fine for the task at hand.
    I'm not trying to run with all the bells and whistles.
    I'm trying to run with all the bells and whistles turned off.

    @BlueKobold:

    You want to set up a HA cluster and then all is doubling the price for sure, but what you are thinking

    This is where I think we're hitting a bit of a language barrier. To be clear:
    I do not want to run HA. (even if I did, I don't have a second device to fail over to)
    I do not want to run CARP. (I don't even know what carp is)

    I only mentioned HA and CARP because the first error I caught on the serial console before a crash-boot was "pfr_unroute_kentry: delete failed." and the only topics I found through Google for that error was on HA and CARP.

    @BlueKobold:

    Did you enable the DMA mode for your Microdrive?

    This is the first I'm hearing of any DMA setting when using a compact flash card. If I was plugging the microdrive into the IDE port with a CF-IDE bridge, then DMA might come up, but is there a DMA setting to set when it's plugged into the CF slot? Where/how would I set that? (tried googling, came up empty for DMA on CF).

    Grepping the serial output for DMA, I do see UDMA2 as part of the capabilities announcement:

    
    ada0 at ata0 bus 0 scbus0 target 0 lun 0
    ada0: <hms360404d5cf00 dn4oca2a="">CFA-4 device
    ada0: Serial Number N2NW57HA
    ada0: 33.300MB/s transfers (UDMA2, PIO 16384bytes)
    ada0: 3906MB (7999488 512 byte sectors: 16H 63S/T 7936C)
    ada0: Previously was known as ad0</hms360404d5cf00> 
    

    I also see this:

    
    atapci0: <amd cs5536="" udma100="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 15.2 on pci0
    ata0: <ata channel="">at channel 0 on atapci0
    ata1: <ata channel="">at channel 1 on atapci0</ata></ata></amd> 
    

    which is the announcement for the controller, but there is nothing else in the logs for ata0, ata1 or atapci0 (because there is nothing plugged into the IDE port or pci slot).

    @BlueKobold:

    Is this Alix board sorted with 256 MB or 512 MB of RAM?

    This is the 256mb version. I've seen mention of a 128mb version, so I'm wondering if 256mb was the biggest available when I bought the board (early 2014).

    @BlueKobold:

    Were you inserting mini PCI cards?

    I am not using any pci cards.

    @BlueKobold:

    The most peoples will not be willing to spend much money for a smooth and liquid running system
    but the hardware is there able to get, I really don´t understand this, the mostly users are saying first
    that they not really needing this or this but after buying then something the WebGui is slow, the widgets
    are slow, the throughput is slow and so on.

    I really only care about network performance, but single user web interfaces don't require that much hardware. Php is a bit slow, and most complaints I've heard about the slowness of the pfsense web interface are more about php as the backend. Ruby would also be slow. For the work I've done on raspberry pi's, python is surprisingly fast (I initially tried to use ruby because my background is c/c++/java/perl/ruby, but I've been getting more into python because of it's speed on low power devices).

    Since you asked, my core goal is a low power (lowest wattage possible, no fans, longest lasting) NAT'ing firewall that will let me do the same dns tweaks that I did on my old firewall, but will give me a 3rd network interface for a DMZ. I don't want wifi built into the firewall and I don't need HA. I would like it to log bandwidth usage but I realize that all I need there is a process on the firewall to shoot network interface stats to another server at regular intervals.

    I will say that the 5 gige ports on the boards you linked to are attractive (because it'd be nice to have gige between the LAN and DMZ), but I need to look at the power supply for that. Price also seems a bit steep, but that's more because I spent $150ish on this alix+case+power just over a year ago.

    @BlueKobold:

    Would be perhaps much better to set up there something like a Linux OS like CentOS together with
    CACTI and MRTG or plain a Asterisk PBX VOIP server for the DMZ of your pfSense!

    I'm having a hard time parsing this due to language barrier, but I think you're trying to say I'd be better off buying the hardware you linked for pfsense and using the alix to run cacti or mrtg.

    If that is what you meant, I'm confused by this because earlier you were arguing that the pfsense webgui needs more processing power. If I had two boxes, one much faster than the other, and I needed a firewall and a server monitor (cacti/mrtg), I'd put the firewall on the weaker of the two (assuming it could handle the max possible bandwith). Why do you think the web config gui for any nat/firewall (that, in most cases is only used by one person and rarely at that) needs more hardware than a status monitor that tends to get hit by multiple people and also used on wall status monitors? (to be fair, I wouldn't be hitting it with multiple users, but I might throw it on a old monitor as a live status monitor).

    Honestly, you've caused me to consider why I think cacti/mrtg needs bigger hardware (but it's been more than a year since I've worked with either, so if there are valid reasons, I've forgotten them). I seem to recall one of those (ganglia?) could lose data if it wasn't ready to respond to a status report, but that's probably if you've got the clients firing over udp.

    You've inspired me to test my gut feeling that cacti/mrtg needs big hardware so I'm going to try throwing cacti on a raspberry pi and see how that goes. Thanks!



  • The new CF card did not help and it's still crashing randomly in the 10-75min time frame.

    Going to try downgrading the bios to 0.99h and possibly upgrading back to 0.99m.

    If both of those still crash, I'll start a new post with crash dumps.


  • Banned

    Uhm… Is the HW new, or what? This definitely is not pfSense issue at all.



  • I bought the board and power supply from a mini-box.com in the beginning of 2014, but haven't had a chance to set it up til now (older electricity wasting firewall still worked, so it wasn't a priority).

    I've run memtest86 (linked from the software section of pcengines alix page: http://www.pcengines.ch/alix.htm) for about 75min, so I'm fairly certain it's not a memory issue, but I have no idea what other test software I can/should run.

    I've seen suggestions in old threads about swapping out the power supply, so going to give that a shot as well.



  • Just to update this, I never tried downgrading the bios (which turns out to be good, since there's a note in the 0.99m bios that the 0.99h can't write bios changes on a newer flash chip, which I assume I have).

    However, I did try a supply swap. The board came with a 15v, 1.1a supply, but I had a 12v,0.5a supply from an unused wifi hub. PC Engines support says the board needs about 6w, so the 12v,0.5a is right at the limit (when using a flash CF card. I'm guessing a microdrive CF may push the wattage req's too far above that supplies abilities).

    The board became more stable (running for at least a day before crashing). I swapped back to the 15v,1.1a to verify that it was indeed the supply that was causing the hourly crashes.

    Irritatingly, the board is now more stable with that supply as well. One odd thing, it'll run for days without issue, but if I cut power to it (again, it's running nanobsd, so shouldn't need to carefully shut down), then the next few boots will be highly unstable (rebooting two or three times (or much worse, locking up without rebooting) over the next hour before it stabilizes).

    I have two other devices that run on 12v AC/DC converters as well (both plugged into different outlets, and the alix is plugged into a third outlet) that are also having stability issues, but all other electronics in my home have no issues.

    While all three could be flakey supplies, the other two devices (lcd monitors both running off the same rca output) seem to fail at exactly the same time (but a third device also attached to said RCA output has no issues).

    Long story short, now wondering if there's some odd voltage fluctuations in my building that most devices are able to handle, but these converters can't.

    Going to get a voltage regulating UPS and see if that resolves the situation (and I bought a few AC/DC converters as well to try swapping all of these out and see if things improve).

    One final point: I'm a little horrified to see how cheap msata ssd drives are now (30gb msata ssd for 2x what I paid for a 8gb cf). Despite my response to BlueKobold above that the alix should be able to handle the routing without issues (my cable is still under the 100mbit limit of the alix network hardware), if I do end up replacing the alix, I'm leaning towards amzn.com/B00LGCTQDS (because it's got at least three network ports and it's the cheapest I could find that ships directly from amazon).

    However, I notice it's a dual core atom vs all the ones BlueKobold mentioned are quad core celerons. Both cpus are 64bit, but otherwise I don't know how these compare in performance (though the celerons are have a higher clock cycle and have double the cores, so that may be indicitive. Still shipping directly from amazon is a big deal for me because it means any returns are handled by amazon, which has an awesome return policy).

    My only real concern with switching to a more powerful box is that, while I subscribe to the concept that a firewall should be an "appliance" (changes relatively infrequently, can be power cycle'd at will without having to do any "shutdown" procedure, and is otherwise rock solid), I'd be extremely tempted to run non-firewall stuff on the firewall because of the unused cpu cycles, etc.


  • Banned

    @Alixy:

    However, I did try a supply swap. The board came with a 15v, 1.1a supply, but I had a 12v,0.5a supply from an unused wifi hub. PC Engines support says the board needs about 6w, so the 12v,0.5a is right at the limit (when using a flash CF card. I'm guessing a microdrive CF may push the wattage req's too far above that supplies abilities).

    The board became more stable (running for at least a day before crashing). I swapped back to the 15v,1.1a to verify that it was indeed the supply that was causing the hourly crashes.

    Hmmm… We have bunch of these powered by 24V POE with some ~50m of cabling from the switch, so just about the limit of what the board takes. Never had issues with that.



  • Hmmm… We have bunch of these powered by 24V POE with some ~50m of cabling from the switch, so just about the limit of what the board takes. Never had issues with that.

    Huh?

    The specs I see from  PC Engines say:

    "Power supply                7  to  20V  DC,  about  3  to  4W  at  Linux  idle ,  peak  about  6W  without
                                            miniPCI cards and USB devices. Suggest a 18V / 15W
                                            supply.
    "
    The OP was originally using a 12V 0.5A supply => 6W (at best) so I would rate this as potentially marginal all else being equal.

    Your POE setup is likely overvoltage at 24V). Because you have a longer run (50m) for the current required (6W/24V=0.25A) your final voltage at the device is dropped to ~ 21.8V, still high but much closer to spec tolerances.

    As far as your POE being stable in the long run, if it has the output capacity the device needs (in W) I'd bet it would be.  If you can operate with higher voltages => lower current for the same power you get better capability of surviving "blips" in power and your power supplies tend to run cooler.

    The basic issue is to supply the device with the power it needs according to the manufacturer's specs.
    12V@.5A=>6W - Marginal
    12V@1.0A=>12W - Good
    18V@.83A=>15W - Good with room for growth.

    All said I definitely agree with the OP's suggestion to try a UPS if only to eliminate a noisy device (big motor, AC unit, etc) on their power lines.



  • Just try a new PSU with something like 12-20V DC and 1A current.
    Use a big capacitor 1000µF/32V and a small one (1nF or so) in parallel on the supply lines to bolster current and cut noise.

    Depending on your setup it could prove beneficial to use UTP cables. This cuts hum-loops which can do all sorts of nasty things.


Log in to reply