Overkill or Under Qualified?



  • Hey Guys,

    First off, this'll be my first build. I started looking into pfSense when my router started having issues. The issues being randomly dropping the lease time. It'll have 20 hours left on it, and it just drops. A friend who I have high regards for, suggested I build a pfSense box. Here's what he gave me to work with;

    CPU: AMD - Ryzen 3 1200 3.1GHz Quad-Core Processor
    MOBO: ASRock - AB350 Gaming-ITX/ac Mini ITX AM4 Motherboard
    RAM: G.Skill - NT Series 8GB (2 x 4GB) DDR4-2133 Memory
    NIC: Intel Quad Port Gigabit

    I've always been a huge fan of VMWare, being a former contractor there, I've learned quite a bit about ESXi. I wanted to give it a shot, however I'm on a bit of a budget. I'm trying to get my build below $500 in total. I abuse my internet like crazy, I push a ton of traffic between services I hosted, my own projects (Such as Archiving with IA), and other things. My question is;

    ESXi: Yay or Nay?
    If Yay, will the build be powerful enough?
    Secondly, how does vmotion work with pfSense?



  • Nay, waste of resources, money and probably won't even work as well as a basic Intel build or ARM box from Netgate.

    To get somewhere helpful, we'll need information:

    • Uplink type (PPPoE?, static?, ethernet with DHCP?, DSL?, Coax?)
    • Up/Down speeds
    • Are you going to run IDS/IPS
    • Are you going to run VPN (and if so, which type)
    • How many subnets do you think you need, and are those going to be VLANs or separate interfaces, and will they need to be routed a lot

    Say you wanted a gigabit WAN link with some sort of NTU/G.PON/Coax/Bridge consumer connection, and you have one LAN, and you just wanted to do gigabit internet with no special services, then you can get away with a dual core pentium, 2GB of RAM and a USB drive. Make sure you have two Intel network ports and you'll be fine. Probably gonna cost you between 150 and 250. SG-3100 would work fine. Qotom box would work fine. Some used office PC would work fine.

    If you wanted multiple OpenVPN instances to bridge your full wan to some sort of endpoint elsewhere, you'd be looking at an i7 or Xeon with at least 4 real cores and a high clock, maybe 4GB RAM, and again, a USB drive, CF, SATA HDD, SSD or whatever will be fine, and again, 2 Intel NICs will do.
    Probably gonna cost you between 450 and 650. A high-end C-series based SG would do, an i7 Qotom box would do, a build based on some good parts would do (i.e. random SuperMicro board, CPU of choice, basic value Kingston/Crucial RAM, 16GB SSD/USB drive.

    If you are going to do a lot of logging, caching, IDS/IPS, add more RAM and more disk space to the above receipe.

    Don't get the latest and greatest from AMD, it's not tested well, and probably not going to perform as well compared to a same priced Intel or ARM setup. Also, if you are going to pull in ESX, Xen, KVM or another hypervisor, add one core with HT or two real cores and 512MB more RAM, and a second drive.



  • @johnkeates:

    Nay, waste of resources, money and probably won't even work as well as a basic Intel build or ARM box from Netgate.

    To get somewhere helpful, we'll need information:

    • Uplink type (PPPoE?, static?, ethernet with DHCP?, DSL?, Coax?)
    • Up/Down speeds
    • Are you going to run IDS/IPS
    • Are you going to run VPN (and if so, which type)
    • How many subnets do you think you need, and are those going to be VLANs or separate interfaces, and will they need to be routed a lot

    Say you wanted a gigabit WAN link with some sort of NTU/G.PON/Coax/Bridge consumer connection, and you have one LAN, and you just wanted to do gigabit internet with no special services, then you can get away with a dual core pentium, 2GB of RAM and a USB drive. Make sure you have two Intel network ports and you'll be fine. Probably gonna cost you between 150 and 250. SG-3100 would work fine. Qotom box would work fine. Some used office PC would work fine.

    If you wanted multiple OpenVPN instances to bridge your full wan to some sort of endpoint elsewhere, you'd be looking at an i7 or Xeon with at least 4 real cores and a high clock, maybe 4GB RAM, and again, a USB drive, CF, SATA HDD, SSD or whatever will be fine, and again, 2 Intel NICs will do.
    Probably gonna cost you between 450 and 650. A high-end C-series based SG would do, an i7 Qotom box would do, a build based on some good parts would do (i.e. random SuperMicro board, CPU of choice, basic value Kingston/Crucial RAM, 16GB SSD/USB drive.

    If you are going to do a lot of logging, caching, IDS/IPS, add more RAM and more disk space to the above receipe.

    Don't get the latest and greatest from AMD, it's not tested well, and probably not going to perform as well compared to a same priced Intel or ARM setup. Also, if you are going to pull in ESX, Xen, KVM or another hypervisor, add one core with HT or two real cores and 512MB more RAM, and a second drive.

    Uplink: Static IP
    Speed: 1Gbit Up/Down Unmetered
    IDS/IPS: No clue, but if I were a betting man, keeping options open would be it.
    VPN: Yes, Haven't decided on which yet. Frankly, I'm not 100% sure on them all. I'm trying to secure my network as much as possible. I guess I should build a network diagram.
    Subnets: Well, right now I have about 5 or 6. However, I can drop that down to 3 easily. I segment my IPs, I have ACLs to keep guests off my network and from accessing my storage arrays.

    I have some Intel Xeon E52620s, 2650s, 2670s, and 2690s at my disposal. I was trying to keep this small because I live in an apartment, and I'm about at the end of the 300ft distance for copper. I was trying to keep the costs down.

    Based on the feedback from johnkeates. An Alternative build

    CASE: https://www.newegg.com/Product/Product.aspx?Item=N82E16811139022
    MOBO: Some Supermicro Motherboard
    CPU: Intel E5-26xx Processor (6 core / 8 core)
    RAM: 64GB DDR4 Memory
    NIC: I have Dual / Quad Intel NICs
    SSDs: 100GB / 200GB
    HDD: 3TB for Logs, which will be uploaded to my Google Drive. I'm not entirely sure what would be better, server grade CPUs or a High-End Gaming one.



  • U want run pfsense on a VM?  Is this box experimental/lab not production?



  • For your needs, an E3-level Xeon, 4GB of RAM and 100GB of log storage is enough. So what you have selected at this point with the E5 is overkill but will definitely work. I'd suggest virtualising but passing the NIC to pfSense as a PCIe device (or use VF if it's supported).

    Server CPU is better than gaming CPU. This is because of the workload differences.



  • @johnkeates:

    For your needs, an E3-level Xeon, 4GB of RAM and 100GB of log storage is enough. So what you have selected at this point with the E5 is overkill but will definitely work. I'd suggest virtualising but passing the NIC to pfSense as a PCIe device (or use VF if it's supported).

    Server CPU is better than gaming CPU. This is because of the workload differences.

    CPU: https://www.newegg.com/Product/Product.aspx?Item=N82E16819117790
    MOBO: https://www.newegg.com/Product/Product.aspx?Item=N82E16813183013
    RAM: https://www.newegg.com/Product/Product.aspx?Item=9SIA98C5JA9264

    The core pieces I want to spend around $500 - $600. The Case / PSU are cheap enough that I can get them any time I need too. Would it be best if I did a baremetal install, and left virtualising out of the picture?  I'm trying to keep the build small, but powerful. I've provided my Network Diagram, Note that ESXi-03 and 04 aren't finished yet. All of the servers have 10G Connections, I plan on pushing 10G in the cluster of the Storage Servers & ESXi Servers, it'll be limited to just those, and won't hit the network. The SG300-10 is L3 right now.

    @SammyWoo:

    U want run pfsense on a VM?  Is this box experimental/lab not production?

    This is going to be a home router. I need it to be beefy, and upgrade-able. I'm trying to decide, which would be better, VM or Baremetal Install. I host quite a few ESXi Servers, and services at my place. A lot of the guys I attend college with, can't afford to purchase extra computers, or rent them for educational reasons. I built a few spare servers, and let them have access to them for the duration of their schooling. I pay a cheap price for my connection, and my power bill is less than $100 / month. I need something beefy that can take a beating on a constant basis, and continuous beating.




  • That'll work fine. I suggest you don't virtualise it since playing with virtual stuff while also running your network on top of it is going to lead to outages. Also, if you need to upgrade later on, it will probably end up being much, much different. In a few years we might get good QaT, DPDK and other fancy stuff, so instead of upgrading the hardware, a software upgrade will get you more performance.

    Regarding VPN, most connections are limited to about 60% of WAN speeds, mostly due to the providers not having anything better to offer. I would not recommend running everything behind a remote VPN all the time, those services are basically one big man-in-the-middle attack. Doing it for traffic you don't care about or traffic that you know is encrypted well (not talking about the tunnel here, talking about the application protocol, i.e. HTTPS, IMAPS, S-SMTP, SSH) is fine, but you may not want to use it for normal applications.



  • @johnkeates:

    That'll work fine. I suggest you don't virtualise it since playing with virtual stuff while also running your network on top of it is going to lead to outages. Also, if you need to upgrade later on, it will probably end up being much, much different. In a few years we might get good QaT, DPDK and other fancy stuff, so instead of upgrading the hardware, a software upgrade will get you more performance.

    Regarding VPN, most connections are limited to about 60% of WAN speeds, mostly due to the providers not having anything better to offer. I would not recommend running everything behind a remote VPN all the time, those services are basically one big man-in-the-middle attack. Doing it for traffic you don't care about or traffic that you know is encrypted well (not talking about the tunnel here, talking about the application protocol, i.e. HTTPS, IMAPS, S-SMTP, SSH) is fine, but you may not want to use it for normal applications.

    Well, the Jumpbox is there to reduce the amount of open ports on the network. Anything that has to reach out to the internet will have https, or it will only work by accessing the jumpbox. I need to build a better jumpbox, something people can remote to via VNC or something like that so they can view the web portals for ESXi. I learned my lesson to not leave those open to the world. I paid for a Dedi, and put ESXi on it. So many SSH Attempts, I was permanently locked out of my OS, lol.

    As to the build, I'll start picking up gear here shortly. Rent is coming due :( Once I have all the parts, and everything is online, I'll come back and post. Thanks so much john, and Sammy!



  • Instead of using the jumpbox for everything I'd suggest using OpenVPN.



  • @johnkeates:

    Instead of using the jumpbox for everything I'd suggest using OpenVPN.

    Well, the way I was doing it is; VPN -> Jumpbox. I'm trying to reduce the amount of management I have to do. I'll hit you up with a pm once I've got everything, john. This has gotten off-topic a bit, and the goal of the thread was reached.



  • Based on the feedback from johnkeates. An Alternative build

    CASE: https://www.newegg.com/Product/Product.aspx?Item=N82E16811139022
    MOBO: Some Supermicro Motherboard
    CPU: Intel E5-26xx Processor (6 core / 8 core)
    RAM: 64GB DDR4 Memory
    NIC: I have Dual / Quad Intel NICs
    SSDs: 100GB / 200GB
    HDD: 3TB for Logs, which will be uploaded to my Google Drive.

    If you're Running this as a VM under ESXi (or any hypervisor) a single HDD of any kind will make you sad. even for a handful of lab VMs I would recommend a RAID10. If you have => 5.5 vCenter then you can use the SSD as read cache (configured per VMDK in vCenter) otherwise you would be stuck using for swap (total waste) or as a small datastore is which case I would spend less on the HDD and more on the SSD and get the biggest one you can.

    As for running running your home gateway/router as a VM, don't. Especially if your using vlans. Its just a pain in the arse. You get stuck changing you PC IP and switch port all the time to fix little things like needing to reboot your host.

    For your home gateway/router, just spend the $$ and build or buy a separate router. Also if you don't mind getting your hands dirty in CentOS and need the best possible speed you could take a look at the new tnsr platform. but that's a whole nother animal. ;D

    Edit: added closing quote tag.


  • LAYER 8 Global Moderator

    "As for running running your home gateway/router as a VM, don't. Especially if your using vlans."

    Huh??  Running pfsense on a vm with multiple vlan is no different then if single network.. As long as you have a switch that handles vlans, and know how to setup the switching in your VM host its really quite simple and easy to run/manage.

    A simple 4095 setting on your vswitch in esxi for example allows you to tag any vlans you want to pfsense - which you can then just setup vlans in pfsense.  Or you could use port groups on your vswitch with the tag of the vlan you want to pass to the vmnic you connect to pfsense, etc.

    I ran like this for years on multiple vlans on esxi, even once I moved to pfsense on hardware I still run vlans into different VMs and even run a downstream pfsense VM via a transit vlan from the edge sg4860.. My sg300s are in L3 mode, but as of current only using L2.. I just put them in L3 for future lab/testing work, etc.  They are more than happy to function as L2 when in L3 mode, etc.  You can use both at the same time where some could be routed at your sg300, and other vlans are just L2 and routed via pfsense, etc.



  • @johnpoz:

    "As for running running your home gateway/router as a VM, don't. Especially if your using vlans."

    Huh??  Running pfsense on a vm with multiple vlan is no different then if single network.. As long as you have a switch that handles vlans, and know how to setup the switching in your VM host its really quite simple and easy to run/manage.

    A simple 4095 setting on your vswitch in esxi for example allows you to tag any vlans you want to pfsense - which you can then just setup vlans in pfsense.  Or you could use port groups on your vswitch with the tag of the vlan you want to pass to the vmnic you connect to pfsense, etc.

    I ran like this for years on multiple vlans on esxi, even once I moved to pfsense on hardware I still run vlans into different VMs and even run a downstream pfsense VM via a transit vlan from the edge sg4860.. My sg300s are in L3 mode, but as of current only using L2.. I just put them in L3 for future lab/testing work, etc.  They are more than happy to function as L2 when in L3 mode, etc.  You can use both at the same time where some could be routed at your sg300, and other vlans are just L2 and routed via pfsense, etc.

    I guess what I was getting at is the chicken egg situation. vCenter is on vlan 50 pc is on vlan 2 and you need to reboot your only host. well then you have to plug you pc into a vlan 50 port or login to the switch to reconfigure your PC port (if its on your PCs vlan) startup your pfSense VM and go back and put you PC back in its normal vlan.

    Yeah You can do it and it works, but IF your using vCenter automatic startup of VMs is unsupported. If its just a stand alone host just make sure you don't have any dependence on routing for your VMs to boot up. One example would be a routed SAN. This is never a good idea but I have seen people do it in the FreeNAS forums. (lots of odd network setups over there)



  • @curtisgrice:

    Based on the feedback from johnkeates. An Alternative build

    CASE: https://www.newegg.com/Product/Product.aspx?Item=N82E16811139022
    MOBO: Some Supermicro Motherboard
    CPU: Intel E5-26xx Processor (6 core / 8 core)
    RAM: 64GB DDR4 Memory
    NIC: I have Dual / Quad Intel NICs
    SSDs: 100GB / 200GB
    HDD: 3TB for Logs, which will be uploaded to my Google Drive.

    If you're Running this as a VM under ESXi (or any hypervisor) a single HDD of any kind will make you sad. even for a handful of lab VMs I would recommend a RAID10. If you have => 5.5 vCenter then you can use the SSD as read cache (configured per VMDK in vCenter) otherwise you would be stuck using for swap (total waste) or as a small datastore is which case I would spend less on the HDD and more on the SSD and get the biggest one you can.

    As for running running your home gateway/router as a VM, don't. Especially if your using vlans. Its just a pain in the arse. You get stuck changing you PC IP and switch port all the time to fix little things like needing to reboot your host.

    For your home gateway/router, just spend the $$ and build or buy a separate router. Also if you don't mind getting your hands dirty in CentOS and need the best possible speed you could take a look at the new tnsr platform. but that's a whole nother animal. ;D

    Edit: added closing quote tag.

    I have ESXi 6.5+, being friends with VMWare employees has its perks. I've thought about tossing it into a VM, but that's more complicated. I'm going to install it baremetal.


Log in to reply