Stepping up my game. My plan, and any helpful advice requested.

  • So, for years, I have just strung by like most other sheep on this planet with the standard 4 port user class 4 port wireless router ( which at least I was running dd-wrt on it ) we are all familiar with in some way shape or form.

    I have decided that it is time to step up my game. I am pretty IT / IT security literate, so I know this isnt beyond my knowledge. I am just looking for any helpful advice before i put in the time and invest some money in hardware.

    I want to revamp my entire security structure. First thing first, was to get rid of the crappy 4 port router.

    I went with

    Ubiquiti EdgeRouter X Advanced Gigabit Ethernet Routers ER-X 256MB Storage 5 Gigabit RJ45 ports for the router.

    And for wireless.

    Ubiquiti Networks Unifi 802.11ac Dual-Radio PRO Access Point (UAP-AC-PRO-US)

    This will take care of the in-side needs of the network.


    I have a couple ways I can accomplish this next step. Either way, it will require me to buy a beefy new computer. Oh Darn.

    I will put together a box that is capable of running multiple beefy VM's. so, high on ram / disk pace / cpu, 32 GB of RAM.  If any one can recommend a good ATX board for this. AMD preferably to reduce cost, that has at least 2 LAN interfaces on it. and a good number of expansion slots and SATA hookups, I would like to hear any recommendations. Sa I said. I kind of want this box to be fairly beefy.

    From this point, I can go with either a windows or linux option

    Windows server would be fairly easy. set up server, install vm workstation, run VM's as needed.  VM's needed would be of course. Pf Sense, and a Snort box. Also would likey have to serve as a SAN device.


    Build my own ESXi setup under linux. and run it all that way. I am not incredibly linux or esxi literate. I know enough to get buy. Thats about it.

    either way. once again, pf sense, snort, SAN,

    outside internet plugs into said box dedicated to my pf sense interface. edge router plugs into my outbound interface. I will need a virtual interface here to be able to do a port span, so I can port all traffic to my IDS instance.

    Thoughts? Tweeks? Suggestions?

  • LAYER 8 Global Moderator

    "Build my own ESXi setup under linux"

    huh?  esxi does not run under linux.  It is a type 1 hypervisor.. So no windows or linux needed here.  You can run whatever vms you want be linux, windows, freebsd, etc. on that.. So sure pfsense could run on this VM host.

    Confused why you got the edge router?  Why not just use pfsense be it running on hardware or vm for your networks router/firewall.

    You don't really need another router if your going to run pfsense.  You would want a smart/managed switch sure.. But you don't really need another router.. Good thing the edge router x is only $50.. But don't see what use its going to be if your going to be using pfsense?

    "I will need a virtual interface here to be able to do a port span, so I can port all traffic to my IDS instance."

    huh?  You do understand that pfsense can be your IPS/IDS… so not sure why you think you need to span a port..

    You want to draw up this network and we can discuss best practices, different options for setting it all up, etc.

    Your AP great choice.. Pfsense great choice.  Running it on VM.. I like it - do it myself! ;) on esxi..

    What switch are you looking at?  What kind of budget you looking at?  How many ports you going to need?

  • I know esxi is wondows only / vmwares solution. Bad terminology. But some form of open source hypervisor.

    As for pfsense. I dont know a ton about it no. Which is why i want to play with it. It has been recomended by many people i trust and respect, but I have little knowledge of it myself. But regardless. I want to setup a secondary snort server for my own purposes. And I want it to get the firehouse.

    Do you know if pfsense supports XFF off the top of your head?

  • I think you might have a misconception of what ESXi is.  As John said, it's a Type 1 hypervisor, closed source developed by vmware.  As such, it allows you to run Linux and Windows VMs (and more), but it's not reliant on any other OS underneath it.  The hypervisor is usually installed on a SD card or a USB drive attached directly to the motherboard.

    Normally I'd be all for suggesting that you test out something before you put it into production (even if 'production' is your own home), but rest assured that pfSense doesn't need to have its hand held.  I've also used my fair share of Ubiquiti equipment, but I'd not a huge fan of their routers.  APs and Switches are great and good choice on yours.

    My suggestion would be to dive in on pfSense.  If you're already planning on building a server for ESXi, you might as well run pfSense in a VM and be done with it.  I'd return that edge router while you're at it!  I'd also suggest getting a Gold subscription which includes an OVF of pfSense pre-provisioned for vmware among a million other things.

    I have two other suggestions while you're in the market:

    1.  Buy a managed Switch.  You're already in the UniFi line, so picking up a US-8-60 is a good start.  It'll power your UniFi AP, but most importantly it will allow you to start segmenting your network with VLANs.  If you need more than 8 ports, size up accordingly.

    2.  Buy a good network interface card for your server.  Friends don't let friends buy motherboards with Realtek NICs.  Intel -1gbe or Chelsio - 10gbe.

    Good luck!


  • Thanks for the replies.

    No, I promise I understand what a hypervisor is. I have never set one up, but I know what it is. just a bad use of terminology on my part.

  • Just buy an SG-2220 and use that as your main router. Then get a switch in whatever flavour you want and use the AP for wireless..

  • LAYER 8 Global Moderator

    Or the sg-3100 is available very soon.  Would be a good option as well if you want to run pfsense on its own box.. The sg2220 is end of sale.. So the 2240 or 3100 would be the replacements for that line.

    The 3100 would give you a few switch ports to play with.

    I am big fan of running pfsense on esxi though.. Works great, and as vm allows you the freedom to play development versions without little worry if something goes funky with a snap or something, because you can always roll back or just fire up your production VM running stable version, etc.  I had 3 different pfsense running as VM the other day so I could take some screen shots for a thread showing how to setup a transit network between 2 pfsense and route traffic across while using either pfsense internet connection in a failover mode, etc.

    Running VM host gives you lots of freedom to play and simulate without having to really mess with your "production" setup..

  • I feel compelled to mention a few things about ESXi. It is POSIX ish and includes busybox as well as a few other open source/GNU utilities which is why a lot people think it's Linux related/open source (also early versions ran on Linux).

    As mentioned, it's close source and type one (it IS the OS).

    ESXi standalone (no management/orchestration server) is free and with 6.5u1 can be managed (almost) entirely from the built-in web interface. This means you can run ESXi free with no Windows and no Microsoft period.

    Also as mentioned, don't use built in network cards unless server class chips are employed. Trust us.

    32GB of RAM is cute. Get a second hand server with a modern Xeon CPU and at least 64GB of RAM. Look for something like an R710 with dual hexacores (the broadcom NICs are fine).

    SAN running IN your hypervisor is great for lab environments but extremely touchy… Get a second server with a single quad core and 16GB of RAM for a SAN box. I use a purpose built Linux for this called ESOS. It works beautifully with iSCSI (use quality Intel nics) and fibre channel (I have some 8Gb qlogic cards running point to point because FC swiches are $$$).

    Because your two servers, modem, switch, router, console, main PC, and other misc equipment use a ridiculous amount of power, build a small atom based server running freeNAS (with Intel NICs) a few (less than 6) drives and use this as your main datastore (on ZFS) while running all of your critical services in BSD jails so you don't have to burn 700watts running Plex, Minecraft, and a seed box.

    But seriously, I do love the single pane of glass the Ubiquity offers. Just not the lack of flexibility. Forget the edgex. Get a SG-2440. If your under 200mb WAN you can run a full Snort setup somewhat comfortable. For the rest, run VMs and build out virtual networks for testing and experience. I had a small internet setup. Something like 6 pfSense VMs to simulate my company and show proof of concept for pfSense based IPsec VPNs (moving away from Cisco and my boss is open source phobic)

  • LAYER 8 Global Moderator

    You sure and the hell do not need all that to run pfsense on esxi ;)

    I have it running on a HP N40L, with 8GB of ram.. It is running multiple VMs not just pfsense.. 5 Currently running, one is my storage and plex server, another is running unifi controller on ubuntu.  So while yes its fantastic to have lots of ram and beefy cpu.. You sure and the hell do not need to drop $4k on your esxi setup.. Kind of defeats the purpose of it being FREE ;)

    That being said.. This setup can not push more than 200mbps to wan or between vlans.. I just tested that to since moving to a 500/50 home connection..  And going to get something new.. I would love a sg4860 - but sg2440 is better budget, but will most likely get the sg3100 because of the grief dropping 500+ dollars normally causes me with the committee (wife)… Will see what happens after I get her new car in nov ;)  For now I ordered the unifi usg since everything I read says it can push gig if you don't use qos, etc.  And it was only a 100 bucks so such purchases can sneak by the committee without any grief...

  • @johnpoz:

    For now I ordered the unifi usg since everything I read says it can push gig if you don't use qos, etc.  And it was only a 100 bucks so such purchases can sneak by the committee without any grief…

    You're going to return that in a week.  The USG is a total pile of hot garbage (or at least it was a year ago when I tried it last).  If you wanted to do anything useful, you have to edit config files and reprovision.  I'll admit it has probably gotten better with newer firmware/controller features, but when you have pfSense at your disposal, stick with pfSense man!

    I have 1000/250 at home, and if you're adamant about it, I'll test it for you and let you know what overall throughput is with no QoS.  But really, return it  ;D

  • LAYER 8 Global Moderator

    So I will agree that its clunky.. The only thing it has going for it is cheap, and it can handle my new 500/50 at speed.

    I am running the latest 4.4.8 code.. But I have left my pfsense vm running to handle dhcp and resolver - those features in the usg are pretty bare.. And no resolver at all.  Just forwarder mode. So good luck running your own dns on it, etc.

    And your correct anything like openvpn or HE tunnel has to be done at the cli - which wouldn't be the end of the world.  But if controller does a re provision seems that all gets wiped.. If your a simple home user with 1 network and no need for vpn or tunnel (it doesn't seem to do any ipv6 on its own) might be a good thing.  But from the day I have had to play with it seems more a toy than the mature very feature rich easy to use pfsense.. The dpi stuff looks interesting, but that seems to be the only thing its got going for it.  And with pfsense you could just install ntopng and get all that kind of info as well, etc.

    But again thanks for your thoughts and appreciated your willingness to test its top limit, etc.  Prob going to have to live with it till nov when I can replace it real pfsense hardware ;)  But after that it will be either on my self collecting dust - or looks like I might have some buyers for it ;)

Log in to reply